21
[[Second Pass, 8 July 2014. Equation 6.3 (cited below) in Statistical Test Theory for the Behavioral Sciences by Dato N.M. de Gruijter and Leo J. Th. van der Kamp, 2008, is the same as the calculation used in Table 29, in my 9 July 2014 post. On the following page they mention that the error variance is higher in the center and lower at the extremes. That distribution is the green curve on Chart 73. I did not see this relationship in the equation when this post was first posted, but do now in the visualized mathematical model (Chart 73).
[[Second Pass, 8 July 2014. Equation 6.3 (cited below) in Statistical Test Theory for the Behavioral Sciences by Dato N.M. de Gruijter and Leo J. Th. van der Kamp, 2008, is the same as the calculation used in Table 29, in my 9 July 2014 post. On the following page they mention that the error variance is higher in the center and lower at the extremes. That distribution is the green curve on Chart 73. I did not see this relationship in the equation when this post was first posted, but do now in the visualized mathematical model (Chart 73).
Also the discussion of Table 24
has been updated to match the terms and values in Table 24.]]
Working on the conditional standard error of measurement
(CSEM) is new territory for me. I always associated the CSEM with the Rasch
model IRT analysis commonly used by state departments of education when scoring
NCLB tests. I first had to Google for basic information.
If you are interested in the details, please check out these
sources for sample (n-1) equations: (Equation 6.14 that corrects the relative
variance was not included in the 2005 version of the current 2008 version. This
represents a significant progress in applying test precision.)
- Absolute Error Variance Equation 5.39 p. 73
- Relative Error Variance Equation 6.3 p. 83
- Corrected Relative Variance Equation 6.14 p. 91 or GED Equation 3 p. 9
My first surprise was to find I had already calculated the
CSEM for the Nursing124 data when I put up Post 5 of this series (in Table 8. Interactions with
Columns [Items] Variance, MEAN SS = 3.33) as I discovered five ways to harvest the variance [mean sum of squares (MSS)]. Equation 6.3 n, Table 22, produces the same
result (test SEM = 1.75) when it divides by n [unknown
population] rather than n-1 [observed sample].
[n = the item count. Test SEM = AVERAGE(CSEM).]
I then used what I learned in the last post to table data to
obtain the conditional error variance for student scores (Table 23a). The 21 items in Table 22 became the number of right marks on each of 11 item difficulties on Table 23a. The
values in this tabulation were then converted into frequencies conditional on the student scores; the sum of which added to one, for each score (Table 23b).The absolute error variance for each score was computed by Excel (=Var.P). Multiplying the absolute error variance (0.14382) by the square of the item count (21^2) yields the relative error variance (63.42). [Equation 5.39 (0.14382) * n^2 = Equation 6.3 (63.42)] The square root of the relative error variance of each score yields the CSEM for that score. [An alternate calculation of the absolute error variance is shaded in Table 23b. Here the variance was calculated first and that value divided by the squared score to obtain the absolute error variance. This helps explain multiplying the absolute error variance by the squared item count to obtain the relative error variance for each score.]
Complete curves were generated from Equation 6.3 for n-1 and
for GED n-1 (Table 24). The GED n-1 analysis includes a correction factor (cf) for
the range of item difficulties on
the test [cf = (1- KR20)/(1-KR21)]. This factor is equal to one if all items
are of equal difficulty. For the Nursing123 data it was 1.59; the difficulties
ranged from 45% to 95%, from the middle of the total possible
distribution to one extreme.
The CSEM values from the six analyses are listed in Table 24.
Five are fairly close to one another. The GED n-1, with a correction for the
range of item difficulties, is far different from the other five (Chart 61).
Values could not be created for the full curve for conditional frequencies as
you must actually have student marks to calculate conditional frequency CSEM
values. The gray area shows the values calculated from an equation for which
there were no actual data. Equations produce nice looking, “look right”,
reports.
The CSEM improves the reportable precision on this test over
using the test SEM. Good judgment (best practice) is to correct the CSEM values
as done on the GED n-1 analysis.
[I did not transform the raw test score mean of 16.8 or
79.8% to a scale score of 50% as was done by Setzer, 2009, GED, p. 6 and Tables
2 and 3. The GED n-1 raw score cut point was 60% which is comparable to most
classroom tests. If 25% of the score is from luck on test day that leaves 35%
for what a student marked right as something known or could be done, as a worst
case. If half of the lucky marks were also something the student knew or could
do, the split would be about 10% for luck on test day and 50% for student
ability.]
In Table 24, the GED n-1 analysis test SEM of 2.98 for the Nursing124
data is, as a range, 2.98/21 or 14.19%. For the uncorrected Equation 6.3 n-1
analysis, 1.79, the range is 1.79/21 or 8.52%. The n SEM was 1.75 or 7.95%.
The n SEM range, 1.75, fits within the uncorrected n - 1 test SEM value, 1.79. The corrected GED n-1 test
SEM value, 2.98, exceeds it.
Student score CSEM values are even more sensitive than the
test SEM values. The maximum range for the GED n-1 analysis is 3.73 or 3.73/21
or 17.76% and for the Equation 6.3 n-1 analysis 2.35 or 11.19%. Both are beyond
the maximum n CSEM value of 2.29 or 10.41%. This low quality set of data fails to qualify as a means of
setting classroom grades or a standardized test cut score.
[However the classroom rule of 75% for passing the course
and the rule for grades set at 10 percentage points over rule these statistics.
Here is a good example that test statistics have meaning only in relation to
how they are used. If the process of data reduction and reporting is not
transparent, the resulting statistics are suspect and can produce extended
debates over a passing score in the classroom.]
The CSEM for each student score does improve test precision.
It can be calculated in several ways with close agreement. But it cannot
improve the quality of the student marks on the answer sheets made under
traditional, forced-choice, multiple-choice rules. These tests only rank
students by the number of right marks. They do not ask students, or allow
students to report, what they really know or can do; their judgment in using
what they know or can do.
The CCSS movement is now promoting learning at higher levels
of thinking (problem solving) with, from which I have learned, some
de-emphasis on lower levels of
thinking that are the foundation for higher levels of thinking. A successful
student cycles through all levels of thinking, as is needed. Yet half of the CCSS
testing will be at the lowest levels of thinking, traditional multiple-choice
scoring. The other half will be as much of an over kill as traditional
multiple-choice is an under kill in assessing student knowledge, skills, and
student development to learn and apply their abilities. Others
have this same concern that centralized politics (and dollars) will continue to
overshadow the reality of the classroom.
There is a middle ground that makes every question function
at higher levels of thinking, allows students to report what is meaningful, of
value, and empowering, and has the speed, low cost, and precision of
traditional multiple-choice. Knowledge and
Judgment Scoring and partial
credit Rasch model IRT are two examples. They both accommodate students
functioning at all levels of thinking.
Lower ability students do not have to guess their way through a test. With
routine use, both can turn passive pupils into self-correcting highly
successful achievers in the classroom. If you are really into mastery learning,
you can also try something like Knowledge
Factor.
- - - - - - - - - - - - - - - - - - - -
-
Free software to help you and your students
experience and understand how to break out of traditional-multiple choice (TMC)
and into Knowledge and Judgment Scoring (KJS) (tricycle to bicycle):
No comments:
Post a Comment