#4
The first thing I noticed when inspecting the top of the test
scoring math model (Table 25) was that the variation within the central cell field has a different
reference point (external to the data) than the variation between scores in the
marginal cell column (internal to
the data). Also the variation within the central cell field (the variance) is
harvested in two ways: within rows (scores) and within columns (items).
The mean sum of squared deviations (MSS) or variance within a column or a row has a fixed
range (Chart 64 and Chart 65). The maximum occurs when the marks are 1/2 right
and 1/2 wrong (1/2 x 1/2 = 1/4 or 25%). [Variance also equals p * q or (Right *
Wrong)/(Right + Wrong)] The contribution each mark makes to the variance is distributed
along this gentle curve. The variable data
are fit to a rigid model.
I obtained the overall shape of these two variances by
folding Chart 64 and Chart 65 into Photo 64-65. The result is a dome or a depression above or below the upper
floor of the model.
The peak of the dome (maximum variance) is reached when a
student functioning at 50% marks an item with 50% difficulty. Standardized test makers try to
maximize this feature of the model. The larger the mismatch between item
difficulty and student ability, the lower down the position of the variance on
the dome. CAT attempts to adjust
item difficulty to match student preparedness.
Chart 66 is a direct overhead view of the dome. Elevation
lines have been added at 5% intervals from zero to 25%. I then fitted the data from Nursing124 to the roof of the model. The data only
spread over one quadrant of the model. The data could completely cover the dome
in an ideal situation in which every combination of score and difficulty
occurred.
The total test variance within items is then the sum of the
variance within all items (0.04 to 0.25 = 2.96). The total test variance within
scores is the sum of the variance of all scores (0.05 to 0.24 = 3.33). See
Table 8.
The math model
adjusts to fit the data in the marginal cell student score column (variance
between scores). The reference point is not a static feature of the model but
the average test score (16.77 or 80%). The plot of the variance between scores
can be attached to the right side of the math model (Chart 67).
The variance within
columns and rows spreads across the static frame of the model. The model then
adjusts to fit the variance between
scores (rows) to match the spread of the active within rows.
I can see another interpretation of the model variance if
the dome is inverted as a depression. As a flight instrument on a blimp: pitch,
roll, and yaw (within item, 2.96; within score, 3.31; and between scores, 4.10)
the blimp would have the nose up, rolled to the side, and with the rudder hard
over.
- - - -
- - - - - - - - - - - - - - - - -
Free
software to help you and your students experience and understand how to break
out of traditional-multiple choice (TMC) and into Knowledge and Judgment
Scoring (KJS) (tricycle to bicycle):
