Wednesday, July 10, 2013

Visual Education Statistics - Equating

The past few posts have shown that if two tests have the same student score standard deviation (SD) they are easy to combine or link. Both tests will have the same student score distribution on the same scale.

Equating is then a process of finding the difference between the average test scores and applying this value to one of the two sets of test scores. Add the difference in average test score to the lower set of scores, or subtract it from the higher set to combine the two sets of test scores.

This can be done whenever the SDs are within acceptable limits (considering, all factors that may affect the test results, the expected results, and the intended use of the results). This is IMHO a very subjective judgment call to be made by the most experienced person available.

There are two other situations: same average test score but the different SDs are beyond acceptable limits, and both test score and SD differences are beyond acceptable limits for the two tests. In both cases we need to equate the two different SDs, the two different distributions of student scores.

Chart 48 is a re-tabling of Chart 44. The x-axis in Chart 48 shows the set Standard Deviation (SD) used in the VESE tables in prior posts. Equating a low SD test (10) to a high SD test (30) has different effects then equating a high SD test (30) to a low SD test (10). The first improves the test performance; the second reduces the test performance.

There is then a bias to raise the low SD test to the high SD test. “The test this year was more difficult than the test last year,” was the NCLB explanation from Texas, Arkansas, and New York. [It was not that the students this year were less prepared.]

The most frequent way I have seen mapping (Livingston, 2004, figure 2, page 14) done is to plot the scores of the test to be equated on the x-axis and the scores of the reference test on the y-axis. The equate line for two tests with similar average test scores and SDs is a straight line from zero through the 50% point on both axes (Chart 49).

If the average test scores are similar but the SDs are different, the equate line becomes tilted to expand (Chart 50) or contract (Chart 51) the equated values to match the reference test. Mapping from a low SD test to a higher SD tests leaves gaps. Mapping from a high SD test to a low SD tests produces clumping, in part, from rounding errors.

Mapping a new difficult test to an easier reference test with the same SD increases the values on the equating line, as well, as truncates it. Any new test scores over 30 on Chart 52 have no place to be plotted of the reference test scale. 

The equating with an increase in both SD and average test score expands the distribution and truncates the equating line even more (Chart 52). A comparison of the two above situations as parallel lines (Chart 53) helps to clarify the differences.
Both increase the new difficult test average test score value of 20 counts to 30 counts on the reference scale. In this simple example based on a normal distribution, the remaining values increase in a uniform manner of equal units of 10 with the same SD and 15 when mapping to the larger SD.

The significance of this is that in the real world, test scores are not distributed in nice ideal normal distributions. The equating line can assume many shapes and slopes.

The unit of measure needed to plot an equating chart must include equivalent portions of the two distributions. Percentage is a convenient unit: equipercentile equating. [More on this in the next post.]

Whither Test A is the reference test, or Test B is the reference test, or both are combined as one analysis is the difficult subjective call of the psychometrician. So much depends on the luck on test day related to the test blueprint, the item writers, the reviewers, the field test results, the test maker, the test takers and many minor effects on each of these categories. 

This is little different from predicting the weather or the stock market, IMHO. [The highest final test scores at the Annapolis Naval Academy were during a storm with very high negative air ion concentrations.] The above factors also need to include the long list of excuses built into institutionalized education at all levels.

On a four-option item, chance alone injects an average 25% value (that can easily range from 15 to 35%) when students are forced to mark every item on a traditional multiple-choice (TMC) test. Quality is suppressed into quantity by only counting right marks: Quality and quantity are therefore linked into the same value. TMC high test scores have higher quality then lower test scores, but this is generally ignored.

It does not have to be that way. Both the partial credit Rasch model IRT and Knowledge and Judgment Scoring permit students to report what they trust they know and can do and what they have yet to learn accurately, honestly and fairly. No guessing is required. Both paper tests and CAT tests can accept, “I trust I know or can do this,” “I have yet to learn this,” and if good judgment does not prevail, “Sorry, I goofed.”  Just score 2, 1, and 0 rather than 1 for each right mark (for whatever reason or accident).

A test should encourage learning. The TMC at the lower scores is punitive. By scoring for both quantity and quality (knowledge and judgment) students receive separate scores, just as is done on most other assessments. “You did very well on what you reported (90% right) but you need to do more to keep up with the class” rather than “You failed again with a TMC score of 50%.

Classroom practice during the NCLB era tragically followed the style of the TMC standardized tests conducted at the lowest levels of thinking. The CCSS tests need to model rewarding students for their judgment as well as right marks. [We can expect the schools to again doggedly try to imitate.] It is student judgment that forms the basis for further learning at higher levels of thinking: one of the main goals of the CCSS movement. The CCSS movement needs to update its use of multiple-choice to be consistent with its goals.

Equating TMC meaninglessness does not improve the results. This crippled form of multiple-choice does not permit students to tell us what they really know and can do that is of value for further learning and instruction.

- - - - - - - - - - - - - - - - - - - - - 

Free software to help you and your students experience and understand how to break out of traditional-multiple choice (TMC) and into Knowledge and Judgment Scoring (KJS) (tricycle to bicycle):

No comments:

Post a Comment