Unexpected Student Performance

Two findings in the Rasch Model Audit posts are important in understanding what item response theory (IRT) analysis actually does. One relates perfectly to the PUP Table 3c. Guttman Mark Matrix (student test performance) and the other to the estimation of student quality (% Right).

A Guttman Mark Matrix is created by sorting student scores and question difficulties from high to low. Student scores are displayed vertically and question difficulties horizontally. The end result is that the highest scoring student and easiest item land in the upper left of the table; the lowest scoring student and most difficult question land in the lower right of the table. A diagonal line from lower left to upper right divides more able students and easier questions from less able students and harder questions.

In a perfect Guttman scalogram, all marks in the upper left would be right and all marks in the lower right would be wrong. The average test score and question difficulty would be 50%. The average score on the selected test, Fall8850a, is 70%. Therefore the scores and difficulties are distributed about the average test score of 70%.

Winsteps develops an “unexpected” value for each cell in the table. PUP imports this information to color “most unexpected” values in a dark color and “less unexpected” values in a light color on Table 3c. Guttman Mark Matrix. As can be seen, this is a perfect match.

Purple “most unexpected” omits are most prominent on the chart. The assumption is students with these abilities should not have omitted. However, this is based on another set of assumptions: all students learn at the same rate, understand at the same level of thinking, and have common experiences. There is a wide difference between theory and application in the classroom. This provides the basis for several stories.

One commonly occurring story is that students doing poorly are often “pulled out” of their classrooms for special attention. Could it be that the only five students that omitted question 16 were “pulled out” at such a time they missed the instruction the rest of the class received? They did the right thing by omitting rather than guessing. This same option should be available on standardized tests. Students should be able to omit to honestly adjust the test to their preparation. Guessing degrades the value of multiple-choice tests. Each right answer from a guess is a lie. It is not a report of what a student knows. It cannot be the basis for further meaningful instruction. It degrades the test score to a meaningless ranking. These five students sent a very clear signal of what happened (or in this case, did not happen) in the classroom.

Three students failed to use good judgment to accurately report what they trusted to be of value as the basis for further instruction and learning. Coloring makes this as obvious as scanning the poor judgment (PJ) column. Why did they guess?  Two show several “most unexpected” right answers. One story is student Order 016 had his grade made, so he just marked whatever popped into mind and did not take the time to think through each question on this last hour test in the course. He not only answered many questions correctly that were expected as too difficult for his apparent ability, but also, missed several expected to be very easy for him. He turned his paper in early in the test period.

Student Order 035 presents a different story. She took more time but again displayed the behavior of a student not needing a high score on this last hour test or a student that just failed to voluntarily switch from guessing at answers (at lower levels of thinking) to reporting what she trusted (at higher levels of thinking). She has the same test score as three other students who exercised good judgment indicated by much higher quality scores (% Right). Quality is discussed in the next post.

The last exceptional behavior is from student Order 31. This student is a model (lower level of thinking, traditional, right mark scoring, guessing, forced-choice) test taker: every question falling to the upper left in the table is marked right and 2/3 of the questions falling to the lower right are marked wrong.

Clearly, the information from Winsteps Table 6.6 adds greatly to the ease of use of three PUP tables: 3a. Student Counseling Matrix, test maker view; 3b. Student Counseling Matrix, test taker view (unique to KJS); and  3c. Guttman Mark Matrix, combined teacher view.