Wednesday, October 23, 2013

Alternative Multiple-Choice Origins

Two alternative forms of multiple-choice (AMC) to the traditional multiple-choice (TMC) developed from independent sources.  Geoff Masters from Melbourne, Australia, is credited as the developer of the parcel credit Rasch model (PMC), a form of Information Response Theory (IRT) analysis in 1982 (Bond and Fox).  It allows students to report what they know (2 points), what they do not know (1 point), and wrong answer (0 points). It never became popular on classroom or standardized tests.

The second form of AMC was developed at NWMSU. It started as net yield scoring (NYS) on both essay and multiple-choice. I needed a way to reduce the amount of reading required in scoring “blue book” essays. A 20-point essay started with 10 points. A point was added for acceptable, related, information bits. A point was subtracted for unacceptable, incorrect, unrelated information bits. An information bit was basically a short sentence with correct grammar and spelling. It could also be a relationship expressed as a diagram, sketch, or drawing.

This reduced the amount of reading by more than a 1/3 and improved student performance. Snow, filler, and fluff had no value but distracted a student from doing good work. Students needed to exercise good judgment in selecting what they wrote. This was no longer the case of their writing, and the teacher searching, for something that could earn them sufficient credit to pass the course; a lower level of thinking operation that is very common in high schools and colleges. NYS required students to use good judgment as well as be knowledgeable and be skilled.

This same idea was applied to computer scored multiple-choice tests with interesting results. When both TMC and NYS were offered on the same test, most students selected TMC on their first test. This is what they were familiar with. Over 90% of students elected NYS on their third test. Students also agreed that knowledge and judgment should have equal value.

By 1981 NYS was renamed knowledge and judgment scoring (KJS) to reflect what was being assessed: good judgment and a right answer (2 points), good judgment to report what has yet to be learned with no mark (1 point), and poor judgment, a wrong mark (0 points).

KJS requires and rewards students for using higher levels of thinking. The quality score is independent from the right count score. A struggling student with a test score of 60% may have also earned a quality score of 90%.

With TMC there is no way of knowing what a student with a score of 60% actually knows (when a right mark is a right answer or just luck on test day). With KJS we can know what this student knows with the same degree of accuracy as a student earning a 90% score on a TMC test.

More importantly, this reinforces the student’s sense of self-judgment and encourages effort to do better. It is the equivalent to the note a teacher marks on a special paragraph in an essay, “Good work!”

KJS provides the information needed to tell student and teacher what has been learned and what has yet to be learned in an easy to use report. Often a trail of bi-weekly test scores would follow a backward J. Reducing guessing by itself did not increase the test score but moved the score to a higher quality. Low quality students needed to change study habits. Low scoring high quality students needed to study more.

Learning by questioning and establishing relationships provided students the basis for answering question correctly that they had never seen before. They then stumbled onto what I meant by, “Make things meaningful (full of relationships) if your learning is to be really useful, empowering and easy to remember”. They did not have to review everything for each cumulative test.

The most interesting finding was that when students mastered meaning-making, they found themselves doing better in all of their courses. This is what inspired me to continue to promote Knowledge and Judgment Scoring. Students learn best when they are in charge. The quality score was the “feel good” score for struggling students until their improving development produced the high scores earned by successful self-correcting students.

Welcome to the KJS Group: Please register at Include something about yourself and your interest in student empowerment (your name, school, classroom environment, LinkedIn, Facebook, email, phone, and etc.).

Free anonymous download, Power Up Plus (PUP), version 5.22 containing both TMC and KJS:, 606 KB or, 1,099 KB.

- - - - - - - - - - - - - - - - - - - - - 

Other free software to help you and your students experience and understand how to break out of traditional-multiple choice (TMC) and into Knowledge and Judgment Scoring (KJS) (tricycle to bicycle):

No comments:

Post a Comment