Wednesday, August 22, 2012

Rasch Model Audit Finished

It was over two years ago that I took on the task to audit, make sense of, the Rasch model IRT analysis as performed by Winsteps, the software many state departments of education have used with NCLB standardized testing. It is now 46 posts later (they are being released on weekly intervals). The Winsteps software performs as advertised for psychometricians and test makers. No pixy dust is required. But the developmental environment it creates is problematic.

First is the collection of features provided to cull raw data to fit the perfect Rasch model. It creates a very freewheeling environment in which items and students are culled to get the results to “look right”. This is not a problem for competent experienced operators. In fact it is probably the best set of features for their work. But it is also an invitation to cheat in the hands of desperate state education officials needing results that “looks right” to obtain federal funding. It is an invitation that has been taken by politically motivated city and state officials (see prior posts that prompted the audit).

Early on some officials lost their jobs for making predictions based on research results that were not supported by application results. That no longer happens for a number of reasons. [Edit: It just happened again in Florida.] No one, to my knowledge, has been charged with a crime when test results and cut scores were manipulated for political gain; even when several million dollars were lost and millions of hours of instruction and learning time diverted to cramming for “the test” that at best produced a meaningless ranking, and worst, destroyed the validity of the results for instructional improvement and assessment of teacher performance.

A new threat to schooling is now being developed by predatory testing companies in collaboration with state departments of education, several of which were cheating. Two groups have formed (PARCC and SBAC). This is a good development. Each group can serve as a check on the other. One requirement being voiced outside these two groups is that the actual test scores must be published rather than the passing rate or a ranking of good, better, and best. Whenever the actual test scores are hidden there is no way to know what passing rates or rankings mean. However, the real threat is that, in the name of “formative assessment,” they want to increase standardized testing from once a year to three or four per year. One test has been destructive enough. More would only make things worse except for the predatory testing companies that will publish and score the tests and test preparation materials.

We have a problem. The federal government has shown its inability to improve education in the classroom by bullying schools with money. This failed program is now culminating in a political revolt, and perhaps, in legal action (see below). State departments of education have been too weak to resist being bullied: they needed money. Most public schools are operated as failing establishments by design (lesson plan, teacher presentation, and student performance rank [how well students adjust to the needs of the school’s administrators, staff, and teachers] at the lowest levels of thinking). Schools designed for success are organized around student learning at all levels of thinking. They adjust to the needs of their students. Failing school administrators are more managers than leaders. The result is their teachers are no longer free to be professional teachers (responding to their student's needs). In some states teachers now function as readers of the assigned daily lesson, as coach for "the test", and as day care servants (responding to the perceived needs of the bureaucracy).

There is one branch of government that I have yet to see get involved. That is the office of the state attorney general. That branch of government has the responsibility to prevent tax payers from being ripped off by unethical business practices. Predatory testing companies in conjunction with self-serving state education officials may be about to pull off the biggest scam yet: not one but three or four standardized tests per year per course that, for the most part, will be scored at the lowest levels of thinking (using traditional multiple-choice and high speed evaluated essays).

If they justify this use of multiple tests as “formative assessment”, it will be a patent lie; pure fraud; marketing a product under the guise of a current education fade. Formative assessment occurs in seconds in the heads of students functioning at higher levels of thinking. It occurs in the classroom in minutes between students and teachers (in person and through educational software). It does not occur in weeks or months by way of standardized tests (unless you are promoting the tests). Marion Brady sums the situation up in neat operational terms that everyone can understand: "Do not subject my child to any test that doesn't provide useful, same-day or next-day information about performance."

Are you, or do you know someone, in the office of your state attorney general who is interested in forming a group to act on this matter? Nine-Patch Multiple-Choice, Inc, for-profit, now has facilities to support this action. It can perform multiple-choice test analyses at both lower and higher levels of thinking (right count scoring and Knowledge and Judgment Scoring); classical test theory (CTT) and item response theory (IRT); and offers students the choice between being assessed by traditional guessing or reporting what they trust.  I would suggest that you submit a test data set of 25 to 50 questions and 120 to 300 students for analysis as an entry into the group. We need real test data. Not research or theoretical data. Once a group is formed Nine-Patch Multiple-Choice, Inc. can be re-registered as a non-profit, to give the mission even greater stability. Email or phone 1-573-808-5491.

Currently there is no way to audit and verify test results from predatory testing companies or state departments of education. A proposed law could require that random samples be selected, and be examined by an independent company or the office of the state attorney general using appropriate software, that currently runs on a personal computer, and produces meaningful results in minutes rather than months (see next post). 

Richard A. Hart, PhD
Professor of Biology, Emeritus, NWMSU - 1990
Treasurer - 1992
Educational Software Cooperative
Treasurer and Founding Board Member - 1994
Educational Software Cooperative, Inc. (non-profit)
President - 2006
Nine-Patch Multiple-Choice, Inc. (for-profit)
803 Somerset Drive, Columbia, MO 65203-6436 USA
Phone:     1-573-808-5491
Organizer - ????
Nine-Patch Multiple-Choice, Inc. (non-profit)

No comments:

Post a Comment