It was over two years ago that I took on the task to audit, make sense of, the Rasch
model IRT analysis as performed by Winsteps, the software many state
departments of education have used with NCLB standardized testing. It is now 46
posts later (they are being
released on weekly intervals). The Winsteps software performs as advertised for
psychometricians and test makers. No pixy dust is required. But the
developmental environment it creates is problematic.
First is the collection of features provided to cull raw data to fit
the perfect Rasch model. It creates a very freewheeling environment in which
items and students are culled to get the results to “look right”. This is not a
problem for competent experienced operators. In fact it is probably the best
set of features for their work. But it is also an invitation to cheat in the
hands of desperate state education officials needing results that “looks right”
to obtain federal funding. It is an invitation that has been taken by politically
motivated city and state officials (see prior posts that prompted the audit).
Early on some officials lost their jobs for making predictions based on
research results that were not supported by application results. That no longer
happens for a number of reasons. [Edit: It just happened again in Florida.] No one, to my knowledge, has been charged with
a crime when test results and cut scores were manipulated for political gain;
even when several million dollars were lost and millions of hours of
instruction and learning time diverted to cramming for “the test” that at best
produced a meaningless ranking, and worst, destroyed the validity of the
results for instructional improvement and assessment of teacher performance.
A new threat to schooling is now being developed by predatory testing
companies in collaboration with state departments of education, several of
which were cheating. Two groups have formed (PARCC and SBAC). This is a good development. Each
group can serve as a check on the other. One requirement being voiced outside
these two groups is that the actual test scores must be published rather than
the passing rate or a ranking of good, better, and best. Whenever the actual
test scores are hidden there is no way to know what passing rates or rankings
mean. However, the real threat is that, in the name of “formative assessment,”
they want to increase standardized testing from once a year to three or four
per year. One test has been destructive enough. More would only make things
worse except for the predatory testing companies that will publish and score the
tests and test preparation materials.
We have a problem. The federal government has shown its inability to
improve education in the classroom by bullying schools with money. This failed program is now culminating in a political revolt, and perhaps, in legal action (see below). State
departments of education have been too weak to resist being bullied: they needed money. Most public schools are operated as failing
establishments by design (lesson plan, teacher presentation, and student
performance rank [how well students adjust to the needs of the school’s
administrators, staff, and teachers] at the lowest levels of thinking). Schools
designed for success are organized around student learning at all levels of
thinking. They adjust to the needs of their students. Failing school
administrators are more managers than leaders. The result is their teachers are
no longer free to be professional teachers (responding to their student's needs). In some states teachers now function
as readers of the assigned daily lesson, as coach for "the test", and as day care servants (responding to the perceived needs of the bureaucracy).
There is one branch of government that I have yet to see get involved.
That is the office of the state attorney general. That branch of government has
the responsibility to prevent tax payers from being ripped off by unethical
business practices. Predatory testing companies in conjunction with
self-serving state education officials may be about to pull off the
biggest scam yet: not one but three or four standardized tests per year per
course that, for the most part, will be scored at the lowest levels of
thinking (using traditional multiple-choice and high speed evaluated essays).
If they justify this use of multiple tests as “formative assessment”, it will be a patent lie; pure fraud; marketing a product under the guise of a current education fade. Formative assessment occurs in seconds in the heads of students functioning at higher levels of thinking. It occurs in the classroom in minutes between students and teachers (in person and through educational software). It does not occur in weeks or months by way of standardized tests (unless you are promoting the tests). Marion Brady sums the situation up in neat operational terms that everyone can understand: "Do not subject my child to any test that doesn't provide useful, same-day or next-day information about performance."
If they justify this use of multiple tests as “formative assessment”, it will be a patent lie; pure fraud; marketing a product under the guise of a current education fade. Formative assessment occurs in seconds in the heads of students functioning at higher levels of thinking. It occurs in the classroom in minutes between students and teachers (in person and through educational software). It does not occur in weeks or months by way of standardized tests (unless you are promoting the tests). Marion Brady sums the situation up in neat operational terms that everyone can understand: "Do not subject my child to any test that doesn't provide useful, same-day or next-day information about performance."
Are you, or do you know someone, in the office of your state attorney
general who is interested in forming a group to act on this matter? Nine-Patch
Multiple-Choice, Inc, for-profit, now has facilities to support this
action. It can perform multiple-choice test analyses at both lower and higher
levels of thinking (right count scoring and Knowledge and Judgment Scoring); classical test theory (CTT) and item response theory (IRT);
and offers students the choice between being assessed by traditional guessing
or reporting what they trust. I
would suggest that you submit a test data set of 25 to 50 questions and 120 to
300 students for analysis as an entry into the group. We need real test data.
Not research or theoretical data. Once a group is formed Nine-Patch
Multiple-Choice, Inc. can be re-registered as a non-profit, to give the mission even greater stability. Email rhart@nine-patch.com
or phone 1-573-808-5491.
Currently there is no way to audit and verify test results from
predatory testing companies or state departments of education. A proposed law
could require that random samples be selected, and be examined by an
independent company or the office of the state attorney general using appropriate
software, that currently runs on a personal computer, and produces meaningful
results in minutes rather than months (see next post).
Richard A. Hart, PhD
Professor of Biology, Emeritus, NWMSU - 1990
Treasurer - 1992
Educational Software Cooperative
Treasurer and Founding Board Member - 1994
Educational Software Cooperative, Inc. (non-profit)
Website: http://www.edu-soft.org
President - 2006
Nine-Patch Multiple-Choice, Inc. (for-profit)
803 Somerset Drive, Columbia, MO 65203-6436 USA
Email: rhart@nine-patch.com,
Website: http://www.nine-patch.com
Phone: 1-573-808-5491
Organizer - ????
Nine-Patch Multiple-Choice, Inc. (non-profit)
Organizer - ????
Nine-Patch Multiple-Choice, Inc. (non-profit)
No comments:
Post a Comment