While Project STAR dominated education conventions and the media, valid critiques of the study began to emerge.
Harvard University's Caroline Hoxby argued that the methodology of STAR was lacking. Its biggest flaw was that study participants knew they were being studied and hence tended to work to achieve outcomes desired by the researchers. As Hoxby writes, "the schools in a class size experiment may realize that if the experiment fails to show that the policy is effective, the policy will never be broadly enacted. In such cases the schools have incentives that the fully enacted policy would not give . . . . the experiment alters the incentive conditions . . . In addition, some individuals temporarily increase their productivity when they are being evaluated."
Eric Hanushek, then at the University of Rochester, also examined the study's methodology, but pointed to different shortcomings. Among these were:
* Between 20 and 30 percent of students in STAR quit each year, leaving less than half of the original group by the study's end.
* The students who quit were disproportionately low performers, providing a statistical boost to smaller classes.
* No pretests were given to students at the beginning of the study, providing no baseline off of which to measure achievement gains.
* While students for the program were chosen randomly, teachers and schools were not.
The analyses revealed major shortcomings of Project STAR. But public policy had already begun to be implemented.
The STAR study seemed to substantiate all that pro-class-size-reduction forces had ... finding no evidence that class-size-reduction has produced improved scores
, ... And the cost to achieve so little? To date, an estimated $8 billion.
Center for Education Reform - Sizing Up What Matters: The Importance of Small Schools