measurement

In the book Punished by Rewards, Alfie Kohn describes the perverse effects of incentive systems. Incentive systems fail because it's always easier (and cheaper) to game the system than to actually win. If you incentivize "call length" in a call center, for example, employees will hang up on people. If you incentive "resolutions" the staff will find short cuts to mark calls as "resolved" rather than to actually solve people's problems. He found that the best way to get people to do good work is to compensate them adequately and support their efforts to do good work.

Several years ago I wrote a post about educational measurement in which I described a phenomenon I have come to call the Melamine Effect. In 2007 and 2008, pet food and then milk were found that were contaminated with melamine. The consequences were horrific. In 2007 hundreds of pets were killed. In 2008, tens of thousands of infants were poisoned, many suffering liver damage. Several died. It turned out that unscrupulous people were diluting milk with water and then adding this cheap, industrial chemical — that coincidentally increases the score on a widely-used test for protein content — as a way to increase their profits.

Seeing the effects that standardized testing and so-called education reform were having on schools, I realized that the circumstances are perfectly analogous. If you measure something and use that measure to understand natural systems, you're fine. But if you start looking for treatments that will shift the measure, you're inviting all kinds of perverse effects, because educational measures can't actually measure what people are interested in (i.e. learning or understanding): they only measure factors that tend to covary with them in natural populations. Once you start applying treatments -- especially the cheapest ones -- you're almost assured of toxic effects.

And it should go without saying that just changing a measure doesn't mean you will actually produce better outcomes either. For years, people were told to take niacin to improve their cholesterol test scores. But a long-term study revealed that, although it did improve the scores, those gains were not actually associated with reduced risk of disease.

For a while the term was "data driven", but more recently the term is "evidence based". When people start using words like this, your hackles should rise. Look critically at the underlying model and how it relates evidence to the dimension of interest. This isn't always easy with the "dashboards" of the modern analytics systems. But it's the only way to avoid the "melamine effect".

I sent a quick letter to secretary of the faculty senate and the president of the MSP this morning:

I would like to call your attention to this article which describes how Clemson has been trying to game the rankings (and jumped from 38th to 22nd). It includes such chicanery as artificially limiting class sizes to below 20 (while allowing larger classes to get much larger) -- because "below 20" is a magical cut-off in the rankings. Student admissions are determined by how the SAT numbers will make the institution look. And there was a push to get large numbers of alumni to donate just $5 (because it makes the number of donors look good, even though the money itself is financially insignificant).

This kind of chicanery is exactly what I was warning about in my response to the Chancellor's Framework for Excellence.

These kinds of changes do nothing to actually help the research, service, and teaching of the University. They do not benefit -- and actually undermine -- the students, faculty, and mission of the institution. They represent diversions of funding that could be used to actually advance the institution, but are instead being spent to "influence" the rankings only.

Moreover, whatever benefits might be accrued by such changes in the rankings are bound to be short-lived. The rankings are artificial and contrived -- and will probably be changed as soon as it becomes apparent that institutions are gaming them.

I hope the Faculty Senate and Union will hold the chancellor accountable for his plan and work to ensure that this kind of game doesn't get played here. We need to look critically at our actual needs and devote spending to solve the problems we actually face. We must not engage in magical thinking that by gaming the rankings we can actually improve the institution.