You are here

Educational Measurement

Educational measurement doesn't work and shouldn't be called measurement. The reductionism and worship of quantification in our society is twisting education as a mantra of "improving scores" drives every decision in the schools. We should make decisions about education based on what makes sense, not merely on what improves test scores.

The premise of educational measurement is that, if you can't measure things directly, you can find indicators that vary in the same direction as as what you want to measure and you can aggregate those together in a "construct". In other words, we can't measure how good something is, but we can measure other things that contribute to what make it good and use those together as a measure of quality. The problem is that while these so-called measurements may work reasonably well with a natural population, they don't substitute for a prescription. Let me give you an example.

Let's develop a measure for how good cookies are. Let's say we look at a hundred different kinds of cookies and decide that the biggest and sweetest cookies are the best. We could develop a measure that uses the weight and percent sugar as indicators and aggregate those as our construct. Our construct is simple to apply. It might work great with real cookies. But you could also make cookies with mud and sugar -- they might score really high, but I'll bet they wouldn't taste very good.

What do you do? Well, the standard approach is to try to add more dimensions as part of the index. You can measure fat, starch, hardness, etc, etc. Your measure gets more and more complex to apply. But no matter what you do, you're still not going to have a measure of how good cookies are. For one reason, because it isn't possible to know all of the dimensions that make cookies good. People invent new ways to make cookies good every day. People thought Oreos were good until Doublestuff cookies came out. The second reason is that there's no agreement about what makes a really good cookie. Some people like chocolate chip and others like vanilla wafers: its stupid to argue about which one is better.

Here's the worst problem: Even if you measure a hundred dimensions, it still won't tell people how to make a good cookie. Once you start applying the "instrument" (that's what the educational measurement people call a test) people start using the scores to decide what makes cookies good, rather just trying to make good cookies. This is how disasters like the melamine contamination happen. They were using a test for quality in children's milk that included a test for protein. It turns out that adding melamine to products makes products test higher for protein and is really cheap. When the tests drive school policy, schools are compelled to start look for anything like melamine: something that improves the scores on the tests, regardless of it's actual value.

Just like we all know what a good cookie tastes like, we all know good education when we see it. Education should be about engaging children in interesting work that requires them to construct knowledge in meaningful ways. We need to return to a model that uses common sense to improve education, and not be slaves to measures that don't really measure what we care about.