I study testing of the sort he focuses on, and is at the core of a system he points out is more similar across Chinese and Western societies than people think.
What I've come to realize is that everything he wrestles with in the essay is not hidden away, it's right there within the testing system. It's studied empirically, the markets are there, it's all in plain view. It doesn't require any special wisdom or insight and anyone trying to simplify things either way are wrong.
These tests that are scrutinized bear only a modest relationship with what they are trying to measure, and what they are trying to measure is only weakly related to what people want to know. That is, the best estimates suggest that they're only correlated about .30 or .40 with actual ability or understanding in any deeper sense at most. And this is over the entire range of ability, meaning that it includes comparisons between people who score in the 2nd percentile and those who score in the 98th percentile -- a huge portion of that correlation is surely due to gross comparisons of that sort.
What's sobering to me is that we know this. We know these tests are not only imperfect, but severely so. And yet we fetishize it. The money being put into the testing field is astounding.
The reason of course, is because, while these tests are poor measures, they're the better than nothing. And we know that they tell us something about people, on average, across many people.
I'm not sure why we overrely on these things, why these discussions are often framed in extremes, where either a test is the criterion for success, or react to it with claims that the tests are completely invalid.
What I've come to realize is that everything he wrestles with in the essay is not hidden away, it's right there within the testing system. It's studied empirically, the markets are there, it's all in plain view. It doesn't require any special wisdom or insight and anyone trying to simplify things either way are wrong.
These tests that are scrutinized bear only a modest relationship with what they are trying to measure, and what they are trying to measure is only weakly related to what people want to know. That is, the best estimates suggest that they're only correlated about .30 or .40 with actual ability or understanding in any deeper sense at most. And this is over the entire range of ability, meaning that it includes comparisons between people who score in the 2nd percentile and those who score in the 98th percentile -- a huge portion of that correlation is surely due to gross comparisons of that sort.
What's sobering to me is that we know this. We know these tests are not only imperfect, but severely so. And yet we fetishize it. The money being put into the testing field is astounding.
The reason of course, is because, while these tests are poor measures, they're the better than nothing. And we know that they tell us something about people, on average, across many people.
I'm not sure why we overrely on these things, why these discussions are often framed in extremes, where either a test is the criterion for success, or react to it with claims that the tests are completely invalid.