My wife and I just finished watching a documentary about the recent college admission scandal. As you probably know, the company billed itself as guaranteeing admission to various prestigious colleges for a fee utilizing very questionable and often illegal assists such as, posing the students as athletes, bribing officials and having someone else take admission tests like the ACT and SAT. About fifty families were prosecuted along with the principal, a man named Rick Singer.
While this company and its activities were fraudulent and its methods deceitful and illegal, some issues were raised for me that are worth our attention. Back in the dark ages when I was a high school junior, my English teacher who hailed from Boston informed us that students in New England studied for the SAT exams, something which we in Chicago did not do. At the time, I do not believe that there were commercial outfits devoted to test preparation so the preferred regimen was centered upon vocabulary which made sense since it has the highest correlation with total verbal score. In the decades since, there is a huge industry in prepping for exams of all kinds, not only SAT's but law boards, GRE's and MedCATs, to name a few. I was uneasy when my English teacher made his suggestion and I learned why later in my graduate education.
SAT stands for Scholastic Aptitude Test. Aptitude is another word for ability. In its purest form, the SAT is supposed to measure readiness for college level courses. The validity of the test as a measure of readiness rests upon its integrity as a test i.e. a sampling of knowledge that correlates positively with ability to perform successfully in college. Preparation for the test amounts to "teaching the test" and "teaching the test" is like giving the test takers the answers to the test before they take it. Doing so invalidates the test as a measure of ability or readiness. Instead it measures the success of the preparation. Interestingly, the company that offers the SAT also sells test preparation which undermines its own product's value. In the years since my teacher's suggestion, a huge industry has developed to actually invalidate aptitude or readiness testing.
Scoring is also a problem for most if not all of these predictive exams. Most people taking and interpreting the test results do not really understand what they are seeing. Often we hear about students who got a "perfect score" on the SAT's, an 800. Most people hearing this think that the test taker answered all the questions on the test correctly. That is untrue. The "800" is a stanine score which tells us where the test taker's score ranks among a representative sample of other people who took the test. In the case of the SAT's, it is, or used to be, the group that took the test three years before. The "800" means the test taker scored three standard deviations above the mean of "500" and places the person in the 99th percentile i.e. their score was better than 98% of the other test takers who took the SAT three years before they did. A score of "600" is one standard deviation above the mean and places the test takers at the 84th percentile. But what exactly does a standard deviation mean? It is essentially a margin of error measurement which tells us what confidence we can have in the score. A "600" for example is different than a "500" roughly four out of five times. A "700" is different than a "500" roughly nine out of ten times. What that tells us about the difference between a "580" and a "610" is that they are virtually the same statistically. When students retake the SAT's in order to improve their scores, the differences are likely to be negligible statistically and probably can be attributed to a "practice effect" which just means that the more you take the test, the better you get at taking it.
Getting back to Rick Singer and the college admission scandal, it seems to me his actions are really an extension of an almost universally held belief that sabotaging the integrity and validity of the admissions process by subverting the standards of measurement is acceptable and even desirable. Admittedly, he engaged in deception which is essentially different than test preparation but both are problematic if we are serious about predicting readiness for college. I think most of us would look askance at a teacher giving the answers to the test ahead of time. Test preparation courses essentially "teach the test" and accomplish the same thing. Tests are not readiness in themselves, they are predictors of readiness. If they are not valid, they cannot fulfill that goal.