[00:00:15] Amy H-L: I’m Amy Halpern-Laff,
[00:00:16] Jon M: And I’m Jon Moscow. Welcome to Ethical Schools. Our guest today is Harry Feder, executive director of FairTest, a nonprofit that advocates for equitable and transparent educational testing. Harry came to FairTest after a long career in public education in New York City as both teacher and researcher.
This is Part 1 of a two-part interview. Today we’ll discuss college admissions tests, specifically the SAT. In Part 2, we’ll focus on high stakes tests in K-12. Welcome, Harry.
[00:00:46] Harry F: Hi, Jon. Hi, Amy. Nice to be here.
[00:00:48] Amy H-L: Would you tell us a little about the history of college admissions testing?
[00:00:52] Harry F: Well, it goes back about a century with the invention of the SAT, which was derived from the Binet intelligence tests used to sort officers for the army in World War I to make sure that we don’t have too many Southern Europeans and Black people in the officer corps.
There’s a lot been written about this. I recommend folks check out Nick Lemann’s writing, for example. I’ll plug. He’s got a new book coming out about the SAT in September from Princeton University Press that promises to be good.
There was an idea that “upper elite” Ivy League education was sort of an old boys network and it wasn’t capturing some of our best and brightest. And so, the Conants at Harvard and others wanted to create a test that would essentially actually capture folks, based on merit, of course. Over time, that test proved to have its own cultural and racial biases. This was the post-World War II era. No, I guess a little bit before that, around World War II. Don’t quote me. I’d have to go back to the readings to get all the dates.
Then the ACT was the competitor to the SAT. The SAT was created by the Educational Testing Service, which came out of folks from the Ivy League who tried to create this test where you could compare people from all around the country. And so you do selective college admissions that way. And then the ACT and the SAT basically became the coin of the realm. The ACT tended to be more of a Midwest thing, but now it’s spread to be probably a competitor to the SAT, at least in the college admissions realm.
The SAT at one point shifted from ETS to the College Board. The College Board is a member organization, a putative nonprofit, but the CEO makes over two million dollars a year. They administer the SAT, which is now digital. It’s brand new. It’ s now a shorter, more flexible exam. And also what is on the test has changed over time. On the verbal section, there used to be something called analogies like, yacht is to main mast as… That didn’t fly so well. They got rid of that because those relationships and words had cultural biases baked in.
It’s what we call a norm reference test. The scoring of it is on a bell curve. There’s a big middle and people who get at the very high end and the very low end are fewer. Through the exercises and psychometrics, they set the scores and how many you have to get right in order to… It’s interesting. This new, SAT, which we can talk about a little bit later maybe, is an adaptive test but it’s stage adaptive. So if you get x right on the first part of a section, you go to one second part. If you get y right on that section, you go to a different second part. So the kinds of questions you get determine your score, not just the number of questions you get right. And of course, how do we know when a question is hard versus a question is easy? We know it by the number of people who get it right or wrong, right? It’s sort of a psychometric self-fulfilling prophecy.
So that’s the history of it. It was originally essentially a race and class filterer. Then there was an idea that we need to bring in more smart people into higher academia. Jews. But then of course there became too many Jews. But it over time very much became a class and race sorter, as it were.
[00:05:06] Jon M: So what does the SAT purport to test and what does it actually test?
[00:05:12] Harry F: Your guess is as good as mine. I’m only half joking. The College Board and the ETS will make the claim that it is a predictor of freshman year grades in college. And thus it is purports to through some, you know, miracle, it purports to indicate who will do well at least in the beginning of college.
That’s all it purports to do. It is not. You know, they used to call it the SAT, it used to stand for Scholastic Aptitude. So there was this idea that was a, it was a kind of intelligence test, but it’s sort of gone away from that idea. It does, what does it actually test? Now, let’s be very clear. It’s in a timed setting. So one thing it does test is speed. They would say processing speed, I would just say the ability to move through things quickly or at a certain speed.
It tests, some math — algebra, basic geometry — not the highest level high school math. Some of the harder questions have some trig and stuff. So it does test math and logic now. It had gone to some verbal math problems, but on the new leaner computer version, that’s decreased. That’s an interesting change because that was a big complaint, rightfully so, for English language learners, like I know how to do math. I just like, don’t understand the paragraph well. So now they’re more pure math problems, solve for X or whatever.
And then the verbal section, there is grammar and composition and then there’s reading comprehension. Okay. There used to be analogies. They got rid of that. So the test has changed a bit over time. For a while, there was a writing component. Now it’s sort of optional. The ACT has a science section, which is more like interpreting graphs. And so that’s interesting. It sort of gives the ACT an interesting little leg up.
It’s being used for ESSA accountability, because –ESSA meaning the Every Student Succeeds Act. Yeah, that’s right. That’s right. It was the reauthorization of Child Left Behind. So. That is what is on it, and so I don’t know exactly what it tests. It does test some reading, it does test some reading comprehension, like find the main idea here, the author means that, and it tests math, some problem-solving in math.
What skills does it test it? Obviously, because it was three hours. Now it’s closer to two, it tests how fast you can do all this. I don’t know that that’s actually the mark of success either in school or in life. There have been studies that tried to sort of correlate SAT scores to future income, but it’s hard to factor out who your parents were and also college graduation. It has some correlation to that, but not as good of a correlation as high school grades. So really, it’s first year grades. That’s the main claim.
[00:08:33] Jon M: I have a couple of questions. When you were talking about the math, I was thinking about all of the fights that are going on about algebra, and who takes it, and when they take it, and so on. So it really does measure, among other things, what opportunities students have had in terms of their school curriculum. If students haven’t had an opportunity to take algebra, or if they took algebra, but later on, so they didn’t get a chance.
[00:08:59] Harry F: That’s less of a problem with the SAT. That problem rears its head in college admissions with the calculus expectancy. Because the problem is if kids don’t take algebra before the ninth grade, then it tends to be difficult for them in the current math sequence to be on track to take calculus before they graduate high school, and lots of schools view calculus as important, particularly for anything in the STEM field, right. Calculus is an indicator of whether you’re ready for college level math. There’s a group called Just Equations. You might want to talk to those people if you’re interested in this topic. They really have focused and drilled down on the inequities in math education and how we do math education. I mean, I could go on about math education. I think the focus in high school on calculus is preposterous. I think if you’re going to be an engineer or maybe even or an economist, maybe you need calculus. And there’s a beauty in calculus and it certainly gives you a certain mind training, but I’d say your average citizen needs to understand probability and statistics a lot more than to understand calculus.
[00:10:10] Jon M: That was less of a problem with the, with the SAT itself.
[00:10:13] Harry F: Yeah, because, because the stuff on the SAT tends to be algebra 1 and geometry and even if. ..Now, there’s always like the quality of the school and the quality of the teaching generally, but pretty much all high school math gets to algebra and geometry if you’re actually doing that.
[00:10:33] Jon M: The other question I had was in terms of reading, I know there’s been a lot of of criticism of many standardized tests on the idea that you’re just getting these little out of context paragraphs. Is that an issue with the SAT? Is it something you address?
[00:10:50] Harry F: What’s interesting now is theclaim they make that it’s better because the passages are shorter. They’re really short now. And there are fewer in the new version, the newest version, in the online version. You barely have to have the stamina to read anything anymore. Maybe that’s a genuine reflection of the requirements of society. If you can post on Twitter and read a Twitter post, you’re good. What that does to our capacity to think deeply and to function as a economically vibrant democracy, I argue that that’s a problem. But even in the old one where the passages were longer, you’re right, Jon. I think they tended to be about sometimes esoteric things. I think that I will say, I think the College Board and the ACT both try to respond to some of those criticisms and have tried to find things that are… I have not in a long time like done an analysis of what’s actually in these passages. It’s interesting. I did do, there’s an article in Mother Jones. I went through and read the Classical Learning Test, which is supposed to be a competitor to the SAT. It’s the one in Florida. It’s supposed to be like, we want kids to read Plato. I wish they read Plato. But the test is still the same thing, like the SAT, except the passage is about, you know, nuns. I think if you went to Catholic school, you probably had a little bit of a leg up on that passage. But there was also a passage from one of the Federalist Papers. I don’t know that the the critique, the SAT has become woke, you’ll get a passage about something written by Toni Morrison. That’s unfair because the classical learning test had Toni Morrison as one of the approved authors. So I think that you’re right that that is an issue that is a constant struggle for the test designers. I think what they’ve done has been now is they’ve basically said, “Oh, those passages were too long anyway.” So now they can’t be more than 50 words or whatever it is.
[00:12:52] Amy H-L: Harry, over several years, many of these colleges went test free or test optional. Which universities took that route and what impact did that have on who got in?
[00:13:05] Harry F: Sure. Well, that’s a big question. So there had been a movement, a small one, probably starting in the ’90s among some small liberal arts places. Bowdoin, for example, was a leading proponent of test optional policies. And then some larger schools like Wake Forest joined. So there were 100 schools from, let’s say, the 90s to the 2010s that looked at what they were getting with the test, saw that the test really was not necessarily getting them anything beyond what looking at grades and other stuff would get them in their class, and was perhaps an inhibitor of creating a socioeconomically and racially diverse class, because the data from the standardized tests show that the one thing that the scores correlate fabulously to is family income. And also there is a big disparate impact in terms of performance by race, with Asians and whites performing on average, several points higher than Blacks and Latinos. And so that’s baked in. There were some studies done years ago that showed that when they picked items. If Blacks and Latinos tended to score better than whites, those test items weren’t put in the test. They would give them in the experimental section. I think they’ve, they’ve seen the light on that particular item response problem. The College Board’s own data sort of shows that, so there were those schools. And so there was like a movement as it were.
Then, 2017, 2018, some– U of Chicago, I think– the Ivy pluses, because that’s who we think of as the way here, even though 90 percent of all schools in America are not particularly selective at all and take most applicants. So we’re really talking about entry into the, my colleague Akilbello coined the term, the “highly rejective schools.” And that gets used, but we won’t poke the bear. I’ll call them selective. . So once U of Chicago went, people started to think, oh, whoa, do we really need, what do these tests really give us? I think U of Chicago had a little bit of the philosophy of what’s a two and a half hour, three hour, one-sitting test going to tell us about a kid’s intellectual capacity or capacity to do this work? It seems like a little strange because it’s certainly not a test of deeper thinking.
So you asked, Jon, what does this test actually test? And that’s why I say, does it really test what you’re supposed to do at a high level college? I don’t think so. It’s just bought because of this correlation that there is data for. We just don’t know why it correlates necessarily. But anyway, getting back to the history of it. And then we have Covid.
Let me take a step back. The University of California system was undergoing an audit to see what these tests actually do to admissions, in part because California had this ban on affirmative action before the U.S. Supreme Court ever stepped into the fray. And the numbers at particularly the upper tier of the UC system, UCLA, Berkeley, and those right below it, Santa Barbara, Irvine, the other, UCs. It’s not what the public university was supposed to be about. It wasn’t taking in the population of California, a representative population. So there was a move to get rid of the SAT. And then COVID hit and nobody could take the SAT, so lots of schools and now, and I will still say, and we’ll talk about the, the revanchists, I will call them, who have gone back to it. Lots of schools said we’re not going to give them because it’s hard for kids to take them. We’re going to use a more holistic picture of a student. And the University of California system, after an internal debate… There was a faction that said, no, we have to have them. They correlate to success in college. And then there was another faction and I would say led by, but the data was provided by, the principal researcher for the UC system, a guy named Saul Geiser. And he said, well, the problem with the study, you’re saying that it, it correlates is it suffers from omitted variable bias. You are actually not accounting for race and income, and when you do that, high school grades are the best predictor of success by a lot, and you don’t need the test. So California then made a decision for egalitarian reasons and equity reasons, but understanding that they weren’t really going to lose anything in terms of quality. And there have been a lot of studies about this that we could talk about later. They said no test at all. Don’t put it on, you put it on, we’re going to reject you if you try to sneak it in. I actually heard the UCLA admissions person say that. So they went test free, the entire UC system, in 2020, 2019, but around COVID, and they are maintaining that policy.
Other schools went test optional. Some of them went for a trial period. Others have declared they’re permanently test-optional. The numbers are over 2,000 undergraduate degree-conferring institutions in the United States, which represents about 90 percent of them, are now test option or test-free. And that data is on our website, fairtest.org. We maintain the go-to test-optional list.
[00:18:41] Jon M: I was just going to say that you started to refer to some of the schools that seem to be reversing course.
[00:18:48] Harry F: We will call them revanchists, yes. The revanchists.
[00:18:50] Jon M: So which schools are doing this and why?
[00:18:53] Harry F: I’ll do it in three tranches. One, some big public southern systems — Georgia, Florida, Tennessee, mostly, although they also have a way you can get into the University of Tennessee with if you’re in the top X percent of your class, so for locals, it had a little bit less impact, at least at that level. But anyway, they basically said a test is a test and we didn’t want to go away from them anyway. This is how we do this in our city. And a lot of scholarships, this is an interesting thing also, are tied to test scores. So Florida has something called, I think it’s Bright Minds, state scholarships, and they’re tied to test scores and they’re supposed to be designed to allow low-income kids and kids of color to attend the major public universities, of course, because they are hooked on test scores. It has the effect of disproportionately rewarding those who need the money less to get in. So that’s and we have something on our website. We have a report about that. Even in places where they made a test optional for entry, these scholarships sometimes in the state system, get tied to test scores. And there are a lot of hidden scholarships that are tied to SAT and ACT scores. Second, is a couple of the big tech schools went back early, MIT, Purdue, Georgia Tech. Caltech has also gone back to requiring tests, but they were test-free for a little while longer. What, and I’ll just use MIT as the example, what they say is, look, we have internal data that tells us that SAT and ACT scores, particularly on the math section, are an indicator of whether kids can handle the work at MIT, that if they get below a 700 or 650 on the math section, our internal data shows that they don’t graduate in as great numbers as those who score above it. That was their rationale. Then the third,, which is the one that gets all the press, thanks to The New York Times is some of the more highly-rejective schools. It started with Dartmouth. I believe. Since then Harvard and Yale and Brown have reinstituted the SAT as a standardized test requirement. Columbia declared itself permanently test-optional. Princeton is still on the fence. Penn is pretty solid in the test-optional camp right now. So even University of Michigan declared itself to be test-optional for the foreseeable future the same day as Yale, but Yale got a lot more press because it went in the direction that The New York Times wanted it to go.
And, you know, you ask why. The stated reason is that they found in their internal data that SAT scores. are a better predictor of college success than grades. They also stated that they need the SAT scores to find poor students and students of color from places that aren’t Dalton, Harvard Westlake, or the the feeder schools to the Ivy League. We can find these kids, find the kid from Biloxi, Mississippi, who gets a 1550 on the SAT. And then we will offer them entry because we know that they can succeed here. And there is a study that was relied upon by a group led by a Harvard economist, Raj Chetty, who does a lot of work, and John Friedman of Brown, who basically said that kids who should have been submitting under a test-optional policy, who got maybe a 1400, but came from an underprivileged Black neighborhood in Dallas or whatever, didn’t submit because they didn’t think it was good enough, but they should have. And they might have gotten in. So that’s the rationale. Now, there are a lot of problems with this, of course, because as soon as you require the test, essentially, you’re going to filter out more kids who just won’t apply.
Jon, you asked what has been the impact of test-optional and test-free policies. Applications have gone up. I don’t know what the exact numbers are in terms of percentages, but I know UCLA, for example, now gets 150,000 applications. Their application pool went up by, at least a third, and the people who are submitting more are lower socioeconomic students, Black and brown folks. It’s a little bit rich because when the Ivy league said, well, we need the test to find the qualified minorities, if you look at their data, you know, when they required the test, their diversity numbers, in the days when affirmative action was allowed, were not so hot. And for the two or three years that they went test-optional, those diversity numbers either stayed the same or got a little better. So I think they speak with forked tongues. What they should say is, look, we are elitist institutions. We have standards. We think the SAT is a marker of intellectual ability of success in academic work, and thus we want to use it. And if you want to come to one of our places, you’ve got to take it. We’ve got to see what you can do. And if they had said that, they would at least have been honest as opposed to cloaking their elitism, for want of a better word, in the patina of equity, because it ain’t.
[00:24:40] Amy H-L: Harry, if our goal is not just more equitable college admissions, but a more ethical and democratic society, what other changes in the admissions process need to be made?
[00:24:52] Harry F: Well, okay, that’s interesting because some of the changes that the Ivies are not doing, even that the Opportunity Insights Report, the Chetty R eport. So the first thing I found was that the Ivy plus schools… and why are we concerned about Ivy plus schools? Because when you look at leadership, they have an [unintelligible]. What is success? Success is working at major firms, working at The New York Times, the Wall Street Journal, the Washington Post. So the measures of success used in the study are absolutely right. Those in the one percent of elite leadership, you know, members of Congress, members of the federal judiciary appeals, are higher. Of course, anybody can become a trial court judge. No, I’m joking. I’m joking. That’s a joke. And, you know, Kinsey, Boston, I don’t know which consulting groups made the cut. Investment banks, obviously, Goldman Sachs, JPMorgan Chase, yada, yada, yada. Those were measures of success. And if you look at that group, they are overwhelmingly members of the Ivy plus schools, maybe plus another 10 or 15, you know, the NYUs of the world, USC. And so that’s your measure of success, and so it is like this sort of massive ranking and sorting.
However, if you look at the socioeconomics of the kids who are in it, they are overwhelmingly from the top 10 percent of the population, wealthy households in America and particularly the top, one percent. If you compare the top one percent, I think, I don’t remember the numbers, but there are as many kids from the top one percent as there are from the bottom 60 percent or something like that. So, if you really want to see how they’re doing on equity, look at how many Pell Grant kids they have. And it ain’t so good. Some schools are making genuine progress in this area, but those policies have nothing to do with testing. It’s about outreach. It’s about giving money. Johns Hopkins is a very interesting example of that. How they’ve basically changed the composition of their incoming classes over the last five to 10 years. It’s purposeful.
Why, why does it look like this? Three big drivers. One, legacy admissions, for obvious reasons. Two, sports scholarships., squash, lacrosse. Both prep school country club sports, for the most part. We’re not talking just about the basketball and the football scholarships here. And those kids are disproportionately wealthy. Because they’ve gone to the camps, they’ve gone to all the stuff. And then the third thing is the padding of the resume with fancy internships and The New York Times program and the semester in Cambridge and all that stuff, right. But of course, the SAT. Anybody can take the SAT. Now, have they gotten rid of legacy admissions or squash? No, no, no, no, no, no, of course not. We need the test back so we can get these kids in. But if they got rid of those other two, there’d be a lot more space for a lot more lower-income kids. So that’s why I say they speak with forked tongue. As far as the country club sports and all the extracurricular opportunities, unlike a standardized test, which the number is the number and it’s baked in. And yes, you can say a 1400 when you’re coming from a poor family in rural Idaho is as least as good an indicator as a 1600 from Scardale. Which, for those of your listeners who don’t know, is a wealthy suburb of New York City.
But the problem then becomes that when Ed Blum brings his discrimination suit and says, look, the only objective measure we have is this test score, and you’ve let in the 1400s over the 1600s. So, I’m not quite sure why the general counsel of these Ivy League schools have let them go down this road. But maybe it’s because they just don’t care. I don’t know. Taking one step back, whereas I can make the argument that I have two kids here, one worked at the 7-Eleven after school for five hours a night and the other got cello lessons from Yo Yo Ma. Now I Ivy League faculty, might have more in common with the cellist who studied with Yo Yo Ma and might find that more interesting, but certainly in the admissions office, I think you can kind of weigh those two things and make a decision about who might need this Yale education a little more, or whose perspective and experience, you talk about democratic society, would benefit from the finishing of Yale into higher leadership in America. Would we best be served by having a broader swath of socioeconomic experience in our leadership class as opposed to recycling the same plutocracy over and over again? And so there’s your answer for democracy, right. Because the policies we get are born of the experience of our leadership class. That’s why this shit’s important. I’m sorry. You know, because it does. It seems, sometimes I think, why do I do this? I mean, it’s a test score, the college admissions, kids go to college, they don’t go to college. As long as the broad macroeconomics and political kind of framework of our society are okay, it doesn’t matter. People find happy lives. But the sad truth is it does impact both how our economy is structured and now increasingly how our democracy is dysfunctional because certainly our politics have been corrupted by money. So, yeah, I think who populates our universities. really matters.
[00:30:52] Jon M: Yeah, that makes an enormous amount of sense. I just have one more question, which is actually going back. One of the arguments that we hear a lot about why you can’t rely on grades being a better indicator is this thing about grade inflation. And I was just curious what your response to that is.
[00:31:12] Harry F: Well, where is the grade inflation happening? The grade inflation is happening at Dalton or Harvard Westlake or whatever because there are all sorts of pressure on the teachers and the counselors to make their kids look good for the colleges. So I would suggest the grade inflation is not happening or is certainly it has a lot of catching up to do at a local public high school in a poor neighborhood. And also, you know, grade inflation just means the margins of comparison are a little tighter. It’s not that there is no differentiation in grade scale. It’s just the differentiation is moved up the chain a little bit. So, what used to be a C is now a B. So I don’t think that that’s a justification. The truth of the matter is, and we have these studies on our website that prove that the data on grades is historically, over decades, been really good as the best indicator. And it makes complete sense because you’re looking at a kid over the course of time, as opposed to over the course of two hours. And over the course of time, you get a sense of kind of not just the kid’s intellect, but the kid’s work habits.
The other thing we could look at, for example, is what they call discrepant scores, which is SATs overperforming grades or SATs underperforming grades. What’s the biggest class of kids where the SATs overperform grades? It’s mostly white guys. Underperforming, it’s women and minorities where the SATs underperform grades. So that tells you something about the test themselves, but it also tells you, study after study going back to the 90s. There’s a book by the former president of Princeton, William Bowen, studies that they did saying, look, grades are the best predictor over time of how kids do in college, because also the predictive power of SATs dissipates over time in college. It’s most predictive of the freshman year. By the senior year, it loses its predictive power, which makes complete sense. Because the kids who maybe learned how to do school right over four years, the kids who didn’t maybe didn’t have the best high school experience.
And what is college? Is it there to just like, educate the anointed or is it there to educate people who are willing to be educated. And I would argue it should be the latter. And if we conceive of society more along the lines of the way we conceived it with the exception of our racial blindness, but the way we conceived it from World War 2 to 1980, maybe, basically trying to make life better for people and to educate more people as opposed to funneling people into to let meritocracy rule over us. We’ve shifted our philosophy and perhaps the curve needs to move back in the other direction. I hope it’s not too late. I mean, I hope we’re not doomed to authoritarian populism. But even in India yesterday, it seems like Modi’s losing his mojo.
[00:34:50] Amy H-L: Well, thank you, Harry Feder of FairTest. We’ll look forward to having you back next time to talk about the ethical implications of high stakes testing in K 12.
[00:35:00] Harry F: Thanks so much. Pleasure to be here.
[00:35:02] Jon M: And thank you, listeners. Check out our new video series, “What Would You Do?,” a collaboration with the Harvard Graduate School of Education and EdEthics. Go to our website, ethicalschools.org, and click video. The goal of the series is not to provide right answers, but to illustrate a variety of ethical viewpoints.
If you found this podcast worthwhile, please share it with a friend or colleague. Subscribe wherever you get your podcast and give us a rating or review, because this helps others to find the show. Check out our website for more episodes and articles, and to subscribe to our monthly emails. We post annotated transcripts of our interviews to make them easy to use in workshops or classes. Contact us at [email protected]. We’re on Facebook, Instagram, Threads, and YouTube. Our editor and social media manager is Amanda Denti. Until next week.