Standardized Tests are Maybe Not the Worst? Inevitable @

[wawa]

I no longer have a NYT subscription, but I still get their email morning newsletter and today it was talking about this piece on the SAT's and how maybe they aren't biased in the ways we thought they were. 


www.nytimes.com/2024/01/07/briefing/the-misguided-war-on-the-sat.html

The gist being that somebody with a ton of potential to do well in a typical college academic environment but who's coming from a disadvantaged background might stand out on a test score in a way that they can't on extracurriculars, teacher recommendations, essays, etc compared to somebody who just has lots of money and hyper involved parents paying for help.  From the data they looked at, SAT/ACT scores are more predictive of college success measured by GPA than high school grades. 

It included these graphs - which are pretty stark.  


Thoughts?  Somebody with a subscription lemme know if the rest of the article has anything fascinating.

For the inevitable personal anecdote - I'd say that I didn't stand out in my college applications on many other factors - normal not fancy high school, I took some AP's but didn't actually do amazing on the tests, my GPA was good but not great, I was in a few random clubs but not sports or music, etc etc - but my SAT's were high and I always assumed that was the reason I actually got into my first choice.  It's the only thing that nudged me into the top quartile in the general crowd of generally smart kids with goodish grades.  But just looked it up - LOLOL current admissions numbers I'd be in the bottom 25th percentile apparently.  jesus.  The year I applied I was 75th percentile on SAT score.  Which isn't relevant, I'm just aghast every time I look at numbers like that.

[pixy0stix]

Gift link - The Misguided War on the SAT www.nytimes.com/2024/01/07/briefing/the-misguided-war-on-the-sat.html?unlocked_article_code=1.ME0.mpMC.4qkV-2t-Vfox&smid=nytcore-android-share

[glw]

Predicting student success in college is a major part of my job. I use both test scores and previous GPAs as part of regression formulas and propensity score analysis, and in my experience both are good predicters of future success. "Success" here means retention and graduation, not learning-- we do that through a separate learning outcomes assessment process-- and there is no transfer of direct measures of learning from one school to the next.* Total grades and test scores are not direct measures of learning but are good indicators of "success" because they are also based on things like, "will this person show up, will they study, will they take tests well?" (and also potentially, "will they cheat or pay someone to take the test for them," which is a whoooole other issue)... We use lots of other things in admissions decisions, too, because "building a class" involves more than just picking who we think will be retained/graduate. Diversity (not just racial but many factors) plays a big part in who is admitted, especially to small specialty programs, and some schools consider legacy status, sports, etc. as well.

*ETA: some schools, mine included, are trying to rectify this by encouraging students to use Comprehensive Learner Record systems, but I don't know if that's really taken off.

[VillainV]

How we define “success” matters. If we only define it in terms of having a high university GPA or going on to a prestigious graduate program, then in makes sense that standardized tests would be predictive of that. But are those the right metrics for success?? If someone graduates in 5 years instead of 4, with a 2.5 GPA, and goes on to found a non profit that helps launch minority-owned businesses, isn’t that successful?

[Jalapeñomel]

I think that grade inflation and HS GPA is a huge problem, but I'm not sure standardized testing is the answer.

[Jalapeñomel]

Also, those graphs in the OP have the title, Attending an elite graduate school (and the second one is working at a prestigious firm), as if that is the only predictor of student outcomes after graduating.

This is infuriating...clearly, people are successful without going to some prestigious law firm or graduate school.

EDIT: I see I was mistaken...these are separate graphs, but still! 

[basilosaurus]

Jan 8, 2024 12:00:39 GMT -5 VillainV said:

How we define “success” matters. If we only define it in terms of having a high university GPA or going on to a prestigious graduate program, then in makes sense that standardized tests would be predictive of that. But are those the right metrics for success?? If someone graduates in 5 years instead of 4, with a 2.5 GPA, and goes on to found a non profit that helps launch minority-owned businesses, isn’t that successful?

I'm weirdly skilled at standardized tests, so act, sat, ap were very good, got me into my first choice.

And yet I barely graduated my first degree. I say to this day my college tuition was about getting me the years of therapy I had missed out on that totally caught up with me.

I wouldn't call myself successful in an "typical" sense. Too much moving, career changing, trailing spouse for that, but contentment, experiences, and peace, and I'll take that. I'm not changing the world whole scale, I'm not making huge bucks, but most demons are gone, and I'm adaptable in ways I never imagined.

I just giggled half the night with a friend, like we were high school girls with a passed joint (we weren't smoking but there was rose prosecco). It was something I never experienced then and never could have anticipated, and that's more meaningful than any of these "successful" metrics. Failure of potential my ass.

I get metrics and the reasons for them, and I get why that's in part why schools have leaned towards eschewing them for other stuff, but, yeah, so much of that is purchased.

I don't know if I'm making any sense. It's something that before this I've been thinking about a lot recently, how not to let others' definition of success define mine.

[smock]

Here's my problem: you can't criticize a "misguided attack on the SAT" by only using data from the most selective colleges in the country. Maybe test scores are a better predictor of college GPA at Brown, but maybe not at large public universities. I'd love to see data from the UC schools, which do not allow applicants to submit test scores.

Also, most colleges that have changed their policies are test-optional. How is that an "attack" on the SAT? The disadvantaged students with stellar test scores can still submit their test scores when applying to Ivy League schools. So who does a test-optional policy hurt?

And what about the time, effort, stress, and money that standardized tests require? Some students absolutely have more resources to devote to test prep, which is ignored in the article.

I'm not a huge fan of standardized tests and I think test-optional is better than test required.

[loira]

I didn't have to take the SAT and I'm so glad. American TV shows made it sound like an incredible amount of unnecessary extra pressure. Same with having to interview to get into a university. The whole US college admission process is bonkers to me.

[glw]

Jan 8, 2024 13:58:56 GMT -5 smock said:

Here's my problem: you can't criticize a "misguided attack on the SAT" by only using data from the most selective colleges in the country. Maybe test scores are a better predictor of college GPA at Brown, but maybe not at large public universities. I'd love to see data from the UC schools, which do not allow applicants to submit test scores.

You mean the raw data? This is the UC report linked in the article:  senate.universityofcalifornia.edu/_files/underreview/sttf-report.pdf

"a faculty committee at the University of California system — led by Dr. Henry Sánchez, a pathologist, and Eddie Comeaux, a professor of education — concluded in 2020 that test scores were better than high school grades at predicting student success in the system’s nine colleges, where more than 230,000 undergraduates are enrolled. The relative advantage of test scores has grown over time, the committee found."

I don't have strong personal feeling either way (we are test-optional still and I can see both arguments), but I do agree with the article that the "attack" on test scores is part of the larger conservative anti-intellectualism movement.

[sapenguin]

Lurker here. When I was in high school (late 90s) I had a terrible home life. I was hungry every day, worried about my abusive parents, an emotional wreck. My teachers did not like me because I never did my homework, I never ever raised my hand (I was terrified), and I did not participate in extracurriculars. Honestly, the thing that saved my life was my near perfect SAT score. I only applied to one college (had no idea it was a fairly prestigious state university for technical majors), and got in based on my SAT score and am now pretty successful.

I am 100% on board with increasing socioeconomic and racial diversity in colleges - it's incredibly important to me. But as I've seen these debates about reducing reliance on test scores, it's been nagging at me that people like me will get left behind. I understand I'm an anecdote, but it's nice to see the other side of this being studied.

[glw]

Jan 8, 2024 13:58:56 GMT -5 smock said:

Also, most colleges that have changed their policies are test-optional. How is that an "attack" on the SAT? The disadvantaged students with stellar test scores can still submit their test scores when applying to Ivy League schools. So who does a test-optional policy hurt?

Sorry to double quote you--
The article is saying that by standardized tests being labelled as "racially biased" in public discourse (particularly by liberal-leaning media) that it's an "attack" that is not backed by the research. That results in fewer students taking the tests which may ultimately come back to hurt them as disadvantaged students could find that their test scores are the thing that get them into the top colleges.

[smock]

Jan 8, 2024 14:10:27 GMT -5 glw said:

Jan 8, 2024 13:58:56 GMT -5 smock said:

Here's my problem: you can't criticize a "misguided attack on the SAT" by only using data from the most selective colleges in the country. Maybe test scores are a better predictor of college GPA at Brown, but maybe not at large public universities. I'd love to see data from the UC schools, which do not allow applicants to submit test scores.

You mean the raw data? This is the UC report linked in the article:  senate.universityofcalifornia.edu/_files/underreview/sttf-report.pdf

Love how CEP'ers always come with the sources! I was actually thinking about the data in a couple years, when the UC schools will be able to research the impact of banning standardized test scores in admissions on the outcomes for graduates at UC schools. 

[smock]

Jan 8, 2024 14:40:55 GMT -5 glw said:

Jan 8, 2024 13:58:56 GMT -5 smock said:

Also, most colleges that have changed their policies are test-optional. How is that an "attack" on the SAT? The disadvantaged students with stellar test scores can still submit their test scores when applying to Ivy League schools. So who does a test-optional policy hurt?

Sorry to double quote you--
The article is saying that by standardized tests being labelled as "racially biased" in public discourse (particularly by liberal-leaning media) that it's an "attack" that is not backed by the research. That results in fewer students taking the tests which may ultimately come back to hurt them as disadvantaged students could find that their test scores are the thing that get them into the top colleges.

I hear that, but I don't think the article proves that standardized tests aren't biased. (But maybe I'm the liberal media, too?) It's a very selective use of data. And if we want to focus on the most selective schools, it's a bigger problem that disadvantaged students don't apply in the first place because of sticker shock. 

Over-relying on any one metric is obviously not ideal for higher ed or applicants. The SAT and ACT (and other standardized tests) market themselves as the best predictor of success (and test makers reverse-engineer their tests with that in mind because it's good for business). But building an entire industry around making college applicants take more and more tests is not great, IMO. It adds another hoop that students need to jump through, and that alone makes higher ed less accessible. 

Also, glw, I'm curious to hear more about the Comprehensive Learner Record systems you mentioned. The SAT is popular because it's an effective way to sort students. Before the SAT (like very early 20th century) most colleges had their own entrance exams. And a LOT fewer applicants. So is there a better option than the SAT/ACT for students and colleges?

[glw]

Jan 8, 2024 15:10:48 GMT -5 smock said:

Also, glw , I'm curious to hear more about the Comprehensive Learner Record systems you mentioned. The SAT is popular because it's an effective way to sort students. Before the SAT (like very early 20th century) most colleges had their own entrance exams. And a LOT fewer applicants. So is there a better option than the SAT/ACT for students and colleges?

I think just about everyone (except maybe people who work in marketing for College Board lol) would say YES, but it comes at the expense of, well, actual expense, and also the burden that applicants would have to take many more tests if they want to apply to multiple schools. But, the results would be so much more accurate and useful, IMO. The downside and the plus side of standardized test is that they are standardized. Ideally each test would be measuring if the applicant is prepared to start at that school. There's a pretty standard expectation of general ed classes at 4-years but that doesn't translate to community colleges or specialized schools and not all -year schools have required gen eds (my undergrad did not). So we all default to the SAT/ACT and no one is happy (same with GMAT/GRE in grad schools).

The CLR tool we use is AEFIS. Basically students can select "artifacts" that demonstrate their learning that maps to competencies from courses and also co-curricular activities, including soft-skills. It's like a CV plus evidence. My experience so far is that undergrads like it as a prep for grad school applications, but it doesn't seem like grad students are using it as a job application tool.

[basilosaurus]

I recently was reading about the scantron getting in the way out after so many years of supremacy. It exploded at a time when, like glw, there were more and more applicants, so there was a nice to standardize.

Certainly a lot can be and are replaced with simple computer tests that are still multiple choice format.

But it was also making the case that more sophisticated ai grading might see a large shift in testing format and styles.

www.theatlantic.com/technology/archive/2023/09/schools-standardized-testing-assessment-ai-future/675366/

I'm way too sleep deprived to make any sort of case for if and or how these might influence each other, but it's what I was thinking about as I read the op.

[wawa]

Jan 8, 2024 12:00:39 GMT -5 VillainV said:

How we define “success” matters. If we only define it in terms of having a high university GPA or going on to a prestigious graduate program, then in makes sense that standardized tests would be predictive of that. But are those the right metrics for success?? If someone graduates in 5 years instead of 4, with a 2.5 GPA, and goes on to found a non profit that helps launch minority-owned businesses, isn’t that successful?

this is a point.  I worded this part of my post: "potential to do well in a typical college academic environment" pretty specifically because what we're measuring as success in these numbers is people's ability to get good grades in college. Which is only of limited meaning, but fundamentally colleges are trying to ID the people who can be there, and do well and graduate while ending up with a diverse pool of people hopefully so....not a useless metric by any means for their purposes.

and the squishier you make the metrics of "success" the harder it is to study, so...I'm willing to consider as a useful metric College Grades when we talk about "success at college".  There's never going to be clean data on "long term contentment with life options and trajectory." or "societal worthiness factor" as a success metric.  

Like - how do you do a regression on that? How do you rank it?  who's more successful - the finance bro who makes six figures out of the gate and burns through his second marriage shortly after making partner or the person who founds the transformational nonprofit housing veterans or the person who graduates and gets a job in their field making enough money to pay their bills and when asked is like, "yeah, my life is pretty good" and actually means it?

[glw]

wawa, the next time someone asks me what I do for a living I'm going to show them your post as an example LOL

[wawa]

Jan 8, 2024 15:10:48 GMT -5 smock said:

Jan 8, 2024 14:40:55 GMT -5 glw said:

Sorry to double quote you--
The article is saying that by standardized tests being labelled as "racially biased" in public discourse (particularly by liberal-leaning media) that it's an "attack" that is not backed by the research. That results in fewer students taking the tests which may ultimately come back to hurt them as disadvantaged students could find that their test scores are the thing that get them into the top colleges.

I hear that, but I don't think the article proves that standardized tests aren't biased. (But maybe I'm the liberal media, too?) It's a very selective use of data. And if we want to focus on the most selective schools, it's a bigger problem that disadvantaged students don't apply in the first place because of sticker shock. 

Over-relying on any one metric is obviously not ideal for higher ed or applicants. The SAT and ACT (and other standardized tests) market themselves as the best predictor of success (and test makers reverse-engineer their tests with that in mind because it's good for business). But building an entire industry around making college applicants take more and more tests is not great, IMO. It adds another hoop that students need to jump through, and that alone makes higher ed less accessible. 

Also, glw , I'm curious to hear more about the Comprehensive Learner Record systems you mentioned. The SAT is popular because it's an effective way to sort students. Before the SAT (like very early 20th century) most colleges had their own entrance exams. And a LOT fewer applicants. So is there a better option than the SAT/ACT for students and colleges?

You ARE the liberal media!!  [insert pointing monkey gif here]

I don't think the article is setting out to prove that the tests aren't biased...I kinda took this to be the "thesis statement" of the article if you will (you can tell I scored well on my exams): 

If test scores are used as one factor among others — and if colleges give applicants credit for having overcome adversity — the SAT and ACT can help create diverse classes of highly talented students.

Those "ifs" are really doing some heavy lifting there, but I think they have made the argument successfully that based on the data, and given the realities of the academic environment of most schools, it's worth taking another look at whether schools are actually accomplishing their goals by eliminating test scores from admissions. The tests are easier for those who are white, have more money, and/or grew up in places where their neighbors have more money. That's a given and this article doesn't argue otherwise. But within that truth they can still be a useful tool.

I think it probably says something not really flattering about what college is actually like and what it actually accomplishes that bullshit test scores ARE a good predictor for being able to "do well" college.  But like....an academic environment is an academic environment, and being well drilled on the fundamentals of a high school education and testing well on them is pretty damn vital to succeed in that environment.  It doesn't make you a less worthy human if you're going to struggle to pass classes as a college freshman in a big lecture hall, but it does mean the college would like to admit fewer people like you and more who will be comfortable there.  That's just the nature of the beast. Whether that's like...a good thing for society is a WHOLE ASS OTHER conversation. 

Unrelated Question-   What are you referring to with the more and more tests?  this isnt' something I follow closely. Are there other tests being added to the SAT/ACT world?  Like...there were the subject tests for a while. and the essay they added after my time, but then they dropped both of those.  Is it not still those two tests, and which you take is largely regional? were people taking both?  is there something else giong on? Were there other tests popping up before 2020 reset everything?


wildly unrelated side note now that I've read the gift link (thanks pixy0stix!)  University of Chicago?  I was aware of it's existence since I dont live under a rock, but had no idea it was considered to be like one of THE top schools. (i.e. Ivy+)  Just checked and it appears they do not offer a degree in my field, so I guess that's why I'm not familiar, but wow I didn't realize I was so biased.

[wawa]

I always find statements like this wild:

“When you don’t have test scores, the students who suffer most are those with high grades at relatively unknown high schools, the kind that rarely send kids to the Ivy League,” Deming, a Harvard economist, said. “The SAT is their lifeline.”

You mean like....like, every single person out there attending public schools outside of fancy public magnets???

It cracks me up.  How many "relatively known" high schools ARE THERE even?  

[minzy]

I found this morning’s NYT newsletter blurb about this pretty interesting, as I just finished reading Who Gets In And Why - A Year Inside College Admissions by Jeffrey Selingo. I’d recommend it for anyone who is worried about or wants to know more about college admissions. A friend with a HS senior read it, and although we are still years away from college, I found it interesting.

The author talks about grades (some are adjusted to the university’s own rating scale for easier comparison), and how admissions officers like to see consistently high grades or an upwards trend. They get concerned when grades drop or are spiky. He talks about the standardized tests and how they aren’t perfect but give a different data point on an applicant. I found it interesting that the tiniest thing can separate an acceptance and denial, and it often has nothing to do with the kid. It can depend on the timing of the application (early decision vs regular cycle), geographic location of the applicant, ability to pay, the school’s relationship with the high school, the “shape” of the rest of the class and what holes they want to fill, etc. He also talks about the business side of this, with recruitment and marketing and prospective student engagement, and how only certain parts of the budget are available for scholarships (or basically, what he thinks of as coupons).

(For anyone wanting the TLDR version, the best pieces of advice he offers are that if you want to go to a “buyer” versus a “seller” school, it is particularly helpful to show proven commitment and leadership in an activity/ies rather than just membership, and to show demonstrated interest in an area, like if you want to major in physics, you should not take it easy in math. Despite lots of specific deep dives, his main point is that most kids who want to go will find a good-for-them place to be, even if it’s not an absolutely so-called elite institution that was initially their #1.)
bookshop.org/p/books/who-gets-in-and-why-a-year-inside-college-admissions-jeffrey-selingo/13446573?ean=9781982116293&gclid=CjwKCAiA1-6sBhAoEiwArqlGPuAEb0GoELhCpIo364MeID0cmeoy1dfZ6ZtJ20Y4xgjmdbCDSbrjRxoCzQcQAvD_BwE

[Jalapeñomel]

Jan 8, 2024 17:05:06 GMT -5 wawa said:

I always find statements like this wild:

“When you don’t have test scores, the students who suffer most are those with high grades at relatively unknown high schools, the kind that rarely send kids to the Ivy League,” Deming, a Harvard economist, said. “The SAT is their lifeline.”

You mean like....like, every single person out there attending public schools outside of fancy public magnets???

It cracks me up.  How many "relatively known" high schools ARE THERE even?  

Also, how are these kids suffering? They're suffering, because they don't go to Harvard and have to settle for a state school?

[macmars45]

Honestly going to community college first saved me. You don't need to take the ACT or SAT to get into community college. Just the placement test.

[devonpow]

I don’t know about racial bias but the SAT, anecdotally, is definitely economically biased. I taught SAT prep classes for a large company 20 years ago and it was all strategy -just teaching strategy I could raise a students score 100+ points (note: I only taught verbal). But those classes were thousands of dollars- we mostly held them on elite high school campuses (private and public). I now have a high school junior going through the college admissions process - and yes, I’m paying for SAT tutoring for him as we speak. It’s crazy expensive. We live in a “high achieving” district and the college pressure here is crazy - not just SATs but getting into certain summer programs and trips (that you need to pay a lot for), paying for private college counselors etc. All these things that advantage the wealthy - the SAT is just a part of it. I know everyone knows most of this already and I don’t know the solution. I just know going through it right now as a parent is a crazy experience.

[taratru]

I'm currently working on an article reviewing this topic and the implications for moving forward. There is no good answer, truly, and no shock, any way you dice it, the rich elite kids come out on top with more advantages. It's depressing.

[taratru]

Jan 9, 2024 7:34:07 GMT -5 Jalapeñomel said:

Jan 8, 2024 17:05:06 GMT -5 wawa said:

I always find statements like this wild:

You mean like....like, every single person out there attending public schools outside of fancy public magnets???

It cracks me up.  How many "relatively known" high schools ARE THERE even?  

Also, how are these kids suffering? They're suffering, because they don't go to Harvard and have to settle for a state school?

I can verify that yes, for some parents absolutely believe this.

[formerlyak]

Jan 9, 2024 12:26:28 GMT -5 taratru said:

Jan 9, 2024 7:34:07 GMT -5 Jalapeñomel said:

Also, how are these kids suffering? They're suffering, because they don't go to Harvard and have to settle for a state school?

I can verify that yes, for some parents absolutely believe this.

It's not just parents. DS is a very high achieving student. I worked in higher ed for 20 years so I know that there are things beyond a student's control that go into admissions. We have been discussing this for years and when he made his college list this year, he was sure to include some programs that would be seen as "less than" by the same people that think their kid with similar stats to my kid are entitled to a spot at an elite college. But they are programs he really likes so luckily he doesn't care. But the number of his classmates who refuse to apply to certain schools because they "worked hard and deserve to go someplace better" is crazy to me. 

[Jalapeñomel]

Jan 9, 2024 12:26:28 GMT -5 taratru said:

Jan 9, 2024 7:34:07 GMT -5 Jalapeñomel said:

Also, how are these kids suffering? They're suffering, because they don't go to Harvard and have to settle for a state school?

I can verify that yes, for some parents absolutely believe this.

Oh I know they believe this. I teach AP chem. I see it every.single day, from parents, from kids, from admin, and sadly, from other teachers.

[formerlyak]

Jan 9, 2024 16:51:51 GMT -5 Jalapeñomel said:

Jan 9, 2024 12:26:28 GMT -5 taratru said:

I can verify that yes, for some parents absolutely believe this.

Oh I know they believe this. I teach AP chem. I see it every.single day, from parents, from kids, from admin, and sadly, from other teachers.

Side note - you are a saint for teaching AP Chem. DS took it last year and it was hands down the hardest class he’s ever taken. And he’s in AP Calc BC now which is also hard, but not AP Chem hard. The number of meltdowns I saw from 16 year olds over that class was … lots.