You Can and Should Do Better, Faculty Members - A Commentary on the Reinstated SAT/ACT Requirement for Undergraduate Admissions

Written by: A friend of the students

Editors’ note: The author requested that this piece be published anonymously, for reasons described within. Given the sensitivity of the topic and the nature of the views expressed, we determined this to be acceptable in line with the Tech’s Journalistic Principles.

Undergrads, you might have noticed in the past edition of The California Tech a small section on page 2 that the Faculty Board had passed a resolution reinstating the SAT/ACT requirement. Or in the February 27th edition, an article titled “Faculty Petition Speaks to Broader Implications for Undergraduate Admissions” describing a petition circulating amongst the faculty that the Tech editors report they were repeatedly denied a copy of. I would like to make public the text of the petition. You can find it in full below. (jump to petition)

I expected better from the members of the faculty who signed this petition, who not only ignored the truth of the matter while writing this petition, but repeatedly denied current students access to it. In addition to the Tech editors being denied a copy of the petition, I have been told that members of ASCIT have been repeatedly uninvited from the monthly Faculty Board meetings where discussion of this is taking place and that undergraduate student leaders’ direct lines of access to administrators and faculty members on campus have been uniformly shut off to any information about this petition. (If you don’t believe my word on this, feel free to contact the current student leaders, whom I am certain would be happy to explain their side of the story better than I can). I also find myself disappointed in the Faculty Board, whose committee to collect and present evidence on this topic attempted to find a correlation between SAT math scores and first-year (shadow) grades. Attempting to use first-year (shadow) grades when those students are told to focus on their transition to college and not worry about grades, rather than say sophomore year grades or cumulative GPA, seems misguided to say the least. Perhaps more worryingly, despite this lack of conclusive evidence, the Faculty Board passed the resolution reinstating the SAT/ACT requirement almost unanimously.

As a University whose motto is, “The Truth Shall Make You Free”, in what sense do the actions of the past few months live up to this? In what world does hiding the evidence and covering up the dialogue about current (and future) students help you, or them? Would it have been impossible for you to share your concerns with them directly as I have, instead of gossiping about them behind their backs? How does casting off entire classes of current students as effectively ’too dumb to succeed at Caltech’ encourage them to make up lost ground and perform better in your classes? So students, in the spirit of our shared motto, I believe you should have access to this petition, even though you may find the text itself and the commentary surrounding it painful to hear. You should also know that not all members of the faculty believe in the spirit behind this petition, and hopefully you will be heartened to know that many refused to sign it.

I am choosing not to publish the names of the 150+ faculty members who signed this petition for a few reasons. First, I do not want this article to be used as a personal attack on any one person who signed the petition, but rather as an inspiration for reflection on the state of our campus community. Second, public comments attached to the petition reflect that some members of the faculty only signed the petition to encourage the Faculty Board to investigate the correlation between SAT/ACT scores and academic performance at Caltech, as a request for more evidence on this topic, without buying into the rest of the discourse surrounding this. While I disagree with their decision to sign this flawed document, I do not think that publishing their names alongside those who are fully buying into the idea that current students are fundamentally un- or under-prepared for Caltech academics will be productive.

I publish this anonymously for fear that this discussion will hurt my career and threaten my current employment. For a community that prides itself on freedom of intellectual thought and objective review of the available data, it is shameful that we cannot have an open dialogue about this. Faculty members, you can and should do better. I urge you to engage with and discuss this with the undergraduate community in a meaningful way, instead of continuing to deny them a seat at the table.

The text of the petition is printed on the opposite page of this issue of the Tech.

In addition to my rebuke of the actions taken by some faculty members in the past few months, I would like to provide a critical analysis of the petition and some context that it is lacking.

First, the data from two electrical engineering courses (EE 44 and EE 55) are not representative of the entire student body, and certainly faculty members who pride themselves on their ability to carefully analyze data in their professional capacity should know better than to take a non-representative sample as proof of anything.

Second, even if you did take the data of the two electrical engineering courses to be representative of the student body as a whole, the analysis does not take into account that each year of students has had substantially different high school and core curriculum experiences. The core curriculum has undergone substantial revision between 2019 and now, and it is not inconceivable that different teaching styles and curricula for math and physics core courses could have impacted scores in EE 44 and EE 55. [I might even suggest the Faculty Board investigate how well the core curriculum has prepared students for sophomore year courses and beyond during each iteration of the past few years]. Additionally, while the current undergraduate seniors had only their final few months of high school online, and presumably took calculus and other math and physics courses in person, a substantial portion of the current undergraduate juniors, sophomores, and first-year students took calculus, trigonometry, and physics online during the COVID-19 pandemic. Numerous recent studies [1-4] have shown that students perform poorly on objective high school math standards when taking courses online compared to in person, despite their grades in those courses being comparable. I believe that the undergraduate Academics and Research Committee (ARC) went so far as to collect and present data on this very topic at last year’s Student Faculty Conference and presented this data again at a Faculty Board meeting last spring. In conversations with undergraduate student leadership in the past few months, they have lamented the fact that the Faculty Board has not only failed to take these data into account when making decisions about current and future students, but has also denied them a seat at the table where they could have brought this up directly. Members of the faculty should know better than to conclude that this change in a non-representative sample was caused by the lack of SAT/ACT requirement, when differential math and physics preparation due to a worldwide pandemic could just as easily explain the effect.

Third, the brief paragraph within the petition on the COVID-19 pandemic and its effects on academics fails to take into account the reality of the situation. More than a few students lost parents and other close family members, lost regular access to school-provided meals, and lost access to academic support and extracurricular opportunities during the pandemic. To expect them to master calculus and other math topics that are tested by the electrical engineering “basic math test” during such a tumultuous period is almost absurd. Furthermore, the SAT and ACT do not test calculus or physics topics and thus are not indicators of whether students have mastered these topics. Additionally, AP tests were administered online in a shortened format over the pandemic and the reports from both students and high school teachers from that period indicate that they were not representative of students’ true grasp of these concepts. Moreover, widespread high school grade inflation, especially during periods of online learning, makes it almost impossible for the admissions office to discern which students have actually mastered calculus and other math topics based on their transcripts. As none of these metrics can serve as appropriate measures of student learning over the pandemic, it would be worthwhile for the members of the faculty who signed this petition to take this into account when suggesting that the lack of the SAT/ACT requirement is the sole reason the admissions office is admitting “D & F students” instead of “A & B students”.

While there are many more points I could make about the contents of the petition, hopefully I have demonstrated that it is poorly thought through. The public comments attached to this petition clearly show that many members of the faculty believe that the solution to this “problem” of students un- and under-prepared for their courses is to admit “better” students in future years. What, I ask, is your plan to support the current students whom you have an obligation to teach, to the best of your ability, right now? It is easy to point fingers at the admissions office and at students. It takes much more initiative to help the students you believe are not ready for your courses to succeed, or as an Institute, to not only notice there is a “problem” but actually help resolve it. Members of the faculty, I implore you to do better.

[1] American Economic Association, 2023. The importance of in-person schooling.

[2] Binkley 2022. Associated Press. COVID grads face college

[3] Locke et al. 2021. Learning loss in reading and math in U.S. schools due to the COVID-19 pandemic

[4] National Assessment of Educational Progress 2022

Editors’ Note on the faculty petition below: This petition was written several months ago (dated January 16) and may not reflect anyone’s current views or facts. It is provided only for context to this article. It was also written for an audience of only the President, Admissions, and other faculty — i.e., not the broader Caltech community. It has not been edited or abridged in any way, except as noted in the article.

---

Faculty petition

January 16, 2024

Dear faculty colleagues:

Over the past few years, faculty colleagues across campus have noticed and commented on a sharp decline in the quantitative skills of our undergraduate students. In particular, although many of our undergraduates are of the same caliber as in the past, there has been a concerning drop in preparedness at the low end of the distribution. This decline has worsened with recent changes in our admissions practices, and is particularly acute for the current sophomore class. An inordinate number of students are failing courses, honor code violations are on the rise, and requests for tutors and extensions have substantially increased. Some faculty report having to adjust grading practices, as well as course content, to the change in student population.

We fear that this decline will have disastrous consequences for our students’ training and career outcomes, for Caltech’s educational mission, and for Caltech’s reputation at large.

The goal of this letter is to initiate discussion and action on this critical and urgent matter.

Below we consider possible causes for the decline. Based on these reasons, we believe that the problem requires both immediate action as well as longer term improvement and monitoring in admissions practices.

In the immediate term, we ask the institute to:

• Reinstate the SAT/ACT as an admissions requirement for the next cycle. This should be announced in March 2024, so students can start testing in the spring, preparing for applications in the Fall.
• Form a faculty-led committee to study the effectiveness of current admissions practices on student outcomes and to make recommendations about how to improve the process. Such a longitudinal study was promised to the faculty (see Faculty Board meeting of 6/7/2021), but no such report has been released. This committee should report its findings sometime in Fall 2024 so that it can help shape next year’s admission cycle.

In the longer term, we ask the institute to:

• Establish a faculty-led standing committee whose charge is to regularly gather data on student learning outcomes and use them to evaluate and guide our admissions processes. This is important because so far our admission policies have not been informed by this essential exercise, in contrast to peer institutions like MIT.

Why do student STEM skills matter?

Many of us are committed to Caltech because of its unique place in the higher education landscape, as reflected by the “There is only one Caltech” campaign motto. We view our educational mission as recruiting, educating, motivating, and empowering the next generation of top scientific, engineering and mathematical talent. Our comparative social contribution is to provide a niche for individuals with an extreme passion and talent for these fields. We give those students a protected environment to develop their talent and passion at the highest levels of science and engineering. Then they go and establish the semiconductor industry or find a cure for AIDS. Historically, Caltech has produced one of the highest rates of future STEM PhDs and the highest rate of Nobel laureates. If we give up on the goal of educating students with this unusual intensity and talent, then we lose our raison d’etre, our unique and essential educational contribution to society.

As faculty, we also need to acknowledge the limits to what we can do. The historical greatness of our undergraduates has been largely due to them, not to the faculty. To train top-flight scientists and engineers we have to start with top-flight high school graduates. Our skill is in designing a curriculum of courses and research that challenges these students beyond their comfort zone. But we have no special skills that would bring the median high-school graduate to that level. There is nothing magical about Caltech that turns someone into a successful scientist just because they spent 4 years here. Furthermore, unprepared students struggle here even though they would have thrived at other top schools like Stanford or Harvard. This is why the STEM skills of our entering first-years continues to be crucial to Caltech’s unique educational mission.

Two examples: Student performance in EE44 and EE55

The drop in STEM skills has been observed by many faculty who teach first-year and sophomore courses.

A concrete example is provided by Ali Hajimiri, who analyzed grades in EE44 (Deterministic Analysis of Systems and Circuits). EE44 is the introductory circuits course taken by all EE sophomores, and it uses basic complex number, linear algebra, and calculus concepts. Ali has taught EE44 continuously for the past 12 years. Each year, he administers a basic math test on day 1 to get a baseline on the students’ math competency. He also administers a midterm and final exam. This fall, he reused the 2020 final exam to create a control comparison.

Consider the scatter plots below, which show the relationship between the score in the initial math Quiz 0 and the midterm exam (red dots) and final exam (blue dots). Each dot represents one student. There is a stark difference between the grades of 2020 sophomores (left plot) and 2023 sophomores (right plot). Whereas the top of the class in 2023 (green ellipse) looks similar to the entirety of the class in 2020, the class of 2023 has a sizable cluster of students (the red ellipse) that did not exist in 2020 and who enter the class with weak math foundations and in turn performed poorly in the course.

Another data set is from the EE55 class (Mathematics of EE) taught in alternative years by Victoria Kostina. This data compares the final exam scores of the students taking the exam in 2021 versus. those taking it in the fall of 2023. It again shows a noticeable drop in the performance of the class.

Although this is data from only two courses, it is consistent with the classroom experiences of many other faculty at Caltech. If, as we suspect, the data from other classes at Caltech match these observations, then we are facing a major challenge to our educational missions that requires urgent action. First, a substantial fraction of the current Caltech student population is not well matched to our educational program and not served well as a result. Second, the experience of all students is impacted, for example, by lowering the level of our course offerings. Third, our reputation, and thus our long-term ability to attract Caltech-caliber, students are at risk. Eventually, this could affect recruitment of graduate students and faculty as well.

Decline of Caltech’s performance in prestigious student competitions

Historically, our students have had an outsized presence at the Putnam math competition, with multiple Putnam Fellows (top 6 finishers), and topping the competition more than any school other than Harvard and MIT. But since 2010 there has been a steady decline in Caltech’s showing. Over the past few years, Caltech’s performance fell precipitously: since 2019 we have had zero students in the top 100. This is distressing for a school that touts itself on being a destination for top STEM talent. MIT, on the other hand, is sweeping the top spots.

A similar decline relative to other universities has been seen in coding competitions, such as the ACM-ICPC, where in the past few years Caltech has even failed to qualify for the international competition (before that it was a contender for the top spots).

While we are not suggesting all Caltech students should be top math or coding competitors, our performance in these competitions provides an informative signal about the quality of our student population, and gives us visibility to help attract top high school talent.

Potential causes for the decline in student STEM skills

Several hypothetical causes for the drop have been proposed. We hope that the faculty-led committee that we propose will carry out an immediate quantitative and systematic evaluation of these issues to inform our admission practices.

Here we provide an initial discussion of two of these causes.

Is it fully attributable to COVID? This explanation fails on two counts. First, the top half of our student population performs as well as the pre-pandemic students. Given our large pool of applicants (~16,000), and low admission rate (~2% for non-athletes), it defies reason to think that we cannot find more A & B students and have been forced to admit D & F students to fill the class. More likely, our admissions process is failing to spot the D & F students. Second, the COVID hypothesis does not explain the differences in top achievers across schools. COVID or not, top Putnam performers still exist. They are just not at Caltech.

Is it caused by changes in admissions practices? Our admission criteria have changed in the past few years and thus deserve scrutiny. Starting with the class entering in 2021 (today’s juniors), as a response to Covid, we stopped requiring applicants to take the SAT/ACT test, which in the past was used as an indicator of math and verbal proficiency. Furthermore, we introduced a number of non-cognitive criteria alongside academic merit. In the process, we seem to have lost focus on the need to choose applicants who have acquired in high-school the skills needed to thrive in Caltech’s rigorous and fast-paced academic training.

Why bring back the SAT/ACT as soon as possible?

The case for using the SAT/ACT in our admission process is that it provides a necessary, but not sufficient, signal for success in our challenging educational program. These test scores are unlikely to be predictive of outcome differences at Caltech among students who perform above a high-threshold, as has been the case for our historical student population. However, based on years of experience in the classroom and the lab, we believe students who are not able to score highly on the math sections of those tests are not likely to perform well at Caltech.

Consistent with this view, in March 2022, MIT brought back the SAT/ACT as a requirement [ref1,ref2,ref3]. The report from the MIT dean of admissions is well sourced, and — given the similarity of MIT’s mission to our own — makes for useful reading. Here are some relevant quotes:

• “Our research has shown that, in most cases, we cannot reliably predict students will do well at MIT unless we consider standardized test results alongside grades, coursework, and other factors. These findings are statistically robust and stable over time, and hold when you control for socioeconomic factors and look across demographic groups. And the math component of the testing turns out to be most important.”
• “It turns out the shortest path for many students to demonstrate sufficient preparation — particularly for students with less access to educational capital — is through the SAT/ACT, because most students can study for these exams using free tools at Khan Academy, but they (usually) can’t force their high school to offer advanced calculus courses, for example. So, the SAT/ACT can actually open the door to MIT for these students, too.”
• “[T]here is no pathway through MIT that does not include a rigorous foundation in mathematics, mediated by many quantitative exams along the way. So, in a way, it is not surprising that the SAT/ACT math exams are predictive of success at MIT; it would be more surprising if they weren’t.”

Similar results have been found by several recent studies at other institutions [ref4,ref5,ref6, ref7]. For example, a study by Opportunity Insights looked at admissions records and student outcomes at multiple college Ivy-Plus colleges between 2017 and 2022 and found that “[e]ven among otherwise similar students with the same high school grades, we find that SAT and ACT scores have substantial predictive power for academic success in college,” even after controlling for high school grades. As shown in the figure below, “[s]tudents opting to not submit an SAT/ACT score achieve relatively lower college GPAs.” A related earlier study by Opportunity Insights also found that SAT/ACT scores are substantially more predictive than high-school grades of the likelihood of attending an elite graduate school or working at a prestigious firm.

In stark contrast, three months after the MIT announcement, Caltech announced that we would extend the moratorium by three years. In fact, the press release from admissions making this announcement stated: “…standardized test scores have little to no power in predicting students’ performance in the first-term mathematics and physics classes that first-year students must take as part of Caltech’s core curriculum. Further, the predictive power of standardized test scores appears to dissipate as students progress through the first-year core curriculum.” This claim refers to an internal report that has never been released to the faculty for evaluation and discussion.

In fact, the predictive value of the SAT on Caltech student performance had been studied in the 1990s by Dave Rutledge and colleagues. They found that students with a Math score below 700 have a high chance (larger than 50%) of dropping out. In the wake of that study, the admissions office set 700 as the minimum Math score for admissions.

As recently as 2019, all of our admitted students had an SAT Math score above 700, with the 25/75 percentiles at 790/800. In fact, historically, Caltech students had the highest SAT scores of any university. Now our admission process dismisses the SAT as a useless metric. One of the tenets of empiricism is that extraordinary claims require extraordinary evidence to support them. Given that this claim goes against a practice that has served Caltech and MIT well for decades, that MIT recently looked carefully into this issue and brought back the SAT/ACT, that recent studies have found that SAT/ACT are predictive of student outcomes at Ivy-Plus colleges, that the 1990s Rutledge study found similar conclusions at Caltech, and that the report cited by the Admissions Committee Chair has not been shared with the faculty for evaluation, we are skeptical of the claim that it is not a useful metric on admissions.