Cream-Skimming and Spoiled Milk

Interesting questions in education policy. Part one.

The following two pieces will cover items related to the Economics of Education – a topic which I consider near and dear to my heart. I’m sure its abundantly clear to my readers why the study of education is important, so we won’t spend time discussing that. Instead, I want to focus on some topics which encompass what I consider to be unanswered and often overlooked questions within the broader education literature. Today, we’ll focus on something I’ve always found puzzling: the effects of cream-skimming.

When economists model outcomes they often do so utilizing “functions.”  That is, a mapping of a certain set of inputs (e.g., a student’s race, gender, zip code, etc.) to a set of outcomes (e.g., test scores, graduation rates, etc.). The relationship between these inputs and outputs provide a way to view how there may exist a causal relationship between these variables. So, when I say “test scores are a function of student ability and teacher quality” I’m really saying “a student’s test scores are related to (perhaps causally or are “dependent upon”) their own test scores and the quality of their teachers.”

Economists often model education via a “production function” – essentially, a function whereby a set of inputs are combined to “produce a product.” In this case, “educational attainment,” typically measured by test scores.¹  

We can think of modeling this relationship like this: 

A student's previous test scores 
+ quality of their teacher 
+ demographic characteristics 
+ peer characteristics 
= A student’s current test scores

• The goal here is to assess on the margin what, say, the difference in test scores is between someone of race A vs. race B or something along the lines of: “For every 1 point change in a student’s 1st grade test scores, what is the corresponding increase/decrease in their 2nd grade test scores, all else held constant.” 

  • The important thing to understand here is that these effects need to be decomposed into both direction and effect size. It’s not enough for us to just know whether or not being of low-income has a negative or positive impact on your test scores. It’s important to also know by how much. 

It’s very easy to measure the effect of what we call “observable” characteristics on test scores. Think of these as what the name implies: things we can observe (e.g., race, gender, previous test scores, income, whether your school is in a rural area, etc.). What is perhaps more important, though, is being able to measure the effect of unobservable characteristics. Think of these as being similar to how we think about intangibles in sports: “is the student a leader,” “are they disruptive to others,” “are they hard-working.” Chief among these “unobservable” characteristics which economists are interested in exploring are what are called peer effects.

Peer effects describe any mechanism through which the behavior, actions, preferences, backgrounds, etc. of peers affect others. Within the education context, we can think of decomposing these effects into “positive” and “negative” peer effects.² Examples of these include the following:

-        Positive: You are a student in a class with average test scores. Your deskmate consistently scores at the top of the class and offers to help you with questions when you are stuck or clarifies concepts you don’t understand during class. Here, we would consider the ability of your classmate to “spill over” to yourself resulting in a positive effect on your test scores. That is, being around “high-ability peers” makes you more of a “high-ability peer.”

• See here an interesting paper on random roommates and peer effects - one of my favorites.

-        Negative: You are a student in a class and one of your peers is constantly disruptive. They spitball the teacher, are always on their phone, and the teacher spends considerable time disciplining them.  We would expect this disruption to negatively impact your learning, thus resulting in worse educational outcomes the longer you are exposed to this student. Essentially, “one bad apple spoils the bunch.”

• See here an interesting paper theorizing a model of disruption in the classroom.

To focus on positive peer effects for a moment, we can see an interesting scenario emerge in what we call “cream-skimming.” In essence, cream-skimming is the phenomenon by which private/charter/magnet schools often “skim the cream” of the highest-ability students from the public school system and transition them to schools where the average test score is much higher.

This brings about an interesting question: If private/charter/magnet schools provide a channel through which high-ability students are able to exit from public schools, does this result in “negative peer effects” on the lower-ability public school peers these students left behind?

To illustrate, let’s imagine you are a student in the 8th grade of average ability. A new magnet school opens up in your neighborhood and next year, a significant amount of your high-ability peers will leave for this school. What is the lingering effect of the loss of these students on your educational achievement in high school?

Interesting question, right? Economist Angela Dills, in a paper aptly titled Does Cream-Skimming Curdle the Milk?, studied this exact question utilizing data from the inaugural 1985 class of the Thomas Jefferson High School for Science and Technology in Virginia.

For those unfamiliar, TJHS is one of the top high schools in the country. It is a magnet school for extremely high achieving students (the average SAT score for the graduating class of 2020 was 1528 and the average ACT score was 34.5 - see here) and given that this school selects peers on a purely meritocratic basis (at the time - though, not anymore) provides a great “natural experiment” to measure the effects of cream skimming. To simplify, her analysis is something akin to the following:

• You are in 8th grade in 1984 and you take a standardized test. If you score really, really high (no like really high – 99th+ percentile) on this test you’re offered admission to the first class of TJHS.

  • Everyone in your school district takes this test so admissions are fair across all students.

• A couple of your friends get into TJHS and leave you during high school. This happens across every middle school in the district.

• You go on through high school without these high-ability peers to help you during class or answer homework questions. Meanwhile, these high-ability students are in an extremely accelerated program and themselves continue to score well (if not better).

• By comparing the distribution of test scores of high school seniors in the class above your with your class – is there evidence that test scores are measurably lower after the introduction of the magnet school?

The idea here is that yes, we should see a decrease in students scoring in the top quartile on standardized tests in normal high schools after the introduction of the magnet, as there are less high-ability students. However, what we should not see is more students scoring in the bottom 25th quartile. That is, if there are truly no effects from these peers leaving – we should simply find that the number of students scoring in the top quartile goes down, not that students should be worse.

The main result Dills finds is exactly what you might think: when high-ability peers leave, low-scoring students are affected negatively.

"After adjusting for the scores of the leavers, if an additional 1% of the high ability students leave a school, there is a one percentage point increase in the percentage of remaining students scoring in the bottom national quartile and a one-half of a percentage point decrease in the percentage of remaining students scoring in the top national quartile" (20)

Now you might be thinking: ok, so what? What is the policymaker’s objective here (i.e., how do I solve this, or, does it really matter)?

Well, it depends on your objectives as a policymaker:

We can think about decomposing sets of students into two types of matching protocol – positive assortative matching and negative assortative matching. Under the assumption that we can separate students and teachers into groups of “high-ability” or “low-ability” and that there exists some degree of complementarity between students of similar academic levels (i.e., students who are of the same level tend to learn better) we can observe the following:

• Positive Assortative Matching: Match high-ability students with high-ability teachers and low-ability students with low-ability teachers. This maximizes the mean test score of a pool of students as all students are learning, but not all students are learning at the same speed. Thus, the average test score of everyone increases though gaps between students are not minimized.

• Negative Assortative Matching: Match low-ability students with high-ability teachers and high-ability students with low-ability teachers. Or, do the same by “mixing students of varying abilities together.” This reduces the variance of outcomes as high-ability students (due to the non-complementarity existent in this matching scenario) will “bleed” some of their learning gains to the lower-ability students – bringing these groups closer together.

  • To think of this more simply, instead of giving higher ability students “enrichment” activities, task them with helping the lower-ability students with homework (this is a crude example, but hopefully it illustrates the mechanism behind this effect)

Policymakers who choose to limit the variance in educational outcomes should consider cream-skimming as an important problem. In fact, they do – teachers unions often oppose voucher programs for this exact reason. That said, more research is needed to fully understand the effects of cream-skimming, not to mention peer-effects as a whole.

There are significant normative arguments to be made as to which policy goal is more important, though I hope an understanding of the issues provides a better lens through which you, as a voter, can make your own decisions on education policy.

Next week, we’ll continue along this topic - stay tuned.

1. I understand these are imperfect measures but they are arguably the best we have.

2. There are many kinds of peer effects including sociocultural/ingroup-outgroup effects and others, some of which are well documented in the literature. Today we’ll just focus on those explicitly tied to outcomes through measures of student ability.