Tuesday, June 24th, 2014
50

That Big Study About How the Student Debt Nightmare Is in Your Head? It's Garbage

Doesn't that sound like a fact? Well, it's something that might be a fact.

The Brookings Institute Institution (!!!) is here to tell you that the whole fable of debt-panicked young people in America is a lie! And their study comes complete with a huge announcement in the New York Times, which puts a rather snide slant on the whole thing. It's all in your head, millennials! "Only 7 percent of young-adult households with education debt have $50,000 or more of it," summarizes the Times up top. (There's a quiet and enormous caveat in that sentence, which we'll get to shortly!)

But then they must backtrack from this tale a bit:

The first thing to acknowledge is that student debt has risen over the last two decades. In 2010, 36 percent of households with people between the ages of 20 and 40 had education debt, up from 14 percent in 1989. The median amount of debt—among those with debt—more than doubled, to $8,500 from $3,517, after adjusting for inflation.

So let's see: people with college debt saw their debt double, and also the number of those households with debt more than doubled. That is not exactly undermining this supposedly fake narrative of the increase of student debt! What's more, the Times notes, tuition and fees at public colleges are up 50% in the last ten years.

Then they must come to this graph.


Do you see where that says "based on households with people between 20 to 40 years old with at least some education debt"? That's actually quite a bit of a fudge!

What's the deal with these numbers? GLAD YOU ASKED. It's not what it sounds like!

• Those aren't households with people between 20 and 40; those are households headed by people between 20 and 40. Which is to say, this data excludes all people living in households headed by, say, their parents, or other adults. The way Brookings put this is: "households led by adults between the ages of 20 and 40." Just another way to say it excludes all households led by anyone over 40! (Those households might be identical in student debt to "young" households! Or they might not? WHO KNOWS!)

• One effect of this age spread sample is that it includes college graduates from up to almost 20 years ago. This is literally not at all a study of college graduates of the last five years, or even ten years. We're talking about people up to the age of 40, well into Gen X.

• Also, in this survey, when there are multiple people in the household, the Brookings Institution simply divided the amount of college debt by number of people in the household. So one person's $20,000 college debt becomes two people's $10,000 college debt. This works out mathematically, of course, but not structurally.

• And finally: The number of the people making up this data is quite small.

Where does it come from? GLAD YOU ASKED.

All this data comes from the Survey of Consumer Finances, which is conducted by the Federal Reserve Board of Governors and the Department of Treasury. It takes place every three years, since 1983. It samples about 4500 households in the U.S., usually, but recently expanded to 6500 households. And this isn't new data; this is the data from their 2010 survey. (The 2013 survey will be published in 2015.)

Of all the households in that study, only about 1711 have "household heads" that are younger than 40. That's what they're extrapolating from. (And, intriguingly, a small number of those have a head of household younger than 18.) This is not a big sample!

What, obviously, does this data completely omit? Well, one obvious thing is… households who are headed by someone who is not under 40. One thing we know is that, in 2012, 36% of Americans aged 18 to 31 were not their head of household, because they were living with their families.

This survey also clearly combines family and non-family households. (Also, there's some unknown amount of statistical imbalance from same-sex households; 31% of same-sex households are likely to have two college-degreed people, compared to 24% of opposite-sex married households and just 12% of opposite-sex cohabitating households.)

And finally… this survey is, essentially, of rich people. No, literally!

We apply survey weights throughout the analysis so that the results are representative of the U.S.
population of households. The use of survey weights is particularly important in the SCF because
the sample design oversamples high-income households to properly measure the full distribution of
wealth and assets in the United States. This high-income sample makes up approximately 25 percent of
households in the SCF.

Literally what they are saying there is that the information on which they are basing a sweeping assessment of American student loan debt is based on a sample in which 25% of those surveyed were "high-income households." This is insane. (Update: I wanted to clarify that I get it that they are weighting this over-representation down to represent the population at large; that's not my beef, entirely. Mostly I think it shows a further weakness in their non-rich sample at large.)

Here's a fun footnote in the actual Brookings Institution report:

These statistics are based on households that had education debt, annual wage income of at least $1,000,
and that were making positive monthly payments on student loans. Between 24 and 36 percent of
borrowers with wage income of at least $1,000 were not making positive monthly payments, likely due
to use of deferment and forbearance….

So… they… set aside as much as 1/3rd of people in the survey sample because they weren't paying off their student debt. That's an intriguing class of debtors, don't you think? They claim that dismissing these people from the sample did not "qualitatively alter the pattern of results reported above"; so why dismiss them at all?

It's shocking that the Times presents this survey in this way. This study does actually tell us things! It's not actually a pack of lies. It just doesn't tell us necessarily what people are saying it's telling us. And no one of course will actually read the whole survey, so its repackaging will now enter the narrative, thanks to bloggers….

… and professional policy wonks alike. And that's a huge disservice.

UPDATE

I wanted to specifically call attention to this criticism at Quartz, which actually… doesn't really disagree with anything either here or in the report. It's full of good points, but what it's not is any kind of defense of either the Brookings Institution study or of the marketing of that study. Saying that it's a small-ish percentage of debt-havers that are carrying massive amounts of debt isn't controversial or unreasonable. It's actually probable! But presenting a definitive landscape of America's student debt based on heavily sampled data most recently updated in 2010 and heavily weighted to "reflect" America as a whole is lot less useful. (Have we not been through this on a daily basis with the Huffington Post Science section, after all?)

50 Comments / Post A Comment

this is why 'data journalism' is the new 'objectivity' as far as being a complete scourge when it comes to, you know, journalism's ostensible job of getting the truth out there. you don't have to be *correct*—you just have to have the illusion of being correct, and numbers conveniently provide that because oh isn't math just infallible?

ugh ugh UGH. i hated 'a new study' bullshit when it was telling me incorrect things about pop music, but shit like this is even more corrosive.

frontsidebus (#5,387)

@Maura Johnston I don't think taking liberties with numbers in order to present "a new study" as some revealing counter-intuitive/contrarian new fact that will blow everyone's minds is anything new at all. I mean, Malcolm Gladwell? I guess I don't believe in 'data journalism' as being a thing.

@Maura Johnston Yeah, but how can data journalism be bad when it can tell me if my masturbation frequency is "normal" or not?

@frontsidebus oh it's not new, it's just that it's being repackaged as a trend—vox, fivethirtyeight, the upshot, etc.

It's a pretty effective hedge against the time constraints of blogging, though. "We have numbers!"

skyslang (#11,283)

@Maura Johnston Every time data is used to make a shocking prediction or contrarian conclusion we need to question where the data came from, who it came from, the instruments they used to collect the data, the methodology they used to collect the data, the math they used to come to their conclusions, etc. All of it. Every time. Even when the shocking prediction supports your world view.
We need to exercise our critical thinking skills because journalists certainly aren't.

117873389@twitter (#279,973)

This criticism is almost all correct, and believe me, I'm on your side.

That said, the criticism about oversampling wealthy people is wrong. This sort of thing is standard practice. For example,
Let's say you sample 300 people randomly, and it turns out that 200 are wealthy. You can fix this problem by giving the rich people's responses less weight. Essentially, the response of each poor person gets twice the weight of each rich person. If weights are used correctly, this isn't a problem.
This method is used by just about everybody. Census Bureau, Bureau of Labor Statistics, Consumer Expenditure Survey…etc.

That said, every other criticism here is correct.

thethetk (#8,111)

@117873389@twitter This jumped out at me too. Choire should fix. Piece is great otherwise.

chevyvan (#201,691)

@117873389@twitter @117873389@twitter Agreed all all points. The sample size, oversampling, and weighting are not concerning to me.
<—-Survey researcher

Freddie DeBoer (#4,188)

@117873389@twitter Right. He's just flatly misread the methodology.

Clay Risen@facebook (#243,510)

Nice work. Not to be a pedant, and not that it matters, but … since this is a piece about accuracy, it's the Brookings Institution, not the Brookings Institute.

I'm not sure if it speaks to natural cynicism or hard-won experience with census data, but as soon as I saw the phrase "young-adult households" in the original Times article, my scammy-analysis warning light lit up for exactly the reasons you mentioned.

KarenUhOh (#19)

The reliance on the facade of selected percentages, with utter indifference to aggregates (total debt, in the trillions; total number of students under the debt hammer, increasing yearly within a [lengthy] stagnant economy), smells results-driven.

Buried in that Brookings paper is a graph of income growth vs. debt growth over a period of years. See the lines squeeze closer and closer together.

There's a reason the scary loan anecdotes are effective. There are a hell of a lot of them.

Freddie DeBoer (#4,188)

@KarenUhOh And they have been stitched together into a narrative that is not supportable by data, but you're not allowed to say that. Hence the comments on this post.

Post- a very casual look at the underlying paper, here's another point elided by the Times's analysis :

The study includes only people aged 20 to 40 currently paying against their student loan debt. I think it's a fair assumption that the majority of the people in this group are not incurring additional student loan debt (ie, they took out all the student loans they'll ever take out before the age of 20). As I understand the data, the paper defines "student debt" here as the balance of debt currently held, not total debt incurred. That's (one of the reasons) why it seems low — on average, a person in this study has been paying off their debt for 10 years already, and the current debt reported reflects that.

People in this group have been spending their prime savings years paying down their student loan debt. The low numbers reported by the study just reflect the unprecedented degree to which individuals' incomes have been rerouted from savings to servicing their student loan debt.

@Gef the Talking Mongoose : As I understand the data…

Quoth the paper, p. 7 : "It is important to note that we measure the amount owed at the time of the survey, which will differ in many cases from the amount originally borrowed."

Trilby (#3,897)

Yeah, I thought garbage too! All you need to look at is one undeniable statistic– that more young people than ever are coming home from college to live with their folks. It's not that they want to; they have no choice. They don't have the extra money to spend on starting their own household. What does that tell you, Brookings Institution and David Lionhart!????

Freddie DeBoer (#4,188)

I don't know why it's so hard for people to reconcile these various ideas:

1. The student loan debt crisis is a crisis, both structurally and morally.
2. Its size and damage has been consistently overblown. The notion that it is anything like the size or danger of the housing bubble is absolutely absurd. It's orders of magnitude smaller and vastly less dangerous to the greater economy.
3. American college graduates are without question some of the most economically secure people in the world. They far outperform both international averages and people in America without degrees. People keep trying to argue that it's not financially worth it to go to college anymore, and that keeps not being true. And it's only people who already enjoy the benefits of a college degree that make that claim.

I have been arguing about ways to reduce the cost of college, to reinvest in higher education, and end this student loan crisis. But I've also been consistently frustrated by the refusal of people to take a responsible, non-hysterical look at the actual numbers, which are not what they've been represented to be. But people refuse to do so, and they become very, very unhappy when you ask them to. Because this is just a fact: student loan debt has become a way in which an inarguably privileged class of people gets to play act the role of oppressed. That's just reality. But I have a feeling that the type of people who write for the Awl, and the type of people who read it, have their minds made up, regardless of evidence!

Also, your methodological critiques here leave a lot to be desired. Seriously.

"Literally what they are saying there is that the information on which they are basing a sweeping assessment of American student loan debt is based on a sample in which 25% of those surveyed were "high-income households." This is insane."

No. Read it again.

"This is literally not at all a study of college graduates of the last five years, or even ten years"

Where are you getting that? It includes people in the age range from 20-30, which undoubtedly covers college graduates from the last 5-10 years.

thethetk (#8,111)

@Freddie DeBoer ""This is literally not at all a study of college graduates of the last five years, or even ten years"

Where are you getting that? It includes people in the age range from 20-30, which undoubtedly covers college graduates from the last 5-10 years."

Yeah, but his point is that by including another ten years worth of people, you've skewed the sample away from getting meaningful info about very recent changes.

Freddie DeBoer (#4,188)

@thethetk But you have to actually present info that "very recent changes" are in fact dramatic. And nobody has done that, here. The notion that there is some clear and major difference in the student loan debts of graduates from the past five years and those from the past fifteen is totally unproven to me.

thethetk (#8,111)

@Freddie DeBoer Don't disagree. But I don't think Choire's project here is creating an accurate picture of student debt; it's poking (pretty legit) holes in the Brookings study as reported by Leonhardt. Leonhardt's piece makes frequent reference to changes over "the last two decades," but a sample that goes up to age 40 can't really demonstrate much change over time.

NeonTrotsky (#2,249)

@Freddie DeBoer Isn't it misleading for people to cite a study claiming that student loan debt levels are not *that* bad, when said study excluded all those not currently making loan payments, a group that in 2013 made up more than half of all student loan holders, either through deferral or delinquency? And wouldn't it follow that those who are deferring or delinquent would be more likely to have larger amounts of debt?

Whatever your quibbles with Choire's critique (besides the sample size jibe), it's more intellectually honest than the Times' subheader "Large Amounts of Student Debt Are Not Common" and its attached chart.

Ophelia (#75,576)

@thethetk Furthermore, by measuring not total debt incurred, but total debt outstanding, AND increasing the age range, this means that people who have been paying down student loans for almost 20 years are included in the sample in that "less than $10,000" category. I'm 33, had 20k in student loans, and now have 2k, after 12 years. That said, the effect on my savings rate, consumption, etc., is better reflected by the overall $20k than by the fact that I only have 2k left.

tdp (#5,491)

@Freddie DeBoer But if there is no crisis there wont be any bailout.

"Of all the households in that study, only about 1711 have "household heads" that are younger than 40. That's what they're extrapolating from. (And, intriguingly, a small number of those have a head of household younger than 18.) This is not a big sample!"

1,711 is plenty – the standard error on a group of that size is quite low.

Freddie DeBoer (#4,188)

@Matthew Darling@facebook Right. That's an enormous sample.

KarenUhOh (#19)

@Freddie DeBoer What are the "actual numbers" of those in debt, and the amount of their debt? And what are the "actual numbers" that establish the earning capacity or experience of these "privileged" persons (which, in the grand scheme, I agree they tend to be) such that there is an ability to satisfy that debt, and over what length of time?

And what are the "actual numbers" with respect to the debtors who simply stop paying, or low pay, such that the burden of the debt shifts to the general population, or is written off, with the attendant costs of doing so? Or the "actual numbers" of what it costs for the lender to accept slow pay; or to chase delinquent debtors?

I suspect those numbers pull the sugarcoat off the study's tendency to reduce impact via arbitrarily-chosen statistics that support the conclusion the study wants to arrive at. Such that, at the end of the day, everyone seems to agree that, however we parse it, it's still called a "crisis," both "structurally and morally."

Freddie DeBoer (#4,188)

@KarenUhOh If you don't like the Brookings Institution, this piece pulls from Harvard, the NBER, the NCES, the BLS, UChicago….

http://www.theatlantic.com/magazine/archive/2013/03/myth-student-loan-crisis/309231/

can't tell (#279,997)

I have plenty of student loan debt from grad school (I was in the 50k+ bracket but now down to about 40k) and to be honest, I feel it was worth it and good investment. I got a good job out of it. I'm not really worried about the total amount that I owe, it's the $300 in just interest alone I have to pay every month that pisses me off.

whizz_dumb (#10,650)

@can't tell Yeah, which is a "large amount" when rent is 1/3 to 1/2 of your income.

leo from chicago (#280,053)

@can't tell — Lemme see, I'm 58, have 50k in student loans and, not surprisingly, am in forbearance (though I pay a reduced percentage per month). So hey, I'm not even part of Brookings' stats!

David (#192)

Growth in tuition expense in excess of household income growth combined with increasing numbers of HS graduates entering College and Graduate School will lead to continued growth in education-related debt over the next several years. However, enrollment in graduate and professional schools is likely to slow once the labor market improves. The recent increase in borrowing for school was caused in part by a reduction in wealth following the financial crisis.

KarenUhOh (#19)

@David And there's the nub: Why, pray tell, is the labor market going to improve? What is going to create an increase in wealth? Real wealth, not funny-money assigned to shares in companies that operate on the hope of income at an indeterminate future milestone?

I'm older than most of you people. Don't know much, but have seen quite a bit. This downturn is no blip. Sure, a hell of a lot of people do well. But there is an ever-increasing mass who don't. Inasmuch as the student loan debt situation is both symptom and sign of a much larger issue, it remains a problem that glossing over statistically does nothing to solve, or to assuage my geriatric concerns.

David (#192)

@KarenUhOh Right. Your question is related, but moves away from the reasons why education related debt is growing, in total.

All of this you know already: The statistical measures of the "labor market" are adjusted– generally, to show improvement– to reflect "lower unemployment" by simply understating the real number of people that are actually unemployed– because under the current reporting "rules" the stats-reporting-entities can exclude anyone that is deemed to be no longer actively seeking a job on a week-by-week basis (as reported and tracked by State unemployment agencies). So as people exceed the maximum time they can receive unemployment insurance payments, they are then classified as "no longer seeking employment" because they don't report that they are still searching for a job, and so are statistically no longer counted as "unemployed." In actual fact, the labor market is statistically improving because more people are being hired, and more people no longer counted as unemployed for the above stated reasons. Put most simply, wages remain low (and lower than in the past) because of the "supply" of labor still exceeds the "demand" for employees– i.e. employers don't have to pay them more because someone else will work for less. When wages are low, overall debt is easier to accumulate.

hamsammich (#279,999)

I have a perhaps dumb question. What about roommates? Group houses? Would someone living with a roommate be considered "head of household"? If people age 20-40 living with roommates were not included either, I imagine that would definitely affect the results of the study.

hershmire (#233,671)

Yeah, but the chocolate ration is up to 25 grams a week, so everything's fine.

@hershmire total win :)

holdup!holdmyphone! (#274,038)

no self-respecting student debtor would deign to respond to a government survey

Ralph Haygood (#13,154)

I can argue statistics with erudite refinement (Google-Scholar me if you care), but I refuse to argue the statistical niceties or un-niceties of this or any similar study, because…

…to a good first approximation, student debt shouldn't exist. Until around 30 years ago, it barely did exist in the USA. And for many years prior, American universities generally worked remarkably well in most ways. They educated an unprecedented proportion of the population, whose participation in the workforce contributed much to the broad prosperity of the country. They fostered research that made the USA a leader in nearly every kind of science and engineering and much else besides. They did all this with taxpayer support that enabled most of them to charge students very modest fees. Some of the best, such as the University of California, were practically free. None of this needed to change. There's no intrinsic reason why it couldn't have continued to this day. Instead, under the "leadership" of malicious Republicans and "pragmatic" Democrats, the public has disinvested in higher education, and universities have been given over to greedy fools who prattle about "running the university like a business". Even supposed liberals like Barack Obama routinely speak of higher education as primarily a private good, "the single best investment that you can make in yourselves and your future", rather than a public good, an investment we the citizens should make for our collective benefit, such that anyone who demonstrates ability and commitment can get a university education without borrowing money.

Obviously, these trends are unlikely to be reversed. After all, this is the USA in 2014. Only rich people's "problems" matter, and rich people don't take out student loans. However, arguing over whether student debt is crushing or merely burdensome strikes me as a form of fighting over the crumbs that fall from our masters' tables.

KarenUhOh (#19)

@Ralph Haygood Having passed through that "essentially free" public education era, funded almost entirely via tax dollars–without anyone having considered that a particular evil–I would offer that this gets within reach of the root of the problem.

We are afraid to tax*. Or, we tax and fail to appropriately prioritize expenditures. We've spiraled into a vortex that won't change this. The political agendas of everyone are focused on reducing taxes, with the disingenuous concomitant of eliminating debt.

If one doesn't find it disingenuous, because one would eliminate government expenses as a panacea, then you live in a dream, or in Northern Idaho.

*People with lots of money, or, "wealth."

My rules of modern social science:
1) Any social science conclusion which supports or argues an ideological point is suspect until you read all the fine print and confirm the reasonableness of the conclusion.
2) Once you read all the fine print, most of the time the widely disseminated conclusions are false or unproven by the research.
3) Conclusions which happen to agree with the ideological position of the researcher will turn out to be wrong or unproven 90% of the time.
4) Some social scientists are conservative/right-wing. Any mainstream news coverage of their work will include this fact. A similar number (~10%) are not ideological. The other 80% of social scientists are liberal/left-wing. Those scientists' ideology is not mentioned by the press, except for explicitly conservative press (e.g., National Review, American Rifleman, Fox News).

skyslang (#11,283)

@David Pittelli@facebook Absolutely. We need to question all concussions based on data, not just social science. Obviously, data can be collected and manipulated in such a way to support the scientist's world view / political agenda, sometimes without them even knowing it. Journalists should be doing this work as part of reporting, but since they're not (unless it conflicts with their world view or the viewpoint of their employer), we need to do the work for ourselves. There is so much hysteria in journalism right now, it's ridiculous.

The whole argument of the article is idiotic. We have entered the world where medical degree will cost $400,000. It is astronomic. Typical UC education $160,000. It is regardless of the debt. It is happening and statistics are just a matter of time to catch up to reality. So any morons claiming 7% owing $50,000 bullcrap are simply being an idiot self. It is worse than that. Debt is in fact one thing. Expense that was bored by the family and student and this nation is at a breaking point. We need to destroy all education system and recreate it from a hut, pen and a piece of paper. Current system is simply fucked. And I am sorry to millenials as a X gener who enjoyed what education should have costed.

20564154@twitter (#280,102)

I wonder at what sort of sampling bias there would be in a survey of a population that umm… actively avoids answering the phone. Granted, the methodology specifically excludes people who are behind on their payments, but the population they are trying to generalize about has a significant chunk of folks that go into a fetal position once the phone starts ringing every morning at 8.

What amazes me is people accept the debt for a service they desired. But now that they find their fine arts degrees are worthless, or poly sci party degrees…they need help!

What is more amazing is that people actually believe that you need college to be "smart" and work qualified.

I detest companies that promote or hire based on degree status alone.

318064410@twitter (#280,140)

The Brookings analysis ignores the situation of the most vulnerable borrowers: those who go to for-profit colleges, and those who do not graduate (this population overlaps significantly). And if you disaggregate the data by race, you see that minorities also have much higher debt levels. In a large part, this may also be due to the disproportionate enrollment of students of color in expensive, shoddy for-profit programs. These are the students being most harmed by student loan debt.

As for the other students – the data clearly shows that they now have to pay a greater share of their disposable income for debt service. While they may be able to meet their debt payments, student loan debt keeps them from being able to invest in other areas, like mortgages, retirement, starting businesses. Whether or not graduates CAN pay these loans, we still need to think about whether it's a good thing that an increasingly large chunk of their college wage increase is devoted to paying off college, as opposed to other socially important uses.

Burnerjack (#280,143)

No comment, but three questions:
1) Do educational institutions receive a commission on student loans? Anytime I inquired about furthering my education, those I spoke with were adamant about making sure I would qualify for such loans despite the fact that I was equally adamant about not be interested in taking on such debt.
2) Why are people (of whom I know of both an airline pilot and a professional design engineer) who are quite capable of paying off their debt, do it kicking and screaming or not at all?
3) What might one suppose be the purpose of framing such data in such a dubious manner? "Inquiring minds want to know…"

ashley76 (#280,524)

just a shout testimony about my past and recent result, I want to inform you all that there is a spell caster that is genuine and real. I never really believed in any of these things but when I was losing my husband to another lady, I needed help and somewhere to turn badly. I found consultant.Dr Ekaka spells and i ordered a LOVE SPELL. Several days later, my phone rang. and it was my husband was his old self again and wanted to come back to me! Not only come back, the spell caster opened him up to how much I loved and needed him. Spell Casting isn't brainwashing, but they opened his eyes to know how much we have to share together. I recommend anyone who is in my old situation to try it. It will bring you wonderful surprises as well as your lover back to you. The way things were meant to be you can contact the spell caster on ekakaspelltemple@yahoo.com he's very nice and great.

ashley76 (#280,524)

just a shout testimony about my past and recent result, I want to inform you all that there is a spell caster that is genuine and real. I never really believed in any of these things but when I was losing my husband to another lady, I needed help and somewhere to turn badly. I found consultant.Dr Ekaka spells and i ordered a LOVE SPELL. Several days later, my phone rang. and it was my husband was his old self again and wanted to come back to me! Not only come back, the spell caster opened him up to how much I loved and needed him. Spell Casting isn't brainwashing, but they opened his eyes to know how much we have to share together. I recommend anyone who is in my old situation to try it. It will bring you wonderful surprises as well as your lover back to you. The way things were meant to be you can contact the spell caster on ekakaspelltemple@yahoo.com he's very nice and great.

The study points out how the management team is focusing on students related to THEM and has no interest whatsoever in observing results on the ENTIRE college pool. The sophisticated calculations that go into sweeping the poor under the rug are simply hidden away… Yes there are students loaned 75% of what they need to finish a degree with no job skills getting the cold shoulder… the students without legitimate needs are targeted with help and the poor again swept under the rug! You have local leaders that are rich really putting the shaft to students in the most need because it's their belief that it's the right thing to do…..

Post a Comment