Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

Cathy O'Neil

Crown (2016)

Rating: ★★★★☆

Tags: Political Science, Public Policy, General, Social Science, Privacy & Surveillance, Business & Economics, Statistics

Political Sciencettt Public Policyttt Generalttt Social Sciencettt Privacy & Surveillancettt Business & Economicsttt Statisticsttt

**
Longlisted for the National Book Award

A former Wall Street quant sounds an alarm on the mathematical models that pervade modern life — and threaten to rip apart our social fabric**

We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether we get a car loan, how much we pay for health insurance—are being made not by humans, but by mathematical models. In theory, this should lead to greater fairness: Everyone is judged according to the same rules, and bias is eliminated.

But as Cathy O’Neil reveals in this urgent and necessary book, the opposite is true. The models being used today are opaque, unregulated, and uncontestable, even when they’re wrong. Most troubling, they reinforce discrimination: If a poor student can’t get a loan because a lending model deems him too risky (by virtue of his zip code), he’s then cut off from the kind of education that could pull him out of poverty, and a vicious spiral ensues. Models are propping up the lucky and punishing the downtrodden, creating a “toxic cocktail for democracy.” Welcome to the dark side of Big Data.

Tracing the arc of a person’s life, O’Neil exposes the black box models that shape our future, both as individuals and as a society. These “weapons of math destruction” score teachers and students, sort résumés, grant (or deny) loans, evaluate workers, target voters, set parole, and monitor our health.

O’Neil calls on modelers to take more responsibility for their algorithms and on policy makers to regulate their use. But in the end, it’s up to us to become more savvy about the models that govern our lives. This important book empowers us to ask the tough questions, uncover the truth, and demand change.

**

More Advance Praise for
WEAPONS OF MATH DESTRUCTION

“Weapons of Math Destruction
is a fantastic, plainspoken call to arms. It acknowledges that models aren’t going away: As a tool for identifying people in difficulty, they are amazing. But as a tool for punishing and disenfranchising, they’re a nightmare. Cathy O’Neil’s book is important precisely because she believes in data science. It’s a vital crash course in why we must interrogate the systems around us and demand better.”

—Cory Doctorow, author of
Little Brother
and co-editor of
Boing Boing

“Many algorithms are slaves to the inequalities of power and prejudice. If you don’t want these algorithms to become your masters, read
Weapons of Math Destruction
by Cathy O’Neil to deconstruct the latest growing tyranny of an arrogant establishment.”

—Ralph Nader, author of
Unsafe at Any Speed

“Next time you hear someone gushing uncritically about the wonders of Big Data, show them
Weapons of Math Destruction
. It’ll be salutary.”

—Felix Salmon, Fusion

“From getting a job to finding a spouse, predictive algorithms are silently shaping and controlling our destinies. Cathy O’Neil takes us on a journey of outrage and wonder, with prose that makes you feel like it’s just a conversation. But it’s an important one. We need to reckon with technology.”

—Linda Tirado, author of
Hand to Mouth: Living in Bootstrap America

Copyright © 2016 by Cathy O’Neil

All rights reserved.

Published in the United States by Crown, an imprint of the Crown Publishing Group, a division of Penguin Random House LLC, New York.

crownpublishing.com

CROWN is a registered trademark and the Crown colophon is a trademark of Penguin Random House LLC.

Library of Congress Cataloging-in-Publication Data

Name: O’Neil, Cathy, author.

Title: Weapons of math destruction: how big data increases inequality and threatens democracy / Cathy O’Neil

Description: First edition. | New York: Crown Publishers [2016]

Identifiers: LCCN 2016003900 (print) | LCCN 2016016487 (ebook) | ISBN 9780553418811 (hardcover) | ISBN 9780553418835 (pbk.) | ISBN 9780553418828 (ebook)

Subjects: LCSH: Big data—Social aspects—United States. | Big data—Political aspects—United States. | Social indicators—Mathematical models—Moral and ethical aspects. | Democracy—United States. | United States—Social conditions—21st century.

Classification: LCC QA76.9.B45 064 2016 (print) | LCC QA76.9.B45 (ebook) | DDC 005.7—dc23

LC record available at
https://lccn.loc.gov/2016003900

ISBN 9780553418811

Ebook ISBN 9780553418828

International Edition ISBN 9780451497338

Cover design by Elena Giavaldi

v4.1

a

THIS BOOK IS DEDICATED TO

ALL THE UNDERDOGS

ACKNOWLEDGMENTS

Thanks to my husband and kids for their incredible support. Thanks also to John Johnson, Steve Waldman, Maki Inada, Becky Jaffe, Aaron Abrams, Julie Steele, Karen Burnes, Matt LaMantia, Martha Poon, Lisa Radcliffe, Luis Daniel, and Melissa Bilski. Finally, thanks to the people without whom this book would not exist: Laura Strausfeld, Amanda Cook, Emma Berry, Jordan Ellenberg, Stephen Baker, Jay Mandel, Sam Kanson-Benanav, and Ernie Davis.

CONTENTS

INTRODUCTION

CHAPTER
1
BOMB PARTS:
What Is a Model?

CHAPTER
2
SHELL SHOCKED:
My Journey of Disillusionment

CHAPTER
3
ARMS RACE:
Going to College

CHAPTER
4
PROPAGANDA MACHINE:
Online Advertising

CHAPTER
5
CIVILIAN CASUALTIES:
Justice in the Age of Big Data

CHAPTER
6
INELIGIBLE TO SERVE:
Getting a Job

CHAPTER
7
SWEATING BULLETS:
On the Job

CHAPTER
8
COLLATERAL DAMAGE:
Landing Credit

CHAPTER
9
NO SAFE ZONE:
Getting Insurance

CHAPTER
10
THE TARGETED CITIZEN:
Civic Life

CONCLUSION

Notes

About the Author

When I was a little girl, I used to gaze at the traffic out the car window and study the numbers on license plates. I would reduce each one to its basic elements—the prime numbers that made it up. 45 = 3 x 3 x 5. That’s called factoring, and it was my favorite investigative pastime. As a budding math nerd, I was especially intrigued by the primes.

My love for math eventually became a passion. I went to math camp when I was fourteen and came home clutching a Rubik’s Cube to my chest. Math provided a neat refuge from the messiness of the real world. It marched forward, its field of knowledge expanding relentlessly, proof by proof. And I could add to it. I majored in math in college and went on to get my PhD. My thesis was on algebraic number theory, a field with roots in all that
factoring I did as a child. Eventually, I became a tenure-track professor at Barnard, which had a combined math department with Columbia University.

And then I made a big change. I quit my job and went to work as a quant for D. E. Shaw, a leading hedge fund. In leaving academia for finance, I carried mathematics from abstract theory into practice. The operations we performed on numbers translated into trillions of dollars sloshing from one account to another. At first I was excited and amazed by working in this new laboratory, the global economy. But in the autumn of 2008, after I’d been there for a bit more than a year, it came crashing down.

The crash made it all too clear that mathematics, once my refuge, was not only deeply entangled in the world’s problems but also fueling many of them. The housing crisis, the collapse of major financial institutions, the rise of unemployment—all had been aided and abetted by mathematicians wielding magic formulas. What’s more, thanks to the extraordinary powers that I loved so much, math was able to combine with technology to multiply the chaos and misfortune, adding efficiency and scale to systems that I now recognized as flawed.

If we had been clear-headed, we all would have taken a step back at this point to figure out how math had been misused and how we could prevent a similar catastrophe in the future. But instead, in the wake of the crisis, new mathematical techniques were hotter than ever, and expanding into still more domains. They churned 24/7 through petabytes of information, much of it scraped from social media or e-commerce websites. And increasingly they focused not on the movements of global financial markets but on human beings, on us. Mathematicians and statisticians were studying our desires, movements, and spending power. They were predicting our trustworthiness and calculating our potential as students, workers, lovers, criminals.

This was the Big Data economy, and it promised spectacular gains. A computer program could speed through thousands of résumés or loan applications in a second or two and sort them into neat lists, with the most promising candidates on top. This not only saved time but also was marketed as fair and objective. After all, it didn’t involve prejudiced humans digging through reams of paper, just machines processing cold numbers. By 2010 or so, mathematics was asserting itself as never before in human affairs, and the public largely welcomed it.