89 Wash. L. Rev. 1 (2014)
The Scored Society: Due Process for Automated Predictions

handle is hein.journals/washlr89 and id is 8 raw text is: THE SCORED SOCIETY: DUE PROCESS FOR
AUTOMATED PREDICTIONS
Danielle Keats Citron      & Frank Pasquale
Abstract: Big Data is increasingly mined to rank and rate individuals. Predictive
algorithms assess whether we are good credit risks, desirable employees, reliable tenants,
valuable customers-or deadbeats, shirkers, menaces, and wastes of time. Crucial
opportunities are on the line, including the ability to obtain loans, work, housing, and
insurance. Though automated scoring is pervasive and consequential, it is also opaque and
lacking oversight. In one area where regulation does prevail-credit the law focuses on
credit history, not the derivation of scores from data.
Procedural regularity is essential for those stigmatized by artificially intelligent scoring
systems. The American due process tradition should inform basic safeguards. Regulators
should be able to test scoring systems to ensure their fairness and accuracy. Individuals
should be granted meaningful opportunities to challenge adverse decisions based on scores
miscategorizing them. Without such protections in place, systems could launder biased and
arbitrary data into powerfully stigmatizing scores.
INTRODUCTION TO THE SCORED SOCIETY ......                                .......... 2
I.    CASE STUDY OF FINANCIAL RISK SCORING                              ........... 8
A. A (Very) Brief History of Credit Scoring Systems ................ 8
B.   The Problems of Credit Scoring          ...........    ........... 10
1. Opacity            ........................................ 10
2. Arbitrary Assessments                           ..................  11
3. Disparate Impact         ..................      ............. 13
C.   The Failure of the Current Regulatory Model................... 16
II. PROCEDURAL SAFEGUARDS FOR AUTOMATED
SCORING SYSTEMS                                      .............................. 18
A. Regulatory Oversight over Scoring Systems                     ........... 20
1. Transparency to Facilitate Testing .....               .......... 24
2. Risk Assessment Reports and Recommendations.......... 25
B.   Protections for Individuals                        ..................  27
Lois K. Macht Research Professor & Professor of Law, University of Maryland Francis King
Carey School of Law; Affiliate Scholar, Stanford Center on Internet and Society; Affiliate Fellow,
Yale Information Society Project.
Professor of Law, University of Maryland Francis King Carey School of Law; Affiliate Fellow,
Yale Information Society Project. Cameron Connah and Susan G. McCarty have given us enormous
research support. We are grateful to Ryan Calo, Lauren Watts, James Wendell, and the Washington
Law Review for including us in the Artificial Intelligence Symposium. We would like to thank
Roland Behm, Martha Poon, James Grimmelmann, and Neil Richards for generously offering
comments on a draft.

1

What Is HeinOnline?

HeinOnline is a subscription-based resource containing nearly 2,700 academic and legal journals from inception; complete coverage of government documents such as U.S. Statutes at Large, U.S. Code, Federal Register, Code of Federal Regulations, U.S. Reports, and much more. Documents are image-based, fully searchable PDFs with the authority of print combined with the accessibility of a user-friendly and powerful database. For more information, request a quote or trial for your organization below.



Short-term subscription options include 24 hours, 48 hours, or 1 week to HeinOnline with pricing starting as low as $29.95

Access to this content requires a subscription. Please visit the following page to request a quote or trial:

Already a HeinOnline Subscriber?