How Governments Are Using Big Data to Drive Policy

Big data isn’t just for the enterprise anymore. Governments at every level are increasingly using big data to influence policy decisions.

When Chicago’s homicide rate skyrocketed in 2012, an unlikely hero appeared. His name was Brett Goldstein, and he wasn’t a cop or even a lawmaker. He was Chicago’s chief data officer, and his solution for helping the city’s beleaguered police and municipal service departments was nothing more than two words: big data.

Goldstein proposed building an analytics platform that would aggregate data from the city’s 30 agencies and departments, allowing city officials to view maps showing everything from 911 calls to complaints about potholes. This solution, Goldstein hoped, would give decision-makers an unprecedented look at the patterns of crime, complaints, and civic issues that affect Chicago.

That platform would go on to become WindyGrid, one of the United States’ most comprehensive exercises in public data. Goldstein has since left his civic post for a fellowship at the University of Chicago’s Harris School of Public Policy – a fitting move for a man whose work may have permanently altered the way policy is made.


Chicago’s foray into big data may have been one of the most effective to date, but it’s hardly the only one. There are many who have taken the example of Chicago and launched programs. And the promise of big data has caught the attention of a number of federal agencies, including some that carry significant weight with policy makers.

The Defense Department’s Information Volume and Velocity initiative is designed to “extract data from all sources on the web, including tweets and blogs, using pattern recognition to detect trends in a sea of unstructured data.” That’s government-speak for “it trawls the web for a set of conflict-related phrases.” Despite the dense description, this initiative could have very real applications for predicting conflict at home and abroad.

Not to be outdone, the CIA’s Open Source Indicators program is similar to the DOD’s, but takes it a step farther. Its purported purpose is to “anticipate and/or detect societal disruptions, such as political crises, disease outbreaks, economic instability, resource shortages, and natural disasters” through the same means at its DOD counterpart.

Meanwhile, the United Nations’ Global Pulse program aims to explore “how new, digital data sources and real-time analytics technologies can help policymakers understand human well-being.”

Like Goldstein’s Chicago on a national—or global—scale, the analytical and predictive power of each of these initiatives could soon be driving policy. But, as always, there’s a catch.


Big data’s potential for positive change also comes with the potential for abuse. Wade Henderson, president and chief executive officer of the Leadership Conference on Civil and Human Rights, recently told the Associated Press: “While big data is revolutionizing commerce and government for the better, it is also supercharging the potential for discrimination.”

Henderson’s remark came in response to a White House discussion on employers’ increasing use of big data to screen job applicants. White House advisor John Podesta echoed that sentiment in a recent speech, noting: “it’s easy to imagine how big data technology, if used to cross legal lines we have been careful to set, could end up reinforcing existing inequities in housing, credit, employment, health and education.”

California’s use of PredPol software—short for “predictive policing—is an excellent example. PredPol’s system uses existing crime data to predict the types, locations, and frequencies of certain types of crime. Since adopting PredPol, robberies in California have decreased by a quarter. But despite its effectiveness, this sort of predictive modeling has raised a number of valid concerns. Many of its critics envision a Minority Report-style future in which different groups are punished or discriminated against, simply because they may commit a crime in the future.

Big data’s true potential to affect policy has yet to be seen. The gears of government turn slowly, and new initiatives take time to gain approval, let alone become active. As such, it may be a few years yet before we fully come to appreciate big data’s impact on public policy. Whether we’ll be grateful or regretful is anyone’s guess – but we do know the new world of work is here, if not arriving quickly, and transformation within government agencies and organization is already under way.

Leave a Comment