Back in the mid-2000s (before I was even born), policymakers envisioned a future where data lied at the core of all government decisions. At the time, decisions were still driven by tradition, politics, and intuition. One article from 2007 even described policymaking at the time to be like “driving through a dense fog in the middle of the night” (“The Promise of Data-Driven Policymaking” by Daniel Esty, Summer 2007). Yikes!

But not to fear: the data science boom was beginning! There were high hopes of utopian governments that would be fast, transparent, and evidence-based. Now, 15 years later, how much of that Revolution was actually realized?

Today, I’ll attempt to answer this question by comparing the vision from the 2007 article (linked above) and the progress explained in a 2018 article (“Study: States are getting smarter about data-driven policy” by Matt Parke, July 12, 2018).

Here’s the analysis I came up with:

Topic	THEN: 2007 Vision	NOW: 2018 Reality
Technology	Emerging tools (database, sensors)	Mature but uneven adoption
Data Gaps	Lack of integration	Better collection, but still needs improvement
Performance Metrics	Input/output heavy	More outcome-focused
Culture	Idealism & optimism	Pragmatic experimentation

While the technology and data coverage has gotten much better than before, there is still insufficient use of data-driven policymaking in our government. And sadly, morales have fallen. A lot. Just tonally, you can tell how hopeful policymakers were in 2007, and it appears as though that hype has gone away. In its place: realism.

Nowadays, we’re more honest with ourselves about the limits in creating a data-driven government. First of all, there’s a huge political barrier, and that’s that political incentives often override evidence. Then, on the technical side, we’ve become more aware of just how difficult the integration and standardization of data really is. Putting this in a positive framing, however, we’ve learned that data isn’t a replacement for judgement, but instead a tool to sharpen it. Turns out, data won’t create the utopias we had envisioned, but it’s still helped us get closer than we ever would’ve gotten without it.

In the end, I’d still consider our progress with data-driven policymaking to be a net positive. Definitely something we’ll need to work more towards, but still a net positive. Think of it this way: we might not have gotten a revolution, but we still got pretty solid evolution.