Open data in Chicago: progress and direction

In a wonderfully comprehensive overview of Government 2.0 in 2011 up at the O’Reilly Radar blog Alex Howard highlights “going local” as one of the defining trends of the year.

All around the country, pockets of innovation and creativity could be found, as “doing more with less” became a familiar mantra in many councils and state houses.

That’s certainly been the case in the seven-and-a-half months since Mayor Emanuel took the helm in Chicago. Doing more with less has been directly tied to initiatives around data and the implications they have had for real change of government processes, business creation, and urban policy. I’d like to outline what’s been accomplished, where we’re headed and, importantly, why it matters.

The Emanuel transition report laid out a fairly broad charge for technology in his office.

Set high standards for open, participatory government to involve all Chicagoans.

In asking ourselves why open and participatory mattered, we developed the following four principals. The first two, fairly well-established tenets of open government; the last two, long-term policy rationales for positioning open data as a driver of change.

First, the raw materials. Chicago’s data portal, which was established under the previous administration, finally got a workout. It currently hosts 271 data sets with over 20 million rows of data, many updated nightly. Since May 16 the portal has been viewed over 733,201 times and over 37 million rows of data have been accessed.

But it’s the quality rather than the quantity that’s worth noting. Here’s a sampling of the most accessed data sets.

• TIF Projection Reports
• Building Permits (2006 to present)
• Food Inspections
• Vacant and Abandoned Buildings Reported
• Crimes – 2001 to Present (more block-level crime data than any other city, updated nightly)

Here’s a map view of the installed bike rack data set.

As a start towards full-fledged performance management, we launched cityofchicago.org/performance for tracking most anything that touches a resident: hold time for 311 service requests, time to pavement cave-in repair, graffiti removal and business license acquisition, zoning turnarounds, and similar. Currently there are 43 measurements, updated weekly. Here’s an example for average time for pothole repair.

To be sure, all this data can be inscrutable to residents. (One critic of the effort called it “democracy by spreadsheet”.) But the data is merely a foundation, not meant as a end in itself. As we make the publication of this data part of departments’ standard operating procedure the goal has shifted to creation of tools, internally and in the community, for understanding the data.

As a way of fostering development of useful applications, the City joined its data with Chicago-specific sets from the State of Ilinois, Cook County, and the Chicago Metropolitan Agency for Planning to launch an app development competition. Anyone with an idea and coding chops who used at least one of the City’s sets was eligible for prize money put up by the MacArthur Foundation.

Run by the Metro Chicago Information Center, Apps for Metro Chicago was open for about six months and received over 70 apps covering everything from community engagement to sustainability. (You can find the winners for the various rounds here: Transportation, Community, Grand Challenge.)

Here are some of my favorite apps created from the City’s open data.

• Mi Parque – A bilingual participatory placemaking web and smartphone application that helps residents of Little Village ensure that their new park is maintained as a vibrant safe, open and healthy green space for the community.
• SweepAround.us – Enter your address, receive an email or text message letting you know when the street sweepers are coming to your block so you can move your car and avoid getting a ticket.
• Techno Finder – Consolidated directory of public technology resources in Chicago.
• iFindIt – App for social workers, case managers, providers and residents to provide quick information regarding access to food, shelter and medical care in their area.

The apps were fantastic, but the real output of A4MC was the community of urbanists and coders that came together to create them. In addition to participating in new form of civic engagement, these folks also form the basis of what could be several new “civic startups” (more on which below). At hackdays generously hosted by partners and social events organized around the competition, the community really crystalized — an invaluable asset for the city.

Beyond fulfilling a promise from the transition report, why is any of this important? The overarching answer is not about technology at all, but about culture-change. Open data and its analysis are the basis of our permission to interject the following questions into policy debate: How can we quantify the subject-matter underlying a given decision? How can we parse the vital signs of our city to guide our policymaking?

The mayor created a new position (unique in any city as far as I know) called Chief Data Officer who, in addition to stewarding the data portal and defining our analytics strategy, is instrumental in promoting data-driven decision-making either by testing processes in the lab or by offering guidance for problem-solving strategies. (Brett Goldstein is our CDO. He is remarkable.)

As we look to 2012, four evolutions of open data guide our efforts.

The City-as-Platform

There are a variety of ways to work with the data in the City’s portal, but the most flexible use comes from accessing it via the official API (application programming interface). Developers can hook into the portal and receive a continuously-updated stream of data without manually refreshing their applications each time changes happen in the feed. This changes the City from a static provider of data to a kind of platform for a application development. It’s a reconceptualization of government not as provider of end user experience (i.e., the app or service itself), but as the provider of the foundation for others to build upon. Think of an operating system’s relationship to the applications that third-party developers create for it.

Consider the CTA’s Bus Tracker and Train Tracker. The CTA doesn’t have a monopoly on providing the experience of learning about transit arrivals. While it does have web apps, it exposes its data via API so that others can build upon it. (See Buster and QuickTrain as examples.) This model is the hybrid of outsourcing and civic engagement and it leads to better experiences for residents. And what institution needs a better user experience all around than government?

But what if all City services were “platform-ized” like this? We’re starting 2012 with the help of Code for America, a fellowship program for web developers in cities. They will be tackling Open 311, a standard for wrapping legacy municipal customer service systems in a framework that turns it too into a platform for straightforward (and third-party) development. The team arrives early in 2012 and will be working all year to create the foundation for an ecosystem of apps that will allow everything from one-snap photo reporting of potholes to customized ward-specific service request dashboards. We can’t wait.

The larger implications of platformizing the City of Chicago are enormous, but the two that we consider most important are the Digital Public Way (which I wrote about recently) and how a platform-centric model of government drives economic development. Bringing us to …

The Rise of Civic Startups

It isn’t just app competitions and civic altruism that prompts developers to create applications from government data. 2011 was the year when it became clear that there’s a new kind of startup ecosystem taking root on the edges of government. Open data is increasingly seen as a foundation for new businesses built using open source technologies, agile development methods, and competitive pricing. High-profile failures of enterprise technology initiatives and the acute budget and resource constraints inside government only make this more appealing.

An example locally is the team behind chicagolobbyists.org. When the City published its lobbyist data this year Paul Baker, Ryan Briones, Derek Eder, Chad Pry, and Nick Rougeux came together and built one of the most usable, well-designed, and outright useful applications on top of any government data. (Another example of this, from some of the same crew, is the stunning Look at Cook budget site.)

But they did not stop there. As the result of a recent ethics ordinance the City released an RFP to create an online lobbyist registration system. The chicagolobbyists.org crew submitted a proposal. Clearly the process was eye-opening. Consider the scenario: a small group of nimble developers with deep subject matter expertise (from their work with the open data) go toe-to-toe with incumbents and enterprise application companies. The promise of expanding the ecosystem of qualified vendors, even changing the skills mix of respondents, is a new driver of the release of City data. (Note I am not part of the review team for this RFP.)

One of the earliest examples of civic startups — maybe the earliest — is homegrown. Adrian Holovaty’s Everyblock grew out of ChicagoCrime.org, which itself was a site built entirely on scraped data about Chicago public safety.

For more on the opportunity for civic startups see Nick Grossman’s excellent presentation. (And let’s not forget the way open data — and truly creative hacker-journalists — are changing the face of news media.)

Predictive Analytics

Of all the reasons for promoting a culture of data-driven decision-making, the promise of using deep analytics and machine learning to help us isolate patterns in enormous data sets is the most important. Open data as a philosophy is easily as much about releasing the floodgates internally in government as it is in availing data to the public. To this end we’re building out a geo-spatial platform to serve as the foundation of a neighborhood-level index of indicators. This large-scale initiative harnesses block- and community-level data for making informed predictions about potential neighborhood outcomes such as foreclosure, joblessness, crime, and blight. Wonkiness aside, the goal is to facilitate policy interventions in the areas of public safety, infrastructure utilization, service delivery, public health and transportation. This is our moonshot and 2012 is its year.

Above, a very granular map from early in the administration isolating food deserts in Chicago (click image for larger). It is being used to inform our efforts at encouraging new fresh food options in our communities. This, scaled way up, is the start of a comprehensive neighborhood predictive model.

Unified Information

The same platform that aggregates information geo-spatially for analytics by definition is a common warehouse for all City data tied to a location. It is, in short, a corollary to our emergency preparedness dashboards at the Office of Emergency Management and Communication (OEMC), a visual, cross-department portal into information for any geographic point or region. This has obvious implications for the day-to-day operations of the City (for instance, predicting and consolidating service requests on a given block from multiple departments).

But it also is meaningful for the public. Dan O’Neil recently wrote a great post on the former Noel State Bank Buiding at 1601. N. Milwaukee. It’s a deep dive into the history of a single place, using all kinds of City and non-City data. What’s most instructive about the post is the difficulty in aggregating all this information and the output of the effort itself: Dan has produced a comprehensive cross-section of a small part of the city. There’s no reason that the City cannot play an important role in unifying and standardizing its information geo-spatially so that a deep dive into a specific point or area is as easy as a Google search. The resource this would provide for urban planning, community organizing and journalism would be invaluable.

There’s more in store for 2012, of course. It’s been an exhilarating year. Thanks to everyone who volunteered time and energy to help us get this far. We’re only just getting started.