Earlier this summer, I had the pleasure of speaking at the Future of News and Civic Media conference at MIT.  It was great to be around such a creative and talented group of people, and as usual, this year brought with it a new batch of Knight News Challenge winners. (For a roundup of some conference takeaways, see Paul’s post).

The topic for my talk was “Data into Action” — in other words, how can we build on transparency and use data to create social and political impact?

This was the question posed by Chris Csikszentmihalyi, the director of MIT’s Center for Future Civic Media, and the host of the conference.  It’s a particularly relevant question now, because the past few years have seen a real explosion in public open data, including countless websites and apps that make use of it.  Built into Chris’ question is the assertion that perhaps we don’t fully know how best to make use of all of this data that’s becoming available to us.

My guess is that we are going see a big increase in the number of tools that make public data actionable.  Below is my initial take on how (you can also see the video and the slides from the conference).  This is admittedly not fully formed: I don’t claim to know how to “turn data into action” — rather, I just have a few ideas about some of the factors and how they’re shaping the “data into action” landscape.

Making data actionable

Making data actionable requires focused effort; data on its own isn’t enough.  So where does that effort come from and how does it help?   Each of the following groups can play a part, and I hope to see more initiatives involving collaboration among them:

• Government. Government can play a role, not only in producing data and making it available to the public, but also in accepting data in new ways.  Efforts like Open311 are attempting to create standards for read/write government APIs, and the idea of “government as a platform” is largely about considering and building out these data sets and APIs.  In most cases government alone cannot make data actionable, but they play a hugely important role in making it possible.  Side note: it’s worth mentioning that simply publishing a data set is not enough: in order to produce worthwhile action (and justify the expense of opening data), governments need to cultivate communities of users/developers around their data sets.
• Civic Hackers. In recent years, civic data mashup sites have provided new views into how cities and governments operate.  GovTrack is one of the most mature: a congressional data site that combines many datasets in order to provide insights into what’s happening in congress, it makes that data actionable by providing ways for people to stay apprised of updates (called “trackers”), and provides an API which many other sites have used. On more of a personal utility level, a new application called CabSense takes historical NYC taxi pickup and dropoff data, and uses it to provide recommendations on where best to catch a taxi.  For every corner of life, civically conscious and entrepreneurial endeavors are popping up to make data more actionable (GovTrack in fact runs a modest profit from ad revenue).
• Journalists. Journalists have always used research to provide consumer- or citizen-friendly insights.  In the age of mass data, the need for this hasn’t changed, though some of the methods have.  For instance: the Washington Post recently launched its “Daily Gripe” column, which uses incoming SeeClickFix issue reports as a basis for follow-up reporting.  The NY Times recently did an in-depth analysis of the same taxi data that powers CabSense.  As news organizations start using more real-time data, we can expect to see a big increase in interpretive data analysis.
• Nonprofits & interest groups. Like journalists, nonprofits and interest groups are particularly motivated to produce action out of data. Uncivil Servants, a 2007 project by OpenPlans, Greg Whalin (a dot-com startupper by day and civic hacker by night) and NYC’s bike/ped advocacy group Transportation Alternatives is an example of this working to great effect.  The website  crowdsourced illegal use of city-issued parking placards, and TA followed up on those reports with the city as part of their “Above the Law” [PDF] campaign.  The result was a major restructuring of parking placard policy in NYC.

Each of these groups — and I’m sure there are others I’m missing — can add layers of meaning and actionability to raw data sets, and is a critical part of the “data into action” ecosystem.

Combining the lessons from “data sites” and “action sites”

Above: screenshot from the Haiti Crisis Map

For the most part, we’ve seen innovation along these two separate, parallel tracks: sites that mash data up in creative ways (for example: Outside.in, Everyblock, BigAppleEd, and many of the sites I mentioned above), and sites that imagine new ways for people to interact and collaborate with one another (examples: The Extraordinaries, The Point, PledgeBank, GroundCrew, and on).  I believe that we’ll begin to see a convergence of automated “data sites” and social “action sites.” A frequent thread of criticism about data sites is that they can feel cold and lifeless; I expect that will change as the convergence happens.

That these are largely separate is of course not a hard and fast rule: tools are emerging that combine the two: most notably Ushahidi, InSTEDD and others’ response to the Haiti Earthquake in January.  And here in the US, SeeClickFix’s ”watch areas” are a clever way to make citizen issue reports actionable.  I know there are more examples like these (please link to the best ones in the comments).

The challenges in how to “do data” and how to get people to collaborate are substantial, and it’s understandable that they’ve been tackled mostly independently.  But I think the timing is such that more and more folks will begin drawing the lessons that each class of sites has demonstrated, and start tying the approaches together.

Bridging the civic divide

My last hypothesis is that we can “piggy back” higher-level civic engagement on top of more personally-oriented tasks.

I see perhaps the biggest opportunity for converting data into action in attaching “civic actions” to services that people use for personal reasons.  For instance: “where’s the bus?” has been a huge topic over the past two years — transit has been one of the hottest sectors of open government data, precisely because it serves a very important personal need: how to get around.  Rather than viewing this as an insignificant development, I prefer to view it as an opportunity to draw people in to more civically significant activities.

Washington DC is taking an interesting approach to this: at last month’s Online Engagement for Sustainable Urban Mobility event, DC CTO Bryan Sivak announced that they will be building out a network of QR codes on the DC Circulator bus system (another example of government building the “platform”, or laying the foundation for innovation).  These codes will allow for the expected location-specific interactions between bus riders and the bus system. But it’s only a short step to imagine piggy-backing on those interactions to introduce questions like “which of these changes to the bus system would be most helpful to you?”.

A more subtle example is WalkScore. I recently moved from NYC to Boston, and I used WalkScore to help decide where to live.  This data (how walkable are certain locations?) is immediately actionable (where do I choose to live?).  What’s especially clever is that WalkScore uses this helpful-on-a-personal-level data to stealthily introduce the civic action that they’re advocating (reducing car dependence and encouraging alternative transportation).

Like Maslow’s Hierarchy of Needs, people have a hierarchy of civic needs.  The basics need to be taken care of before “civic self actualization” is possible.  By building a foundation of engagements, beginning with the most personal and necessary, we can build a ladder to more significant civic actions.

What’s next

So, I think this is an interesting topic, and one that’s worth thinking about (and working on) in more detail.

I’ve been talking to Joe Edelman, CEO of Citizen Logistics (the makers of the awesome GroundCrew real-time organizing platform), and Chris Csikszentmihalyi from MIT about ideas for follow up.  The three of us would like to host an ongoing conversation on this topic, and invite others who are working in this space to join in.

To that end, we’re happy to announce the new Data into Action tumblog:

http://dataintoaction.org

We’ll be posting there as we traverse the interwebs; please also feel free to submit your own posts, or if you prefer tag sites on delicious with “dataintoaction.”  We’ve also established a Data Into Action google group, for open discussion:

http://groups.google.com/group/dataintoaction

That’s it; we will see you out there….

StreetFilms has produced a great video highlighting the case studies and benefits of open transit data and how that relates to the broader movement around Gov2.0. Along with websites like CityGoRound.org and the developer resources exemplified by transit agencies like Trimet, BART, MBTA, and the MTA this video does a great deal to show what can be done and what is being done to leverage more innovation and access to information around public transit. If you know of a transit agency or a developer that could use this kind of insight and inspiration, please pass it on. The video concludes by looking further into the whole transportation network by asking what else we could do with access to better data in regards to traffic, safety, and more.

The video features Chris Dempsey and Joshua Robin of the MBTA, MTA Chair Jay Walder, O’Reilly Media founder and gov2.0/web2.0 thought leader Tim O’Reilly, Zipcar co-founder Robin Chase, NYC Councilwoman Gale Brewer, Paul Steely White of Transportation Alternatives, as well as Nick Grossman and Nicholas Bergson-Shilcock of OpenPlans.

Please take a look and spread the word: http://www.streetfilms.org/a-case-for-open-data-in-transit/

A screenshot on traffic conditions in Lower Manhattan at 3:30 PM. (Google Maps for Android)

Yesterday morning Nick Grossman and I attended a developers’ meeting at the NYC Department of Transportation to brainstorm some ideas to better enhance the open data movement at the agency.  The room was very well-balanced in terms of attendance between developers and DOT officials, and there were some interesting requests from the development/programming community.

Real-time data and the consolidation/aggregation of NYC transportation data were two general areas of importance to the DOT and development/academic community.  In the long run, the DOT hopes to provide real-time traffic information along with improving delivery truck routing around low bridges on their data feed portal.  They mentioned that like many New Yorkers, Mayor Michael Bloomberg would prefer knowing the traffic conditions ahead of time and not getting delayed for meetings and events in Midtown traffic.  Better real-time traffic data will be the foundation for building apps to help New Yorkers navigate congested streets.  Moreover, better transportation apps could help people make more informed transportation choices, thereby building a more sustainable city.

Thus, the DOT is seeking the talents in the developer community to help build solutions.

The DOT has made good progress in unlocking their data feeds.  They are now tapping into the developer community to discuss the objectives and  strategies for future projects with DOT data.  The data feed portal contains advisories, speed restrictions, alternate side parking and  real time camera feeds.  To the extent that it’s possible, the DOT has shown interest in working with other agencies, such as the TLC and the MTA to share data.

• synchronized police investigations alerts
• taxi locations
• traffic light timing
• real-time street closure updates, ranging from construction sites to cultural streetfairs (probably through the DOT currently-internal Citywide Event Coordination and Management database)

This is rather a short list, but it would more interesting what other application ideas developers and the DOT can come up with.  Perhaps, some sort Uservoice or Ideascale page to keep track of a “wishlist” of applications the DOT and developers want to develop with the available data (similar to the one in Portland for CivicApps, Edmonton’s AppsforEdmonton or NYC’s own BigApps).  At a recent Gov2.0 event, Sarah Kaufman, a Projects Coordinator at MTA NYC Transit, mentioned that a developers’ wishlist will be published based on the MTA’s datasets.

While there is no existing executive or legislative mandate by the Mayor or City Council to standardize data for all agencies (yet), the DOT is moving ahead to release more datasets in preparation for BigApps II set for Fall 2010, following a very successful BigApps I launch last year.  DOT has highlighted that some of its difficulties will be digging through older or unformatted data, but they are willing to work with developers on data delivery and data content to make NYC traffic more sustainable and user-friendly.

To continue the conversation from the meeting, the DOT has established a Google Group for developers, just like the thriving MTA Developers Group.  Feel free to chime in with ideas to improve the datasets (like Steven Romalewski’s excellent MTA dataset post) or request datasets for application development.  This is an integral step for the DOT to gain momentum going forward and increase their presence in the development community.

We applaud the DOT’s effort in leading this initiative and becoming a leader in open transportation data.  It is a welcome commitment to empowering smarter transportation choices in New York City.

Last month, we hosted a one-day workshop with Manhattan Borough President Scott Stringer’s office called Speak Up, New York.  The purpose of the event was to bring together NYC community groups and “civic hackers” to discuss how community groups are using technology to organize and engage.  With a keynote from Deputy US CTO for Open Government Beth Noveck, presentations from E-Democracy, Neighbors for Neighbors, Transportation Alternatives, and others, and representation from many NYC community boards as well as projects like Localocracy and SeeClickFix, it was a great opening discussion.

In the spirit of building community, our first step is to continue to bring together the people who are interested in these issues, to see what kinds of things we can do together, starting right now.  Long term, OpenPlans and the MBPO are also interested in  building a series of web tools to empower community organizing in NYC, but the more important step is to build connections among the community of people we intend to serve.

With that goal in mind, we are kicking off a series of monthly meetups, where we’ll work together to develop tools and resources to empower local community organizing in NYC.  The next meetup is this coming Monday, July 26th, 6:30-8:30pm, here at OpenPlans.  Projects for Monday’s meetup will include (please click through to vote or leave a comment on each idea):

• Mapping out the communication tools that community groups are using today
• Creating a “how to guide” for starting a community group
• Creating a Public Registry of Community Organizations

To join the group and to RSVP for next week’s meetup, check out the meetup group here:

http://meetup.com/SpeakUpNY

We are excited to move forward with this great group of people, and we hope you can join us!

// Photo from the June 11 Speak Up, NY workshop by Robin Smith

A screenshot of Toronto's Open Data Site, which allows for collaboration about data sets.

Pioneering an open data policy on the government level is not the easiest initiative without support, according to a recent study by the Open Society Institute. It takes various levels of negotiations among parties of the public, elected officials and “civic hackers”.  Furthermore, some government administrations are more risk adverse than others, unsure on how to proceed from being closed to being open in the most cost-effective manner.   Most seek to replicate best practices from successful initiatives and improve on others through news stories and conferences, but never some thing tangible; a resource guide, such as our own OpenMuni wiki (still in development), could be more purposeful in understanding the processes of unlocking accessible and quality data.

The report, commissioned by Open Society Institute’s Transparency and Accountability Initiative and written by Becky Hogge, provided insights on the UK and US processes to unlocking their data to their respective data.gov’s.   The report at the outset seeks to define the feasibility of open data initiatives from both sides to Atlantic on middle and developing countries (such as China and South Africa).  The World Bank and United Nations data are already available (and they also shifted to more openness), but data from the source (the nation and various cities) would be nice.  Building from the U.S. and UK models, can cities and developing nations pursue the same strategies to open government data?

The report concludes that the US and UK approached a three-tiered system to successfully implement their open data initiatives:

1. Motivation from civic hackers or good government groups
2. An efficient and engaged “middle government” workforce (agencies, departments of the government)
3. Top-level support and mandate to encourage this change

While this approach may work for most developed nations or nations close to an economic steady state (neoclassical theory), Hogge suggests, through interviews with international leaders pursuing similar goals, that developing nations will need funding and guidance from international aid donors to achieve the inspiration and successful implementation.  These contributions can help deploy the initiative by creating competitions and educating the next generations of policy wonks and developers.   Many cities (even in the U.S.) lack a developer-base and are holding competitions to entice outside developers to fill the gap and generate open data-driven web applications after the release of several highly demanded datasets.  The economic impact has convinced many cities to release their data, but even this can vary from government to government.  Not many studies have been done to show significant evidence on the ROI of open data.   I recommend reading Dan McQuillian’s observations on the open data movement so we don’t approach an “emperor’s new clothes” (c.f. p. 23 of Hogge) and actually offer the social change we really need.

Nonetheless, Hogge argues that important part of any open data movement is the resulting utility of the data rather than the amount of users downloading (p. 19).    As in economics, one should also seek to maximize the social benefits through utility.  Applications that promote the closure of the digital divide will usually have the greatest impact.  For example, Chicago was awarded for the sustainable adoption segment of the Broadband Technology Opportunities Program (BTOP) of the federal stimulus package in order to assimilate populations without a computer or internet connection.  As Gov 2.0 expands, Chicago’s Civic 2.0 will focus on civic and public uses of the Internet, and it will teach adults and parents how to access civic data for local community improvement purposes and the use digital media tools for community organizing purposes.  An open data movement here can provide new apps and also evaluate the quality of life improvement from data collected.

Hogge’s study is a good start to the growing movement.  Inspired cities can create open cities with some help—collaboration. These cities will have to evaluate available resources before crossing the river. Having a wiki or central list of best practices (OpenMuni and Government 2.0 Wiki, for example) will be an invaluable instrument for cities to explore the various models of open data and then develop their own unique initiatives through civic engagement and collaboration.  The resulting trajectory builds a strong data eco-system towards greater social and economic gains.

You can read Becky Hogge’s Open Data Study in full at http://www.soros.org/initiatives/information/focus/communication/articles_publications/publications/open-data-study-20100519.

Sam Wong is a summer intern at OpenPlans, researching best practices in government open source and open data as part of our OpenMuni.org initiative. Sam previously served as Legislative Aide on Technology in Government for the New York City Council.

Localocracy has a nice way of expanding into new communities.  They avoid the “tumbleweeds” problem by building support for the initiative before it starts, in a kickstarter-ish sort of way.  This is, of course, not a new idea, but I think Localocracy does a nice job of executing on it.  E-Democracy has long used a similar system for building interest before groups launch.  It’s also reminiscent of the genius way that Threadless builds demand for t-shirts before deciding to print them.  I like the way that Localocracy uses the approach to both build excitement and guarantee some level of engagement once the sites go live.

Last night at an event here at OpenPlans, I spent a few minutes talking to Boštjan “Bos” Špetic, the founder of a web startup called Zemanta (a tool for finding related content to blog posts — for writers as well as readers).  Most of Bos’ team is based in Slovenia, where he is from.

We got to talking about OpenPlans’ work and the issue of Open Government in general. As it happens, Bos is on an advisory committee to the Slovenian government on Open Government, and will have an opportunity, come fall, to deliver recommendations on where they should direct their efforts.  He is looking for information on where to start, as it’s pretty much a blank slate for them right now.

First off, I don’t know anything about Slovenia’s government, policies, or infrastructure.  All I can do is point to some resources that are beginning to be developed here in the US, after a few years of real traction towards open government:

Coming from the top is the White House’s Open Government Initiative .  I can testify to the impact that this high-level mandate has had throughout the country; we see language from it in new initiatives, RFPs, etc every day.  Next, there are a number of books, notably O’Reilly’s recent Open Government, Beth Noveck’s excellent Wiki Government (which focuses on collaborative governance) and Personal Democracy Forum’s Rebooting America. And of course, there are a number of evolving online resources, including the Open Government Playbook, which acts as an index to lots of open government documentation, and the (very rough) Open Government Guidebook that’s being developed as part of our OpenMuni effort, and aims to become a step-by-step how to for some processes.

One of my favorite open government resources, that’s now admittedly a bit out of date, is the Public Transit Openness Index, a screenshot of which is at the top of this post.  What it does nicely is breaks the concept of “open” down into concrete, measurable steps, which can be compared across agencies.  It would be great to see something like this for open government more broadly, across functions and jurisdictions.

The White House’s Open Government Directive offers a start, and they’ve begun to track agencies’ progress, but what I’d love to see is an organic document that breaks open government down into pieces, and links it up to efforts that are underway or have already taken place.  Maybe this already exists?  The Playbook and Guidebook, above, are both moving in this direction, but I’ve yet to see a more granular and hyperlinked checklist like the Transit Openness Index. Checklists are great, and social checklists are even better.  Perhaps this is a project for WikiHow (which doesn’t yet have a government section), or maybe it’s something new.

Regardless, it’s clear that there’s still work to be done in packaging the learnings of open government, for ourselves and for folks around the world who are looking for models to follow.

Update: I neglected to mention Phil Ashlock’s excellent recent post covering the State of Open Government which covers this question in much more detail than I did here.

The New York City Department of Transportation needs your help! We want to provide more and more useful information to the traveling public about traffic and transportation. We know we’ve got a lot of the raw data, we need you to help us separate the wheat from the chaff.

Here at the DOT we’re plugged into all of the comings and goings of the city – we’re constantly monitoring highways, ferries, traffic cameras, bridges and road work. This means that we create a ton of data, some of which we share publically on our data feeds page. However, most of our data isn’t currently public.

There are many reasons why you (outside of the walls of government bureaucracy) will be able to build better apps, at ten times the speed and efficiency than we will ever be able to. We see our role as opening up and cleaning up our data to make it easier for you. Together we can provide the public with the information they hunger for, to make better decisions about how to move around the City.

To get started on this, we need your help to identify where the hot spots of demand are, so we can prioritize where direct our efforts to format and open up data. We also need you to help us start assessing where the gaps are between the data we have and the data we’d need to create truly useful “real-time traffic” apps. We’re at the beginning of this process, and the first step is to have an informal brainstorming meetup with developers. We’ll talk about some of the data feeds that we have, and hopefully you’ll come with some questions we haven’t asked yet.

The meeting is full and RSVPs are closed, but it’s not too late to get ideas in.  At the meeting tomorrow, I’ll also be encouraging DOT to host some sort of idea sourcing tool online.

Leave any ideas in the comments section here, or on today’s post on Streetsblog (where there are already a few comments coming in).

Our first goal is to make the OpenBlock software easier to install and use for newspapers in the 25k-100k circulation range.  If you or somebody you know works at such a paper, please take a look at this short survey. The results will help us plan our work in the first few months of the project.

Thanks!

A screenshot from NYC Department of Information Technology and Telecommunications' (DoITT) NYCityMap showing subway entrances/exits and their intersections (green globe), subway station, and the lines serving that station.

The MTA recently released a new dataset with exact latitude and longitude coordinates of station entrances and exits. This was a highly requested dataset for many transit developers and showcases the lastest development in the MTA’s transition from a data licensing model toward community collaboration in pursuit of more innovative applications. This shift has not only inspired programmers to develop much-needed applications that help citizens navigate the subway system, but is also a great example of confidence in the community’s ability to provide low cost innovations while many public transit agencies are facing fiscal challenges.

Last year, the MTA was exploring an exclusive data sharing contract with Google in order to better manage the usage and distribution of MTA transit data. At the same time, the delivery of transit data to private developers (often smaller and more local to NYC) was viewed as cumbersome because the data was only provided in CD-ROM format. This made it challenging to retrieve and keep the data updated in an efficient way. Seeing that other cities were finding some cost-savings by making data public, the MTA began to accommodate an emerging open data movement and leverage the expertise of the developer community.

Under the new MTA leadership of Chairman Jay Walder, the MTA has unveiled more than 50 datasets and held a developer  unconference. This event included software developers, good government advocates, and policymakers in order to facilitate an open exchange between the MTA and the developer community. At the unconference, Walder spoke of how access to information and technology will usher in a new era of mass transit for New Yorkers. Analogous to the “See Something, Say Something” announcements, Walder sought help from developers and promised to work with the community to release more data.

In the week before June 28 when new service modifications were implemented, the MTA released updated timetables and other schedule-related materials to developers to ensure applications would run smoothly through the change, and on July 2nd the oft-requested (and contentious) latitude and longitude coordinates for all individual subway station entrances and exits was released.

Here’s a look at what it contains:

• Longitude (to the sixth decimal place)
• Latitude (to the sixth decimal place)
• Division (BMT, IND, IRT)
• Line (by branches)
• Station name
• Entrance Type (Stair, Easement, Elevator)
• Route (letters and numbers)
• ADA (Yes/No)
• Low Turnstiles
• Hi Turnstiles
• Cross Sections relative to the Street Level (N/S, E/W)

The release of this data had apparently been contested for some time because of perceived security risks. The MTA has declined comment, but it is believed that the NYPD had seen the potential for malicious use of the data. What had confused many about this position was that the data was already displayed in many forms such as on NYCityMap, but was simply not available in raw digital form. Undercover emergency exits and tunnel evacuation locations remain withheld for safety reasons, but the main entrances and exits to the street level and closest ADA drop-off points are now available for developers to integrate into their applications and help riders better navigate the 660 miles of subway lines.

The benefits outweigh the overall concerns about this data. After all, that’s the purpose of the open data movement—releasing data that will enhance citizen use of public services but not put the public at risk. Previously, transit applications and mapping systems like Google Maps could only display the centerpoint of a station and direct riders to the general vicinity, but now applications can provide directions based on the specific entrances and exits of stations. With this additional dataset, we can plan our trips more accurately and even save the frustration of getting off the wrong car or at the wrong exit. Occasionally, this would also be helpful in getting to transfer points faster, especially buses. Such a dataset can also bring public input to topics such as improving station exits or re-evaluations of traffic flow (possibly combined with the turnstile data). Applications such as Exit Strategy and WayFinder have already exceeded expectations but could now do even more by providing true door to door directions. Ultimately, the best use of this data is likely to be an unexpected one, something that combines this information with the growing ecosystem of new data and applications.

At a recent Gov2.0 event Sarah Kaufman, a Projects Coordinator for MTA New York City Transit noted that the MTA will soon release a “wish list” of applications they hope the developer community to build with all the data they have released. Sarah provided a glimpse at this wishlist by emphasizing the need for better visualizations.

MTA data, such as exits and entrances, holds vast potential for new and creative mobile-based transit solutions. We look forward to the ongoing improvement of New York’s transit experience through collaboration with developers, commuters and the MTA.

Sam Wong is a summer intern at OpenPlans, researching best practices in government open source and open data as part of our OpenMuni.org initiative. Sam previously served as Legislative Aide on Technology in Government for the New York City Council.