The Open Planning Project

Our readers know how impactful civic-minded journalism can be, but it is also an honor to receive outside confirmation from esteemed organizations. Two TOPP projects were recently singled out as true movers in their fields:

GothamSchools won first place in the national Educational Writers Association's 'best education blog' category for their excellent coverage of the New York City education scene. This prestigious contest annually honors the best education reporting in print and broadcast media, and it is the only award of its kind. GothamSchools rose to the top among more than 300 entries, and we feel very proud to be recognized and to find ourselves in such esteemed company. Congratulations to the reporting staff: Anna, Elizabeth, Maura, and Philissa.

Out on the other coast, the San Francisco Bicycle Coalition has announced that Streetsblog San Francisco and SFBC Volunteer Michael Helquist are the recipients of this year's Golden Wheelie Awards. This honor recognizes transportation heroes in the Bay Area, and in their announcement the SFBC explained that they made the award to Streetsblog due to the blog's "intelligent journalism that is leading the conversation and helping more people understand the connection between bicycling, great streets and a livable city." Bryan, Matthew, and Michael, well done and keep up the great work.

 

In preparing some data for our next round of training courses, I spent a fair amount of time today processing and cleaning the US Federal Electoral Commission (FEC) database for 2008. The FEC is extremely good about releasing their data, even though it looks like they have to dump it out of a very old database system.

I processed the three main files, and then converted the associated code tables into side tables, so the whole thing is pretty self-contained and hopefully self-explanatory. I had originally hoped to fully replicate something like the FundRace site from 2008, but since the FEC data only has zip-code as a location entity, that is not going to happen this time around. I assume the FundRace folks also had access to a nationwide telephone directory or some other way of taking name and zip-code and using that to leverage out an actual street address.

If you are interested in playing with the FEC data and don’t feel like spending a couple hours mucking about in Perl to get it into tables, I’ve placed a PostgreSQL dump file online. Candidates are linked to individuals via committees. The FEC model has a lot of complexity hiding in it, with some committees not associated with candidates, and so on, so using the data correctly will probably require a little care.

Between 2001 and 2009, the share of trips that Americans made in cars dropped by more than four percent, with walking, bicycling and transit use picking up the slack, according to new data from the U.S. Department of Transportation.

Last year, 11.9 percent of all trips were on foot or by bike, while 4.2 percent of trips were on transit. Both figures signify major increases.

The National Household Travel Survey, the source of the new stats, is the gold-standard for transportation data. As Mobilizing the Region reported, while the Census only tracks how people get to work, the NHTS gathers data on all trips taken. It also distinguishes between, say, driving to a park-and-ride bus area and walking to the local bus stop.

The downside to the NHTS is how infrequently the survey is conducted, which makes it difficult to determine how much the 2009 data reflects a larger trend, and how much may be due to temporary changes brought on by fluctuating gas prices and the recession.

The high quality of NHTS data means that it can supplement NYC DOT's own numbers, which have shown a large rise in cycling over the same period. We've put in a request to the state DOT and the New York Metropolitan Transportation Council to get access to city-level data once it becomes available. 

San Francisco recently established a new policy requiring open source software to be considered equally with commercial software within the city’s procurement process. It’s important to note the actual inclusion of the word “equal” in this policy. Emphasis here is mine:

The Software Evaluation Policy will require departments to consider open source alternatives, when available, on an equal basis to commercial software, as these may reduce cost and speed the time needed to bring software applications to production.

— San Francisco Open Source Policy

This is much like the legislation passed in Vancouver last May:

Open Source Software – the City of Vancouver, when replacing existing software or considering new applications, will place open source software on an equal footing with commercial systems during procurement cycles;

— Vancouver Open Source & Open Data Policy

Back in September, I think Portland actually initiated the “First-In-Nation Open Source Software Policy for City Government,” but the language in Portland’s resolution is definitely not as strong:

Establish best practices for analysis of business requirements in software review and selection processes, identify existing commercial software systems with licenses that are scheduled to expire in the near future, and encourage the consideration of Open Source Software in the review, replacement and continual improvement of business solutions;

— Portland Open Source & Open Data Policy

Portland’s resolution should be amended from “encourage the consideration” to “require equal consideration” and other cities should make sure that they provide measurable policies for using Open Source rather than simply to “encourage consideration.”

These developments are huge and these cities deserve to be lauded as great pioneers, but we also need to help support them and to spread these kinds of policies to other cities. You can learn more and contribute to the creation of resources for open cities with the nascent OpenMuni project.

The quest for open transit data in New York continues, but the Times’ coverage today of the upcoming launch of the MTA’s new website gives cause to be optimistic. As the Times reports, the MTA is set to launch a redesign of its website this Wednesday, giving the agency’s site a much needed — and appreciated — overhaul. The overall design of the site looks to be greatly improved, and the subway service status on the front page is alone reason to celebrate, as anyone who’s been bitten by weekend service changes will surely understand.

Another welcome change is the addition of the trip planner to the front page. Interestingly, the default option now uses Google’s transit planner, though the screenshots reveal that you’ll also be able to plan trips using either Trips 1-2-3 or the in-house MTA trip planner.

The most exciting part for open data geeks though is this promising morsel:

The new site will also make it easier for outside software designers to get free access to system timetables and routes.

The article contains no further information about what this means, though the screenshot does show a “Developer Resources” link on the lower right-hand corner of the page.

The MTA has hinted for a while at changes to its developer and licensing policies, but beyond the cessation of legal threats last August, there’s been virtually no public announcements on the topic. Many people, including those of us here at TOPP who founded the NY Open Transit Data group, have long advocated and worked to open up New York’s transit data. We’ve had increasingly positive interactions with the MTA, particularly since the arrival of chairman and CEO Jay Walder last October, but are still waiting to see results.

It’s unlikely that the launch on Wednesday will be perfect, but I think it will prove to be a significant step toward the goal so many of us share: universal access to free, complete, and up-to-date transit data for New York.

It looks like it’s going to be a good week for open data.

At last night’s PlanningTech workshop, we diagrammed out planning processes to see where the insertion of technology might have been helpful.  The processes we looked at ranged from seemingly simple (getting a bike rack installed on your street) to massively complex (advocating for funding for MTA’s capital plan). Keep an eye out for digitized versions of the diagrams, as well as the opportunity to help expand them.

One idea that stood out, and that solicited some good discussion, was the notion of a “living plan” — a planning document that, rather than being static and falling out-of-date quickly, would evolve as a community’s context, needs, and priorities evolved.  Of course, such a plan would be unthinkable in an analog world, and seems  ridiculously obvious in a wikipedia world.

But that’s not to say it would be easy to implement.  Last night’s group raised a handful of questions, including: “what about dealing with large, complex expensive projects? (e.g., transit expansion)” and “what about blowback from early participants when the plan changes over time?”, and “what about ‘planning fatigue?’”.  All important questions to ask, and I won’t try to answer them here.  But the gist of the idea was that given a set of guiding principles (a constitution, if you will), a “living plan” might allow for more flexible planning and easier decisionmaking down the line.

It was good timing, then, that I came across this post on Streetsblog, covering some potential street design changes around Brooklyn’s Grand Army Plaza.  There’s debate over a DOT proposal to remove eastbound parking on Union Street and add a traffic lane, in order to reduce traffic buildup entering the plaza.  Community advocates are concerned that such a proposal won’t solve the fundamental “traffic vortex” problem, and doesn’t take into account the broader goals of creating a more livable and “world class” public space at GAP.

One section by the Grand Army Plaza coalition’s Robert Witherwax stood out to me:

All the tweaks to the plaza, so far, have been consistent with the planning principles GAPCo and its partners have promoted. The problem, says Witherwax, is the city’s piecemeal approach, which the Union Street proposal has cast into sharp relief. “DOT has been an excellent partner,” he said. “It’s not so much that what they have done, or are proposing, is bad — it’s that they aren’t going far enough.”

Witherwax is calling for a “buildable master plan” — a blueprint that would help guide planning and transportation decisions throughout the plaza area according to consistent goals. “Once you have that structure in place, you can say what happens if you do X, Y, and Z over here,” he said. But to date, he added, DOT has resisted the idea of a comprehensive plan.

Perhaps a “living plan” is somewhere in between the current, incremental planning that DOT is practicing an Witherwax’s notion of a “buildable master plan” — something that nurtures the development of guiding principles and longer-term projects, while still allowing for short-term experiments and responsive design.

This whole discussion makes me wonder how technology, in the form of a living plan or otherwise, might inform the debate here.  I can think of a few ways, to start:

• A website that visualizes competing plans or visions for the future of GAP, where each idea could be discussed, analyzed, and tweaked.
• An online traffic simulator (for all of Brooklyn?!) that lets anyone make tweaks and see the impacts, in a sim-city sort of way.
• A tool that explains, via an interactive diagram, the planning process as it’s currently laid out.  Who the decision makers are, where the input points are, and what the schedule is.
• Anything that generally increase sthe “touchability” of the proposed changes, moving them away from the abstract, wonky world. Perhaps this is something that our friends at the Environmental Simulation Center would be able to help with.
• (insert your idea here)

<shameless-plugs>
If you dig this kind of thing, head over to the PlanningTech google group and come to one of our upcoming meetups.  If you want to help TOPP and others build tools like these in the near future, check out the (as yet vaporware) Cosm project.
</shameless-plugs>

TOPP is looking for a talented IT manager. This is an exciting position that will oversee the infrastructure of a unique technology-driven non-profit. Please take a look at the full job description.

Last night’s New York Public Transit Data Summit was a resounding success. We had a room packed with passionate and thoughtful people eager to help make public transit in New York more efficient, accessible, and and easy to use.

For more than two hours, the group — comprised of over two dozen transit advocates, mobile and web developers, urban planners, lawyers, and open government supporters — discussed the current climate for developing transit applications, its shortcomings, and how the community can work with the MTA to improve things.

While no representatives from the MTA were able to attend, the MTA did provide a statement clarifying their current licensing policies. The statement answered many questions but also raised some more. I’ve forwarded those questions on to the MTA and will post their responses as soon as I get them.

Here are some specific items that came out of last night’s meetup:

1. We’ve started compiling notes and thoughts on the Ideas wiki. Check it out at http://nytransitdata.org. Anyone can register and contribute freely.

2. The group decided on #nytransit as the best hashtag, since it’s not too specific, not too general, and not too long.

3. Everyone wanted to keep the momentum going and stay in contact. Since there are already a number of mailing lists and groups working on related projects, we decided not to start another Google group. If you want to stay in contact please join the meetup, and keep an eye on the Open Government NYC, Transit Developers, and Civic Hacker mailing lists.

4. There are lots of tangible next steps to work on. If you care about seeing this issue move forward, please lend a hand. Help draft our one-page issue summary. Fill in holes on the wiki. Fix typos. Tell other developers and interested friends, colleagues, and groups. Get the word out by blogging and tweeting.

Let’s make sure that the wonderful momentum we all built last night doesn’t fade. Together we can help New York take the lead in providing access to transit data and enable a flood of innovation that will revitalize the city’s transportation infrastructure.

The effort to open municipal data is an initiative with momentum. Inspired in part by the transparency mandate on the federal level that gave us the first ever White House CIO and data.gov, cities across the country are opening up.

One city in particular set the scene before this all hit the national stage: Washington D.C. has delivered precedents like the first online city data directory, the first open API for 311 service requests, and open invitations for developers to produce apps with city data using initiatives like Apps for Democracy.  D.C. has laid the groundwork for a national model by delivering some killer apps. Previously the CTO for D.C., Vivek Kundra is now the national CIO. This model has not yet been fully embraced by New York City, but it’s getting close. In fact the New York State Senate Office of the CIO has been paving the way for opening government data on the state level.

Historically a trend setting city, New York is trying to catch up and even exceed precedents for opening city data. A City Council bill sponsored by Gale Brewer, Int 991, mandates that every city agency will make its data easily and publicly accessible in its raw structured digital form. As a mandate, this legislation would be the first of its kind. Even in D.C. there is only a general policy to release data, not an official mandate. At the public hearing for the bill, a plethora of supporting testimonies were given from the likes of the NY State Senate CIO, the W3C, and the Software Freedom Law Center. While a representative from the Mayor’s office stated that the Mayor supported the spirit of the bill, he expressed that it was too comprehensive and expensive to be practically possible in its current form.

At the Personal Democracy Forum just a few hours before the hearing, Mayor Bloomberg announced the creation of NYC Big Apps, a contest modeled after Washington D.C.’s Apps for Democracy that would let outside developers create applications on top of city data. Given some of the varying approaches to opening city data, the next day Vivek Kundra offered advice to help direct the effort in New York City. The City Council legislation is set to be voted on in early fall.

According to the announcement for Big Apps, 80 of the city’s datasets will initially be made available. The city is currently seeking requests for expressions of interest (RFEI) to determine which datasets should be a priority to open. One problem with access to city data is that it can be difficult to see what data is useful without knowing what data is available. It can also be difficult to know what applications would be useful to build without knowing what New Yorkers want.  To help gauge interest, a Peter Corbett of iStrategy Labs created a simple public venue to let people add or vote for the apps they’re interested in. If you’re interested in submitting a full RFEI, there’s also a process for doing that while keeping it in the public record. The task of showing what data is available is more of a challenge.

New York City doesn’t provide a public directory of the data that each agency maintains. Much of this information used to be available online and the city once even published a full Public Data Directory. Unfortunately, the first and last version of this directory was published in 1993, but it may still provide some utility.  Much of the the city’s IT infrastructure is slow to evolve, so many parts of the 1993 directory are likely unchanged. Regardless of how current it is, the 1993 directory offers a solid framework for developing an up to date version.  An effort is underway to digitize the full directory, allow the public to update it, and submit improvements to the soon to be launched National Data Catalog, a citizen maintained version of Data.gov. With the help of Eric Mill from the Sunlight Foundation’s Sunlight Labs, the challenge of digitizing the NYC Public Data Directory has been broken into small tasks that anyone can contribute to.

One other caveat to opening city data is that some of the agencies perceived to be run by the city are actually run as state agencies. The MTA for example is a quasi-public private enterprise overseen by New York State. While many of the most wanted apps relate to MTA transit, neither the City Council legislation nor the Mayor’s Big Apps program are able to provide MTA data. Because of some of the challenges of accessing and using public transit data, TOPP Labs hosted a New York Public Transit Data Summit to help develop clear and mutually beneficial policies for the relationship among transit agencies like the MTA, their riders, and application developers.

In many cases, the challenge of opening city data is coming up with a sensible policy that sets terms which ultimately provide superior city services. The task of drafting these policies is something that requires a proactive collaboration between developer communities and city government. With sound policy, the challenge then becomes for the city to provide the data in a well structured and timely manner. This too is a place where developer communities can contribute, even if it’s just by doing a task for digitizing the Public Data Directory.  Technology companies can also contribute. Google could help digitize the city’s 6 billion pages of documents much like it has been digitizing the world’s libraries. Amazon has already offered to host public datasets for free. What data would you like to see, what applications would you like to use?

With your help, we can open up the Big Apple.

If you’d like to suggest applications or data for the city to make available, you can add and vote for them at http://bit.ly/bigideas. To submit your own RFEI and make it public, you can add it to http://bit.ly/getnycdata. To help update the Public Data Directory you can first help digitize the old version http://bit.ly/digitizepdd

As a point of reference for those unfamiliar with Community Almanac, here are screenshots of the previous version’s home page and almanac view.

We’re very excited about the new version of the Orton Family Foundation’s Community Almanac that we’ve been working on at TOPP Labs. In this post I’ll share some of the design decisions that went into that project and some of the reasons why things have changed.

The Opening Line

In order to make it easier for users to contribute to the site, we decided to ditch the previous workflow—an impeding multi-step wizard that walked you through registering and adding your story—and we replaced it with a more unobtrusive workflow. Now users can jump right in and start adding whatever content they want in any order they like, even before they’ve registered or signed in.

Below are some of the wireframe mockups outlining this new just-in-time workflow:

Note: There are a few great ideas in these mockups that didn’t make it into this iteration: namely user profiles and an achievement rewards system. Maybe they’ll make it into a future release?

The Well-Worn Page

In addition to improving the workflow, we wanted to push a new book/page metaphor throughout the site. What was previously referred to as a story is now a page. Users add pages to their community’s almanac, or book, with the site as a whole is presented more as a collection of almanacs or library of books.

Not only was the term “story” inaccurate, as users can publish whatever type of content they like—text, images, maps, audio, video, or PDFs, none of which are necessarily narrative—but the term “page” is also more straightforward and familiar. Users are accustomed to viewing and adding “pages” on a website. Additionally, extending the “page” metaphor to include books and a library provided a more cohesive aesthetic direction and was also just more fun.

The Carefully Considered Cover

Once our ideas for the new user workflow and bibliophilic metaphors were established, I kicked off the visual design with a mood board in order to set the general tone for the look & feel:

The mood board sought to create a warm, inviting, and familiar vibe based that reflects the “heart & soul” sentiments of the Orton Foundation by referencing traditional, tactile imagery.

With this mood board and the book page metaphor in mind, I began sketching ideas for the basic structure and layout. The final sketch depicts an off-centered book where the recto page contains the primary content and the verso page acts as a sidebar for secondary content.  The charm of this layout resides in its flexible width. Although it behaves as a fixed-width layout, it’s not. In larger windows the sidebar items float beside each other, forming columns across the left-hand page. If your monitor is ridiculously large you can even see the whole book!

This fluid layout inspired quite a few bells and whistles, not to mention a couple of fun Easter eggs. The informational tour has been moved to home page as a little slideshow-style presentation that directs users to the map or signup. There’s also a cool slide-down login area and shuffling stack of almanacs on the home page. Check out the parallax scrolling that happens in the header illustration while you resize the width of the browser window. And we didn’t stop there! Who knows what else you’ll find… But hopefully you’ll never see these witty error pages in action: 400 (Client Error), 500 (Server Error).

Finding Your Community: The Map Workflow

While we were really excited to launch all these new features,we knew that the current map workflow wasn’t quite right and needed to be changed. In order to solicit user feedback on the other new features, we decided to launch in iterative stages.

In order to find a community in the current map workflow we perform two geocode requests. First, we geocode based on what the user types. Then, if there’s a valid result, we find the latitude and longitude given back by the first result and reverse geocode it. With this method, we only get cities or towns as the result of searching for neighborhoods or smaller communities and can thus limit the fragmenting of communities by creating a canonical, publicly-owned almanac for each location.

Now almanacs are now publicly owned, rather than moderated by a particular user. On the old Community Almanac site the user that starts an almanac would personally own that almanac. If, for example, the user “Hatfield” started a Tug Fork River almanac he could moderate out the user “McCoy”. Then “McCoy” could in turn start another warring Tug Fork River almanac. While this new map workflow is great for creating canonical almanacs, it does expose another problem…

Finding Your Community: Neighborhoods

Sometimes prominent communities are geographically contained within the cities/towns that a map search would return. For instance, a community such as Third Ward, Texas—a neighborhood within Houston, Texas—will geocode simply as “Houston.”

So, in our first iteration, we’ve included a help link under the map for users having trouble finding their community. As a temporary solution, if a user cannot find their community on the map, they can contact us and have their almanac created for them manually. This method isn’t really optimal. What’s needed is a search method that will return canonical results within varied levels of specificity. In our next iteration, we plan to make the map workflow more robust and limit manual intervention.

Finding Your Community: Polishing the User Experience

We had to step back and take a look at our options, and among the miscellaneous fields returned by Google are four gems:

1. Locality (city/town)
2. State
3. AddressLine
4. Accuracy

We’re currently only using the first two, Locality and State but what is potentially useful is AddressLine combined with Accuracy. An AddressLine is what google recognizes as a feature of a specific point. This can be very specific (like “Columbus Circle”), a little more general (like “Central Park”), on up to the city, state, or country that the point is contained in. Accuracy is an approximation of the size of a feature, and helps us clue in to what neighborhoods the point is referring to. It’s not perfect, but it gives us some choices we can present to the user to let them decide.

To use a search for “third ward houston texas” as an example, if we discard any AddressLine results with an Accuracy greater than 5 or less than 3 (to pick limits based on this one search), we can offer up the following choices to the user:

• Greater Third Ward (with an accuracy of 4)
• South Central (with an accuracy of 4)
• Houston (with an accuracy of 4)
• Greater Houston (with an accuracy of 3)

Almanac moderation may still be needed since AddressLine is far less predictable than current city/town results. But if we check them against existing almanac titles, we can hopefully remove any manual intervention for all users, offering neighborhood-level specificity to almanac search and creation.

So, Where’s Your Community?

Dig in. Find your community on the map and start adding to its almanac. Share your heart & soul! We think this redesign turned out great and we’d love to get your feedback. Let us know what you think.