Open All Night: The Great Urban Hack NYC

For 26 hours this weekend, a bunch of journalists and coders got together to make lots of great things designed to help the citizens of New York City.

My blog post summarizing the event and all of the resulting projects is upon Hacks/Hackers.

I helped Jenny 8. Lee, Chrys Wu and Stephanie Pereira organize the event. Then I joined a team working with digital heaps of NYC taxi trip data to make data visualizations and start some other projects. My favorite one is here (with a detail below), which is a representation of taxi usage for 24 hours, set around a clock. Beautiful. It was built by Zoe Fraade-Blanar using Processing and data crunched by the other teammates.

Click image for full view

I came away from the event with many new connections, excitement about learning Processing, some more skills in Sinatra and a note to check out Bees with Machine Guns(!)

Hacking Journalism with Sidewalk Chalk

My presentation before a room full of talented programmers next week Thursday will include hair salons, semi-trailer trucks and sidewalk chalk.

It's TimesOpen 2.0, where digital tinkerers gather to talk about online data from the New York Times and the latest trends in information technology.

Next Thursday is Mobile/Geolocation night (which is free). Presenters will include Mano Marks from Google, John Britton from Twilio, Matt Kelly from Facebook and me. I'll be talking about The Takeaway's Sourcing Through Texting project.

My preso will likely be the lowest-tech of the bunch. Our aim was, and is, to connect journalists and citizen-sources using basic text messages, and our method was brainstorming, learning and prototyping in two neighborhoods -- Southwest Detroit and Miami's Little Haiti. We absorbed a ton. (And we sparked an investigative series on illegal truck traffic.)

There are certainly opportunities here to mash up APIs and build on some nifty platforms. I'll talk about that, too.  But as we continue working toward connecting with sources via texting, some of our best insights have come from coffee shop conversations, church bulletin announcements and short-codes scrawled on sidewalks.

An Experimental Honor

What's so exceptional about the journalism innovation award The Takeaway won yesterday is that it's not for a broadcast, a series or a blog post.

It's for an experiment.

"Sourcing Through Texting" has been a process of immersion, exploration and rapid prototyping. Journalists and community leaders spend time in a neighborhood focusing on a simple question: How might reporters and citizen-sources make better connections through texting?

The answers are still emerging. We're still making prototypes. Yet, yesterday the concept won a Knight-Batten Special Distinction Award for innovation in journalism.

Since the award application went in, we've gone to Miami to run another experiment in Little Haiti, and Detroit's WDET aired a week-long series that evolved from the project.

That the award effectively predates those happenings is a huge jolt of support for experimentation, design thinking in journalism and everyone who contributed to this unique collaboration.

That includes folks from The Takeaway, Public Radio International, WNYC Radio, WDET Detroit, WLRN Miami, The Miami Herald, American Public Media's Public Insight Network, Mobile Commons, the Institute of Design at Stanford and the residents of Southwest Detroit and Miami's Little Haiti.

---

Sourcing Through Texting is a project of The Takeaway, which is produced by WNYC Radio and Public Radio International. It was made possible by a grant from the John S. and James L. Knight Foundation. 

Disclosure and disclaimer: I helped develop and produce this project. As always, the words here are my own and not those of my employer or any of the entities mentioned.

The Secret Decoder Ring

Today our education reporter had a bunch of data from New York State she was trying to match to schools in New York City. But the school codes used by the two governments look radically different.

For example, PS 15 on the Lower East Side is known to the state as 310100010015; the city calls it 01M015.

I once made a nifty formula to make the conversion(!), but a more straightforward and official approach involves the Excel spreadsheet found here. It lists all of the city schools, along with their addresses, various codes, and more. For a data cruncher, that's a secret decoder ring.

What made me smile was that the only reason I knew this document even existed was because of a little prototype I tried during the first swine flu outbreak. That experiment wasn't robust enough to make it beyond this blog, but it taught me a lot ... including where to find this ring!

Code For Good (and Money)

WNYC is looking for news technologists who want to code with a purpose -- helping public radio stations cover the news and connect with listeners. These projects are funded in part by support from the Knight Foundation and the Corporation for Public Broadcasting. Here's the posting:

WNYC Seeks Programmers with a Passion for News for Contract Project Work

WNYC Radio is working on several news application projects for public radio stations (including ours), and we're looking for more people to join our team to experiment, play and code. We're driven by a culture of prototyping, a bias toward action, a principle of show over tell, and a commitment to journalism.

Requirements:

* A passion for the news
* An understanding of the inner workings of the web
* Attention to detail, fairness and accuracy
* A genuine sense of collaboration, innovation, creativity and quality

And, of course…

* Fantastic programming skills and a love for the craft of making software in the public interest.

In particular, we're looking for people who have strong skills with the following:

* HTML/CSS
* Python
* Django
* PostgreSQL
* Ubuntu Linux + Amazon EC2

If you know PostGIS and GeoDjango, that's great, too.

Right now we're signing up people on short-term contracts keyed to specific projects we're starting or building. You don't have to be in NYC to take part (though it's great if you are).

If this sounds like you or someone you know, drop us a note and a sense of your work at jobs@wnyc.org with "Contract ProgNews" and your last name in the subject line. We want to hear from you as soon as possible, but no later than January 15, 2010.

WNYC makes decisions to contract services without regard to race, creed, sex, color, religion, gender, national origin, ancestry, age, physical or mental disability, sexual orientation, marital status, veteran status, citizenship status, or any other basis protected by applicable law.

(Credit where due: This description was inspired by and built from one posted by the Chicago Tribune news apps team, a group for which I have heaps of respect and admiration!)

Super Simple Mapping Tool

Making collaborative maps is easier than ever. But maybe not quite easy enough.

One of the projects I'm working on at the moment is a super-duper-simple tool to help public radio and television stations (and pretty much anyone else) collect and map local information from their audience.

We're in the design phase right now, and we've mocked it up for feedback. The video is below.

Whaddya think? Let us know!

Got Django?

I'm working on a few fun projects, all of which would fit nicely into the themes of this blog. And I need some help.

If you know if a Django coder who is energetic, collaborative, open-minded, innovative, creative, iterative and is based in NYC good at collaborating from afar, please let me know. This is paid work with possibilities for more if the match is right.

Think more prototype-tinker-try-repeat than requirements-to-results.

UPDATED: Check out our more detailed call for programmers here.

The Man Behind The Zipper

I've joined an exclusive club of New York City microbloggers.

Twitter? Ha. Facebook? Kidstuff. We're talking bricks and mortar, baby.

Quite literally.

My missives scroll across the facade of WNYC's building in west SoHo, zipping into your field of view as a parade of little red lights. It's the WNYC News Zipper.

As you walk to work or sit in traffic on Varick Street, I've got your eyeballs.

NY terror-plot suspects indicted

None of this 140-character stuff. Better to use just five words; seven max. (I used a nonessential adjective clause once. Lost everyone by the second comma.)

Media banned from covering Iran protests

And I know where you are, no fancy GPS required.

Building collapse on Reade Street, up ahead

Even if it's partly cloudy in the Bronx, I am absolutely certain you're in a downpour.

This rain ends by evening

User customization? Easy. I can sense you're in line for the Holland Tunnel on your evening commute home. So how about a little news about your governor and his chief rival?

Corzine, Christie speak to biz group tonite

It's tempting to simply repurpose our tweets or web headlines, feeding them automatically to the sign. But it's also clear that wouldn't be as special. Or impactful. Or memorable. So I've been recrafting our material specifically for my particular version of a hyperlocal, mobile user.

I've been doing this for a few weeks as a prototype, and soon WNYC's editors, producers and hosts will feed lines to the sign. What I've learned by writing -- and watching -- those little red words will help our staff craft the phrases that catch your eye as you zip by.

Rapid News Visualizations: Prototype 1

In my quest for timely, interesting, understandable info-graphics, I've set up a prototyping challenge for myself: Upon finding news data, turn it into something visual, compelling and useful ... as fast as possible.

I'm prepared to fail quickly and often.

In this case, "as fast as possible" was three days to make, another two to find time to post it. The result is not wholly useful. And you can't absorb it quickly. And it's a little misleading.

But it's a start.

Here is what I made. It's a visual representation of the attendance rates for every public school in New York City on Thursday, May 21, 2009. The New York City Department of Education started posting this data the previous day as the Swine Flu/H1N1 outbreak was causing kids to stay home.

It tells me two things off the bat:

1) Queens and Brooklyn schools had much lower attendance rates than Manhattan and Staten Island schools.

2) Teens skip school on nice May days.

No. 2 is apparent because almost every red square is a high school, which have notoriously low rates this time of year. For a better indication of potentially flu-related absences, I'd chart the difference between these absentee rates and a typical May day at each school ... which is info I don't have. Yet.

Initially I published this in Google Maps, which was interactive and allowed you to click on schools for specific info. But Google Maps only plotted about 100 or so schools, and there are more than 1,000 here. Instead, I did it in Google Earth on my own computer and took a snapshot. Here's another.

Kinda cool. Was fun to do.

Next!

----------

Anatomy of the process:

Daily absentee data from the school system is here.
An Excel spreadsheet with general data on each school is here.
I crossed these two data sets in Access to match school numbers with addresses.
I got the latitude and longitude for each address, in 500-line batches, here.
I spent a lot of time learning about KML files, writing them, failing, trying again.
I made colored icons in Photoshop, and used Excel to assign each school the correct icon.
I put all of the relevant data into one spreadsheet and fed it to this little helper ...
Which gave me this KML file ...
Which I fed to Google Earth, running on my Mac.

Break Glass Now

Consider, for a moment, the location of your nearest fire extinguisher.

And just how well did you use it last time?

Right. Chances are you're not prepared to skillfully put out a fire where you are sitting. At least you're not practiced.

But what if you had a non-emergency reason to use a fire extinguisher every once in a while? Maybe to clean your desk. Or a spill. (Nearest paper towel, anyone?) Using it occasionally would help insure that in case of a fire, you would both a) actually use it and b) use it well.

To better prepare our on-air and online operations for major breaking news, I've been promoting a point of view that says we shouldn't put our emergency tools, systems and skills "behind glass." Instead, we should incorporate those efforts into our everyday work. (I even believe we shouldn't put our energy into efforts that can't be used on a regular basis because in a crisis, we won't use them anyway.)

The best example of this is the daily production of our national morning program, The Takeaway with John Hockenberry and Adaora Udoji. The show's staff may just be the best breaking-news response team in public radio -- because they make the show in real-time every day, incorporating fresh news as they go. When the news happens to be really big, they're not just prepared ... they're already there.

A few other examples:

Map maker, map maker
In case of a civil emergency in New York, we'd want to quickly map shelters, closed roads, danger zones, escape routes. Even locate our staff. But we weren't prepared to whip together those kinds of maps in mere minutes. Now we're honing those skills by incorporating such work into everyday projects.

Information integration
When news hits the fan, information flies everywhere. Consolidating that data is key ... and also happens to be handy in everyday work. In the course of discussing a Mumbai-like terror attack in NYC, we discovered that our news-editing software can also check a listener email box. That's one less window to watch.

Nobody move
We designed our newsroom so that in a crisis nobody needs to change seats, which would move them away from familiar surroundings. As a byproduct, when something doesn't flow quite right during daily work, I try to make sure we address it now so we don't get caught off guard later.

Expected events as prototypes
In planning for election night, and now for the inauguration, we developed tools and techniques that will serve us again in a major unexpected event. We now know how to quickly rip up our station's home page to focus on a single topic. And in order to provide real-time election-night returns, we found new ways to clear the information path between the editors and the home page.

Share and share again
We share a statehouse reporter with stations across our region. So when former governor Eliot Spitzer imploded in a prostitution scandal, we didn't have to think twice about how to move information and audio between stations. We just used the FTP site and email list we use every day.

I've long been a fan of drills, and there are many more of those in our future. But by incorporating a little drill into our regular routine, we're better prepared for situations that are anything but.

[Photo by IamSAM. Some rights reserved.]

Prototyping Terror: Mumbai in NY

Two hours before our company holiday party, several of us were contemplating mass murder in Manhattan.

Not exactly Christmas cheer. But some good certainly came of it.

Really.

Several top thinkers and decision-makers reviewed the horrible events in Mumbai last month and considered how our news department would respond to a low-tech, coordinated attack on multiple locations in New York.

Here are some of our points of discussion, which could apply to any organization, journalistic or otherwise:

The Visioning Thing. We didn't do a full-scale drill, we simply took time to really visualize how things might happen. It was pretty powerful.

First we tried to think like terrorists and, as a group, picked three targets -- a transportation hub, a hotel and a shopping center. No sense naming them here; suffice it to say, we all knew each of them well and could picture the devastation and chaos.

Then we carefully imagined where each of our key people would be on a weeknight at 9:30, when Mumbai's night of hell began. Who lives where? Who's still at work? Who could get in fastest? What route would they take? Where would the first available reporters go? How would they stay safe? Think.

Our Civic Duty. We're journalists. It's second nature to pursue the facts and try to present them quickly, accurately in context. But as a broadcaster in a city under siege, our public service mission takes on new qualities, and raises questions. What do we do for people still in or near danger? Can we be better oriented to provide public warnings, safety and health info, comfort, maps, conversation, rumor control ... help? The conversation has started to adjust our operating Point of View, and could make a huge difference in how we serve our city (lowercase c) in those first few hours.

Information Overload. Phone calls, reporters, sources, Tweets, network audio, news wires, emails, web comments, TVs ... we easily came up with more than 20 distinct streams of audio, text, and visual information key to covering the story. In an era when all of this information is available to everyone on our staff, are we ready to monitor them all in a sophisticated, organized way? (Ah, no.)

Online, Under Pressure. Our methods for broadcasting have changed since 9/11 and The Northeast Blackout. We now use web alerts, social media, maps, and other tools to convey information. But when the adrenaline pumps, and minutes matter, we have to be ready to take advantage of all of these channels while maintaining our standards of accuracy and context.

Bias Toward Action. It's been 7 days since this meeting, and we're far better prepared than we were 8 days ago. What we've done, and are doing, is the subject of my next post, Break Glass Now.

[Photo by Andy Eakin]

Let's Date!

On your first date, you wouldn't plan your wedding. Or sign a prenup. Or name your baby.

So why do so many collaborations start that way?

Companies and organizations clearly need to learn how to date. See if you click. Get to know each other. Play.

Prototype, really.

Recent discussions I've had with a large company about collaborating quickly turned to who was going to walk the unborn child to preschool and pay for her clothes. It reminded me that people and organizations almost compulsively skip the playful exploring time. And the fun.

So let's take lessons from the millions of partnership prototypes that happen over dinner every Friday night:

Start with dinner. Get together. Talk. Dream. Learn. Over food, of course.

Don't be self-centered. You'll kill a relationship quickly if you spend all evening talking about yourself, your needs, your wants. Instead, find out about your potential partner. Learn about their hopes and dreams. Think about how they may enhance or build on yours.

Don't name the baby. Put off the discussion of branding, naming the project, how credit is bestowed. This gets emotional fast, and quickly moves you out of the realm of low-risk prototyping.

Put off the prenup. In fact, I'd avoid writing anything down at first -- especially anything regarding goals, directions, duties, etc. This starts to define the relationship from the outset instead of allowing for open innovation and low-risk experimentation.

Respect each other. Be nice. Be giving. Be open. And if that costs a little, consider it an investment in the potential of the partnership. Pick up the check here and there.

Meet up again. And again. Make a plan -- and put it in your calendar -- for the key people to meet regularly, preferably over a meal, to check in on how everyone's doing. That's the time to make sure nobody feels disrespected, over-committed, or unhappy. Then adjust accordingly.

Break up gracefully. If the partnership just doesn't click, part ways, remain friends, and be sure your team gets together to learn from, and record, what parts worked.

I won't kiss-and-tell about our newest collaboration, but I will say this is the approach WNYC took when we approached Iowa Public Radio back before the Iowa caucuses. We made a concerted effort to learn about them and focus on their needs. We talked a lot. We shared info and a common effort. And we didn't name the baby. The result was an amazing night of radio, and smiles all around (scroll to the bottom). It's also how we've approached a lasting relationship with the wonderful folks over at the Connecticut Public Broadcasting Network, where we first prototyped this kind of coverage.

Happy dating!

[Photo by hypertypos]

Election Night Design: Post-Vote Post

Prototype. Prototype. Prototype.

It really saved us. Monday we had everyone sit in place for a talk-through drill and discovered issues that would have been problematic on election night. The only thing that didn't work (once) was a wireless microphone setup in the newsroom which, ahem, we hadn't tried before.

One interesting tidbit: Virginia didn't prove to be the early key we thought it might be. In fact, with the earlier-than-expected call of Ohio for Obama, we knew he had won even before Virginia went his way.

And in a departure from many networks, we actually took our newsroom conversation to the air as early as 10:20 p.m.: Our host was honest about how we expected Obama to be called as President at 11 p.m., straight up, when the California, Washington and Oregon were declared his. As it then happened.

We're conducting a full internal critique Monday at Noon -- using the d.school's "I Wish, I Liked, How To" format.

Details to follow.

Election Night Design: Gaming and Planning

ITEM 1: There's a solid writeup of electoral vote scenarios at FiveThirtyEight.com expressing much of what we've been gaming out in our newsroom.

ITEM 2: Several weeks ago we worked to select a few Counties That Count, half-remembering* a prescient article that suggested a few Florida counties could decide the 2000 election.

Andrea Bernstein has counted our counties among her stops for The Takeaway, and points out that in the last few days Barack Obama has been doing the same -- visiting Clark County, Nevada; Pueblo County, Colorado; Prince William County, Virginia; and Palm Beach County, Florida.

This weekend, the journo-programmers at The Takeaway are working to make sure you can track the real-time election results from our Counties That Count on the show's website.

ITEM 3: To get more info in our face, I set an overhead monitor to flip through several websites automatically, for free. Here's how:

-- I put the Firefox 3 browser on a computer attached to the monitor
-- Next I installed the ReloadEvery Firefox add-on, which auto-reloads websites
-- Installed the Tab Slideshow add-on, which cycles through tabs at a set interval
-- Installed the Full Fullscreen add-on, which hides the Firefox toolbars
-- Pulled up several sites in separate tabs
-- Set ReloadEvery to 1 minute, and Tab Slideshow to 5 seconds
-- Turned on the Full Fullscreen feature to hide the toolbars
-- ... and voila!

* Anyone who remembers the specific article, please let me know!

Election Night Design: Confounding Factors

There's a lot to consider when designing comprehensive, contextual coverage for the biggest news night of the year:

- Pinpointing the key stories
- Getting early warnings on those stories
- Scheduling hosts and producers all night
- Deciding where to send reporters and producers
- Engaging the audience
- Ensuring strong staffing the day after

As we prototype and plan for Election Night 2008, here are some of the issues that have come into play:

A Foregone Conclusion?

One of our crystal balls is FiveThirtyEight.com, a fantastic, transparent analysis of polling data. The most beautiful part? Nate Silver runs 10,000 simulations of the outcome based on the errors and fluctuations possible in every poll. Ten thousand prototypes daily. Wow. Below is today's chart of how many electoral votes Obama gets in each simulation:

... which is to say, in nearly 1400 simulations, Obama gets 375 electoral votes. The total possible is 538 (hence the name of the site); it takes 270 to win the election.

The other crystal ball is the Intrade prediction market, where real money is bet on each state's electoral vote. Intrade has been shockingly accurate, from predicting each state Bush won in 2000 to the super-secret selection of the Pope.

Both site show a solid Obama electoral win, and have for a month now. They could be wrong, and they will certainly adjust as we get closer to the election. But in September, a landslide was not a part of our equation; it is now.

The Voting Story

Voter turnout could break records, at a moment when voting machines are untested in many states -- such as, surprise, Florida. Any case of voting failure, no matter what your political leaning, is a story in a democracy and an echo to 2000.

But FiveThirtyEight's simulations suggest there's only a 2 percent chance a decisive state will have a vote close enough to trigger a recount. And the chance of that winner of the popular vote will be different from the winner of the electoral vote is between one-tenth of a percent and zero.

Live Election Companion

We had a smashing success running our live debate companion during the candidate face-offs. People were able to participate in real time and get insights from our public radio luminaries.

On election night, what's the right way to have people involved? Set aside a key hour for a similar chat? When would that be? Have it open all night? Would that be a valuable experience? Better to have a running blog of updates?

On this, we're open to input. Comment below if you have any thoughts.

[WNYC's election night coverage begins at 7 p.m. and runs through the following morning -- online at wnyc.org and in New York at 93.9FM and AM820.]

UPDATE ... Our digital election team met today and decided to run the Live Election Companion, with participation from our on-air hosts, from 8 p.m. to 11 p.m. on the WNYC and The Takeaway websites.

Prototyping Debate Companionship

Back when a dozen people were vying for the presidency, I watched one of the debates at home alone, wishing I was in the company of people I respect to hear their thoughts.

Quick searches found the live blogging to be either slower or noisier than I wanted.

So as the presidential debates approached, I asked a tiger team at the station to come up a "live debate companion" fed by our top thinkers.

For the first McCain/Obama debate, we had John Hockenberry, Brooke Gladstone, Brian Lehrer and Andrea Bernstein tweet into their own accounts, which were presented in a self-updating Twitter/Web solution called Monittor. In a separate window, we fed a steady update of any tweet worldwide that included the word "debate" or the candidates' names -- offering a living, breathing experience, with a nice feel. It was also easy to share across our websites and other stations.

It turns out that the Twitterverse gets reeeeeallly slow during the debates, and that made the end result less interesting than we had hoped. Also -- hard to provide the trademark public-radio context in 140 characters.

For the Palen/Biden debate, we switched to CoverItLive, which provided a rockin', real-time experience. We hit some (yet unknown) room capacity, but for those able to join, it really flowed well. We copy-pasted some analysis into tweets, too.

Two strong signs we're on the right track:

1) The next day, the critiques at the station, including a chunk of a Takeaway planning meeting, centered on the content of the event, not the technology.

2) This:

[Comment From Chris, NYC]
This was a great experience. Thanks for your company.

"Company." Bingo.

We'll do it again Tuesday.

UPDATED OCTOBER 14: We learned today that there 1026 people participated in the Live Debate Companion for the 2nd presidential candidates' debate. That's exciting.

Prototyping Disasters

Last week at a meeting of public radio news directors, I gave a presentation about the importance of prototyping for disaster planning -- getting off our chairs and actually trying out our plans. Here's a quick sketch of the speech, with documents included.

Key Principles

CONSIDER OPERATIONS AND SYSTEMS -- To do good journalism in moments of crisis, your systems and operations have to be ready at three levels: newsroom, station and a backup site in your city/region.

PROTOTYPE PROTOTYPE PROTOTYPE -- To find out if you're ready, try it, test it, simulate it, do it. And repeat. Don't just write emails. And don't get too complicated. Take a page from design thinking and keep it simple. Then you don't get too invested in the test, and are open to changing your ways.

Prototyping Newsroom Systems

Example 1 -- Crisis Information Flow
At WNYC, we used Post-It notes to simulate how information flows through our newsroom to air during a breaking news situation. "Facts" were represented by shapes, and we watched how they moved (or didn't) through the process. We learned a ton, which you can see in our full case study, including a 15-mininute movie. What we learned improved our coverage of a big news story that broke the very next day.

Example 2 -- Full Scale Drill
We've also done a full-scale crisis drill, simulating a dirty-bomb attack. This involved the entire news and technical staff, who responded to information (wire stories, witness information, etc.) sent to them roughly once a minute for an hour. Due to the sensitive nature of this drill, I have not posted our case study. If you are interested in learning more about it, please contact me.

Prototyping Station Preparedness

Example -- Blackout Plans
To test our backup power, we regularly cut power to our facility (usually late at night) and make sure it still works. During the day, we actually relocate hosts to our backup room, as they'd do in a real outage.l Just three weeks ago, two of us got out of our chairs, walked to a key breaker box, and took a new look at which switches we'd have to throw in a blackout. In the process, we realized there were no backup lights in that room, so we wouldn't have been able to see the switches! We solved this with a $15 power-failure light.

Prototyping Operations Elsewhere in Your City/Region

Example -- Make Other Arrangements
Be ready to move somewhere else; we've had to do it twice (once on 9/11, once during the northeast blackout of 2003). We've arranged with another local studio to be our backup facility, and we have key equipment and supplies in place there now. Our full news and technical staff will be visiting the facility to see, touch and feel what it's like to set up there. We're also installing our own set of phone lines so we can do live call-in programs -- which have been essential components of our crisis coverage.

Tips

Click here for a 2-page PDF of helpful tips and tricks for public radio stations.
_______