A little over a month ago, Al Shaw and I made an interactive map of the “Great Migration” of African Americans from the rural south to northern cities during the 20th century. The housing discrimination suffered by African Americans in their new cities led to calls for reforms, resulting in the Fair Housing Act of 1968. But as the fantastic reporting by ProPublica reporter Nikole Hannah-Jones shows, the federal Housing and Urban Development agency has fallen short of its requirement to take affirmative steps to curb segregation in America’s cities and provide fair housing for everybody.

To accompany the story we wanted to make a map showing the Great Migration itself. We also decided early on that it would be the lead art for the article.

My first impulse was to create a vector-based (as in math, not as in SVG) map, like the wind map Martin Wattenberg and Fernanda Viegas made, which would show the paths that African Americans took to northern cities over the decades. But I couldn’t quite wrap my brain around how to apportion the changes between individual counties. For example, if a particular county loses ten black people in a decade, where did they actually move to? The answer was unclear—it could be that five of them moved north and five of them died. When you are dealing with decennial censuses the exact ratio is impossible to know.

Another problem, and the most important one to solve, was finding the data, but I had the answer. In 2011, as part of research for our Redistricting story, I had stumbled across NHGIS, a program run out of the University of Minnesota that has painstakingly collected census and geographic data from the 1790 to the 2010 censuses and provides it to researchers and journalists free of charge. It is a truly amazing resource.

Incidentally, I think my favorite census was the 1890 census, which was the first to use punch cards for tabulation, in addition to handwritten forms. It took only one year to calculate the total population of the United States in 1890 versus eight years for the 1880 census, and is a watershed moment in the history of computing. A fire destroyed most of the 1890 census materials, which led to the creation of the National Archives.

So, ok, now I had the data, and I knew which kind of map I couldn’t make, how did I settle on an animated choropleth map? I had two inspirations:

The first inspiration came from a graphic by Joe Ward, Xaquín G.V., Alan McLean and Graham Roberts published in the New York Times called “How to Win: The Hurdles.” The Times used motion capture to film Olympic hurdler Lolo Jones, and allowed users to click and drag back and forth to see exactly how Jones clears hurdles during races. Most importantly, users could go frame by frame to find the most interesting parts.

The Times graphic had me thinking back to my first job out of college.

The first professional(ish) job I ever had was as a video editor for a small San Francisco production company. One of the best parts of video editing is how good the tools are. In the hands of a capable editor, the editing bay is an extension of the body. Slicing and dicing video is effortless and fast thanks to the engineers and user interface experts who have optimized the typical video editing workflow. If you’re interested in video editing, either as a hobby or professionally, run to the bookstore, right now, I’ll wait—the internet is forever—and pick up a copy of In the Blink of an Eye by Walter Murch.

When it comes to actually editing, the most important innovation in video technology, hands down, is “scrubbing.” Video scrubbing is a neat trick where an editor can preview an edit by dragging their mouse back and forth over a video clip or edit point to preview the faster than normal video. If you use iPhoto, you’ve probably seen this effect when you view your photos in the date view and move your mouse back and forth. This fine-grained control, like in the Times’s graphic, allows an editor to find awkward moments or mistakes quickly.

The key realization I had was that this scrubbing motion feels really natural when using a mouse or swiping with your finger on a mobile device like a tablet. Because my time series of census choropleths was, like a video, really just a bunch of images, I could make a little video in the browser that was easy to “scrub,” in much the same way a video editor does.

When you’re making things on deadline, having data and inspiration is the best possible situation. To make the actual maps, I grabbed data from my favorite GIS warehouse, Natural Earth, to flesh out the county data from NHGIS. I then used Simple Tiles to generate a series of images from 1890 to 2010 representing African-American population. The code was really simple.

The only other implementation trick was to define the breaks between frames in the map, but that also turned out to be really easy. I added a mouseover handler on the map’s element and used this code to figure out which image to show:

var bucket = (e.offsetX || e.pageX) / (width / years.length) | 0;

Basically that picks a year based on which portion of the map the cursor is in. The fun bit is the | 0 part, which confuses some people, but, in javascript, OR-ing a floating point value by zero does the equivalent of Math.floor.

I’m not the greatest designer so Al Shaw created the UI and Lena Groeger helped a bit with the labels on the map.

The Great Migration is a fascinating and historically rich subject, and often it is important in news to look back in time to see how we got where we are. I hope the graphic helps a bit in that goal.

[Cross-posted to the ProPublica Nerd Blog.]

Source Q&A

Jeff Larson kindly described more of the process behind the creation of this graphic with our editors.

Source: Is your team approaching mapping and interactive feature design any differently than you did a year ago?

JL: We’ve certainly leveled up on mapping since last year. One difference is that we’ve been doing a lot more client side rendering of maps for our smaller graphics, for example our North Dakota Spills map is all done client side, thought the background images were rendered with Simple Tiles. In a similar vein, Lena Groeger and Krista Kjellman Schmidt have been on a tear with SVG maps like Where Are the Foreclosure Deal Millions Going in Your State? and Underground Injection Wells. If I had to bet, I’d say we’re going to move a lot more presentation of maps to the client, and not wrapped in a tile-based map next year.

Source: You mentioned that you stumbled upon the data as part of a prior story. Did that prior reporting inform this piece in any other ways? Did you mentally bookmark that data or do you have a systematic way of referring back to previous data sources or tools?

JL: Well, as part of the Redistricting research we wanted to look back at previous districts and see the political breakdown of historical districts. So that is how I found NHGIS. The problem is that historical election results are nigh impossible to come by, so I have high hopes for Open Elections. My filing system isn’t really one: I usually try and file things in my brain and pull them out with Google searches, but that’s something I should work on.

Source: Could you walk us through the decision-making process for this map becoming the lead graphic? Did the interactive and the reporting run in parallel for this project, or did one start first? For this and similar projects at ProPublica, do you find that the interactive and reporting work interacts very much over the course of the investigation/project?

JL: The news apps department at ProPublica gets involved really early on with every story we think will have data. Sometimes it takes a while to play with the data and figure out what to do from an interactive perspective, but for this story we almost had it sketched out from the very first planning meeting we had. For this project, Nikole Hannah-Jones was pretty far along reporting-wise, so we worked in parallel, although I did run a few demographic calculations for her along the way.

Source: Do you have any suggestions on how reporters with local news organizations could use this data or mapping approach to localize this story? Could you talk a little about the graphic for the Westchester-focused followup?

JL: We’re pretty lucky when it comes to data because we are a national news organization and there is loads of data nationally. When anyone needs to go to the lower levels on the census hierarchy margin of error estimates start becoming a real problem. For example, the Westchester map only includes the census designated places within Westchester County and not all the tracts, because in the American Community Survey the margin of error estimates by tract for the number of black households were 100% of the actual estimates. Which is basically like saying we are 95% sure the average American in this tract weighs between -100 and 400 pounds. Now, the ACS is one of the best things on the planet, it just gets more and more uncertain the smaller the geography is.

For local reporters, my advice would be two fold: get in touch with your county’s GIS planning office (Westchester GIS was really helpful), and be careful with small geographies in the ACS.

Source: Are there plans to continue reporting and creating graphics on this story? Have those plans been influenced by the response to the story in the month since it was released?

JL: All I’m gunna say on that one is: keep an eye on ProPublica’s website. :)

• Share on Twitter
• Share on Facebook