What It’s Made Of

All of the backend pieces already existed before we decided to build the game-day site. Most of the front end is extracted from previous projects and partially rewritten.

The feed module

I wrote the app behind the Twitter/Instagram feed, nicknamed DeathStar, for the Ducks GameDay iPhone app we built in August. That’s probably the most complicated piece of code in the project.

DeathStar is the second revision of a client for Instagram’s real-time API. It collects both Instagram photos and tweets using Twitter’s streaming API and uses Socket.io to push them to the app. It collimates these objects into a single stream. (DeathStar…get it? Anyway.) It’s a big hacky CoffeeScript mess than has a slow memory leak. It persists objects to Redis and only does a couple of non-trivial things. For one, an update from Instagram includes an array of images, but only one object is actually new. I started with an in-memory hash of IDs, then moved it to Redis. Existence checking is plenty fast and keeping data out of app code is good. (I also used to write every object to Mongo for analysis, but since it’s on a 32-bit Linode slice, I hit the 2 GB issue and haven’t provisioned a new 64-bit box.)

DeathStar was built to combine Twitter and Instagram posts into one stream. For the iPhone app, this makes the front-end code easier than dealing with two backends. But with seperate columns for Twitter and Instagram, it’s very possible to bootstrap the page with 50 tweets and 0 Instagram photos. I hadn’t anticipated this and didn’t have time to write a new endpoint. I wrote the ad-hoc logging code to see how often this actually happened.

The Instagram real-time API requires a publicly available callback URL. This makes local development hard, so when I wrote the server, I developed on a VPS. Moving forward, I’ll either use localtunnel/similar or write a wrapper that uses a persistent HTTP connection or WebSocket to have dev/prod parity.

The NYT’s Erik Hinton kindly pointed out that since I was serving Socket.io off port 6767, it was blocked on some firewalls. I guess it’s time to get HAProxy rolling.

The chat module

The Heroku-backed chat app was a proof-of-concept from a few weeks before. I wanted to play with the Parse.com API from Node and it was the straightforward demo and turned out to be handy.

I love Heroku. I have a ton of (free) one-dyno projects. It makes me build smarter stuff (environment variables, ephemeral file system) and has plenty of horsepower for the scale stuff we do—probably an order of magnitude more than what we need. I was nervous that it would get massively popular and bring down the social and chat feeds, but it was stable-ish and fast the whole night. (Then the social-realtime server crashed overnight, as it’s for some reason prone to do. That thing needs a rewrite.)

Our dev server cURLs the apps every few minutes to keep the dynos from spinning down.

I used Underscore.js in the chat for one _.each call, actually, but I should’ve rewritten that. Since I slapped all the parts together, I didn’t refactor anything that I didn’t have to and was left with another HTTP request to avoid writing a for loop.

The stories module

The stories column was extracted from the view code in a UIWebView on the Ducks GameDay app. (I’ve been meaning to put the Obj-C part on GitHub, but I never pulled the secret keys out of it. It’s Storyboards w/ custom segues, UIWebView and some JS <-> Obj-C communication. I learned Obj-C to write it. It’s not good code, but it works okay.)

The JSON feed

The JSON feed is provided by a plugin. It’s the same feed that backs everything in the iOS apps. It wraps WordPress’s RSS feeds, so it’s easy to get a feed for whatever you want. Super simple. The only annoying part is it won’t take an offset, so loading more posts isn’t possible. (I haven’t tried recently. But I think it’s due to the RSS feed, which has its length set by WP.) I’ve been trying not to write another one for months now (something that understands LIMIT and OFFSET), but it might be time.

Time & Resources

I made a sketch of the app on Wednesday morning, send a quick prototype to Andy (our editor-in-chief) around noon, then hacked until about 1am (about 11 hours of dev time).

Thursday morning, I spent a couple hours cleaning it up and tweaking to fit it into our WordPress site, then we pushed it around noon on Thursday.

Accommodating Screens

I abandoned phone-size screens, which was a questionable choice. The app is fluid down to iPad-portrait width, but a four-column layout on a phone just doesn’t work the same way as a tablet or 1000px-wide browser window. Instead of rethinking the project for mobile, I abandoned it. Analytics showed three times the use on iPhones than iPads, so thinking it through would’ve been worthwhile for the 350 iPhone users that got a broken page.

@SeanDKennedy also reported it doesn’t work in IE8. I’m not surprised. It would’ve been hard to build this for IE8 and even though there’s plenty of people using it by choice or forced to, I would get much less done trying to support it.

Logging

I set up custom JS logging out of panic—I got early reports that a few people had empty Twitter and Instagram columns, and I wanted to know what the counts were for those columns to see how often this was happening. I pointed it at a Node.js Heroku app that console.log’d the GET param so I could tail -f it to get a feel for how it was performing.

A few places I could’ve stored that data but didn’t:

• Heroku, but there’s a 10k row limit. In hindsight, I should’ve just paid the $9 for a DB.
• Parse.com free tier. But I didn’t know the volume, and that the chat app persists there.
• Build one. SQLite and a single PHP file could’ve done it, but I ran out of time.

Adjusting on the Fly

I added a score box around 5pm that polls a Google Doc every couple of seconds (using Tabletop.js to make reading the JSON trivial) and is a free-and-quick backend for small data sets. I broke the site for a few users for about a minute when I did this, actually—I hadn’t built a way to do an atomic update because all the app code is hotlinked to the GitHub repo and I pushed app code that needed Tabletop.js without updating the template. I got the project into WordPress as a page template, but that meant I was copy-pasting changes from dev (hosted locally with Pow) to the PHP template. This dev/prod mismatch was a mistake, clearly.

The velocity of the Twitter feed was crazy, but I couldn’t think of any easy way to fix it. There seem to be upper and lower bounds on when real-time is useful for this kind of thing. If it doesn’t change every few seconds, polling is fine. But if it’s multiple-per-second, it’s not readable and thus not useful. I made the Twitter and Instagram columns evict old items after I noticed people had thousands (or tens of thousands) of tweets rendered. Here’s the commit, but I’m not totally sure it the fixes the whole issue. I also didn’t have a way to push code changes for people with the page already open. A few sessions were 5+ hours long.

Secondary things—plotting tweet velocity, visualizing groups of tweets around peak moments, etc—might be interesting. I’ll spend a lot of time this summer getting ready for next season’s round of stuff with all the information and experience from the past months.

Traffic & Response

We got 3,200 views (1,900 unique). I should’ve logged enough to know the total-time-on-site, but I just tail -f’d the logs to spot problems. We average 138,000 views per month (that’s an average of Jan-Sept 2012), or ~4600 hits/day (~1900 uniques/day).

It got a lot more use in the hours after launching it than during the game. There were some haters—I think providing anonymous chat was begging for it. I’m surprised there wasn’t more spam.

I haven’t thought all the way through what the lower numbers during the game means. Maybe people don’t want a second screen? I found myself not watching the game sometimes because I was watching a mass of tweets flow by and odd Instagram photos coming through. I also hoped more people would participate in the live chat. Emerald digital sports editor Isaac Rosenthal liveblogged the game with it, adding a nice balance to the free-for-all on Twitter.

There were ~138,000 tweets between 10am Wednesday and 2pm Saturday. After DAT’s opening touchdown, the next 5 minutes saw 12,500 tweets, with an average of 42 tweets per second. (Whoa.) I really want to make something from all this. Here’s Redis’s dumb.rdb and a quick hack to read it.

Log watching.

Last Thoughts

Refusing to experiment won’t lead to new things. This was an experiment that our editor-in-chief and president went for, and I know they would’ve pulled the plug if it sucked, too. Working with a team that’s open to do different things is why my job is awesome.

The project is still up at http://dailyemerald.com/gameday-live, and the code is on GitHub. The only reason I could do any of this is because of the brilliant people behind the projects I’m building on top of. I’d be thrilled if someone else built more cool stuff because of what I’ve done.