Applying a Layer of Varnish

Shortly after Rick Perry entered the 2012 Presidential race, he changed a long-held stance on the HPV vaccine and our coverage of that story set the record for most concurrent users on our site. Huffington Post picked up the story and at one point we were their top headline on the site.

We scrambled all day to keep our servers up and running. All of our servers are on Amazon Web Services so we could scale out, but we had so many servers running we were crushing our database. We needed a way to serve our site and protect our app servers. Enter Varnish.

Varnish sits in front of our Armstrong/Django-powered app servers and caches the responses it gets from them. Its entire purpose is to retrieve some piece of content from somewhere on the Internet, cache it locally in memory, then serve it back as fast as possible and its amazing at that job. Once the period of caching has expired (five minutes by default in our configuration, but each of our views can set their own criteria), Varnish makes another request of the app server to regenerate the page.

The secret of Varnish is the concept of grace: instructions to serve expired content when it can’t regenerate it for some reason. The length of the grace period is configurable, but we use 48 hours. That means our site’s backend could be down for as long as two days and the only indication the public would have is that the content isn’t updating.

Varnish allowed us to serve our homepage and live stream straight out of memory. Every five minutes, Varnish regenerated the both of those pages while still serving the old version until the new version has been successfully regenerated (within seconds). We only did two requests every few minutes to regenerate each of those pages. Our app servers had no idea anything was different than any other Tuesday night.

That’s only half the battle, though. We had nearly 12 times that much traffic on our video live stream.

Stand on the Shoulders of Giants

This part isn’t nearly as interesting. We stayed up because we hosted it on YouTube. We’re not video experts. We grok the web, but streaming live video to hundreds of people, much less hundreds of thousands of them, is hard. Rather than recreate the wheel and try to provide our own stream with a custom player, we’ve offloaded all that work to YouTube.

Interestingly enough, this is similar to the model that SCOTUSblog has used the last few days as they’ve been slammed. Rather than try to come up with their own custom live blog, they’ve simply dropped in CoverItLive. Figure out where your strengths are and play to those. Ours is definitely not video, much less live, streaming video, so we outsource it.

To make all of this happen, we have two Mac Mini servers in a server rack at the Texas Capitol. Each has a Blackmagic Intensity Extreme that converts the raw video and audio feeds from House and Senate into a digital format. Once the conversion is done and fed to the Mac Minis, we ship it off to YouTube.

Our setup is simple and efficient. We offered the world a glimpse into Texas politics at work using consumer hardware and less than a few hundred kilobytes a second of uploaded data.

Closing Thoughts

Protect your application servers. The bottom line is that you can’t handle massive volumes of scale and serve all of your content dynamically all of the time. You don’t have to strip your site down, either, just use what’s available to you. Use Varnish to protect your app servers and make sure that they don’t get slammed. Use external services for the really hard parts so you don’t have to worry about that.