Failure is not an option
At the Kennedy Space Center in Cape Canaveral, Florida, in the summer of 2011, Atlantis was preparing to launch for STS-135. It would be the last mission of the NASA Space Shuttle program. On hand to see the launch were Harper Reed and a few of his new co-workers who were leading the tech team for President Obama’s reelection campaign.
A self-proclaimed hacker, Harper rose from lead engineer to Chief Technology Officer at Threadless, a platform for crowdsourcing custom designed t-shirts. It had been two years since he left Threadless, and he had taken his time to find what was next for him. Now that he found it, he was having a hard time wrapping his head around the opportunity: Chief Technology Officer for the President’s reelection campaign.
Harper wandered through the halls of the Space Center taking in the pieces of history, testaments to what can be accomplished when smart, technical people work to do great things. Badges from early missions, prototype space suits, modules of shuttles. History abounds. It was from Pad A on July 16th, 1969 that Apollo 11 launched on the mission that would put man on the moon.
In Cape Canaveral, surrounded by space history, Harper fixated on a single idea: “the fail-safe.” The idea of anticipating any possible failure and creating a contingency plan.
“NASA wanted to make sure that, whatever happened, the rocket could land without harming the passengers,” Harper explains. “That’s it. Very simple.”
The solutions behind the fail-safe can be complex.
“Maybe it means a plane needs two pairs of landing gear, or it has two systems to control the whole plane, or three maybe. It doesn’t matter, whatever it takes to make sure the passengers aren’t harmed.”
Failure is often seen in a different light in the tech world. It tends to hold a certain charm. “Fail fast.” Failure means you’re moving quickly and taking chances. It means you are swinging for the fences, not playing it safe. Failure is forgiven.
But when you’re putting humans atop a half of million gallons of rocket fuel and launching them out of Earth’s atmosphere, failure is nothing to celebrate. During a NASA mission, somewhere along the line something will go wrong, but that cannot lead to absolute failure.
It struck a chord. At the campaign he would be at the helm of a tech product that was like few others. Failure could not cascade out of control. Complete failure here didn’t mean you took your shot and just missed; it meant that the leader of the free world was out of a job. Harper and his team would have to build a tech stack that could take the strain of getting more than 60 million people out to vote on election day, anticipate the issues that could arise, and build the fail-safes to keep the entire system from crashing to its knees.
“When failure happens – because failure will happen – how do you fail safely?” Harper asked himself.
Getting off the floor
“I was literally laying on the ground from 2009 to 2011,” Harper says with a smile. “It’s important to take time to try and figure out what’s happening in your life.”
In 2009, after more than four years at Threadless, Harper knew it was time to move on.
“When you’re at a startup for a long time, it naturally shifts as you grow. And for a whole bunch of people who’d been at Threadless for a long time, it just shifted to be not the company that we started at. We were able to bring it to that point, so I was really proud of where we got to, it just wasn’t the company I wanted to work at anymore.”
He just wasn’t sure where he wanted to work.
“I spent a little bit of time wandering. I call it my vision quest. I was trying to figure out, what I was doing and why I was doing it.”
He worked at some venture capital firms, helping do due diligence on startups. And he surveyed the scene looking for whatever was coming next for him. He knew what he didn’t want.
Before tech, he worked at an ad agency. “I sold encyclopedias” is how he put it to me.
“The agency was terrible work. It was very ego- and individual-focused. It wasn’t focused on a building a great product.”
That’s what drew him to Threadless in the first place. He wanted to make something. He wanted to work on a product, not just help sell it. His “vision quest” helped him realize that he wanted more.
“I wanted to work for something that I can believe in.”
That was when the President came calling.
Four short years later, hiring campaign staff from the tech world already seems normal. For the 2016 campaign, Hillary Clinton has tapped Stephanie Hannon, Google’s director of product management for civic innovation and social impact, as her CTO. But in 2011, it wasn’t the obvious match, for either the campaign or for Harper.
When they first started talking, Harper wasn’t interested in a job for himself. He just thought he could help out a bit and set them up with some engineers that fit the campaign’s needs. That changed after he met with Jim Messina, Obama’s 2012 campaign manager.
“I wasn’t immediately sold on the job. I had just spent two years basically saying no to every job opportunity that came up, and I was really comfortable laying on my floor. It took a little bit of convincing, mostly by my partner and my father, who were like, ‘What the fuck are you doing? Of course you’re doing this.’”
And so Harper pulled himself up off the floor.
Obama’s 2012 reelection campaign aspired to be unlike any before it. And that meant it had to bring in tech talent that no campaign had brought in before. Harper was one of those people, and he knew how to find more like him.
No campaign had used technology and data in the manner the Obama campaign was planning. The 2008 campaign was the first internet campaign, but it mostly used the internet as a medium to communicate with voters. However, the 2012 campaign had time and resources to build out their tech stack. The campaign would have technology baked into every aspect of the campaign: raising donations, recruiting volunteers, canvassing and registering voters, and getting out the vote – all of which would require having a solid array of tech tools that communicated with each other. And Harper had less than 18 months until Election Day.
Understanding the problems technology can solve
Excited, Harper tried to hit the ground running. He probably tried too hard.
“I came into the campaign like any tech dude would, thinking I knew everything. I’m the smartest guy alive,” he remembers. “And it turns out that the campaign … they already knew everything.”
The rest of the team at Obama for America knew a thing or two about running a presidential campaign. They had done it quite successfully four years prior. They knew what it took.
“The campaign knew all the answers generally about crowds, about volunteers, about motivating voters. What it didn’t have a good answer for was how it could use technology to solve these problems more efficiently.”
“For example, take the field office network that the campaign has to set up,” Harper says. “Thousands of field offices across the country.”
Field offices allow on-the-ground coordination of local volunteer efforts. And a single battleground state can have hundreds of field offices spread throughout the state.
“And they all need cable internet and computers. And then they all shut them down, in the time period that most of us would take just to think about moving.”
Harper knew his team could help by taking the field staff’s efforts and making them more efficient, more effective. He focused on being a force-multiplier for those efforts when they could, and get out of the way when they couldn’t.
“Campaigns are a resource game. You’re just trying to be efficient. If we can make 45 volunteer hours worth 85, then we’d win the election. If that work was worth 65 hours, we’d win. 55 hours, we’d win,” Harper says bluntly.
The time and effort was being put in; the tech stack just needed to enable it.
“But what often happens in technology is we’re not thinking about the real impact of our technology. Sometimes it becomes that 45 hours is worth 35 hours. That’s what we had to avoid. We had to make sure that we were augmenting the process, not replacing, not taking away.”
The end user needs to always be taken into consideration. If you only see the product through your own perspective, you’re bound to miss out on a huge chunk of your potential audience.
He remembers back to his early days at Threadless, a cocky 26-year-old engineer wearing women’s jeans and “the world’s tightest American Apparel shirt.” When he saw the shirts they were using, he wondered, “Why are we using Fruit of the Loom shirts?”
“Eventually someone was like, ‘Harper, we’re not selling to you. All of America doesn’t wear women’s jeans.’”
“That was a really humbling conversation,” he remembers. “Because it’s like, ‘Oh shit, that’s right.’ I think everyone, probably us in technology more than others, can be very focused on our own perspective.”
Technologists can think that they can code an answer to all of life’s problems. When you’re a hammer, everything looks like a nail. Failure doesn’t have to mean that an API is unresponsive or a database is dropping tables. Failure can mean simply wasting resources attempting to solve the wrong problems.
“At the campaign, it was like the Fruit of the Loom example times a million. I didn’t understand our volunteers. I walked in and thought, ‘We can do a paperless field office! All tech, all the time!”
But it turns out that iPads run out of batteries, and they can be confusing and off-putting to a significant portion of the population that doesn’t spend the bulk of their days staring into an LCD screen.
“You know what doesn’t run out of batteries? Paper. You know what both young people and old people know how to use? Paper. It’s this infinitely scalable resource that you can get a lot done on. All you have to figure out is how you merge paper and technology.”
The team’s solution was a system that allowed field offices to use all the campaign’s data to create the best possible targets for canvassing. And if a volunteer was more comfortable with paper, she could have a paper packet with all the information she needed. Then, later, a volunteer who was more comfortable with technology could take her paper canvassing data and put it into the system. It was a fail-safe. If the tech product doesn’t work for a volunteer, the paper will.
It wasn’t the tech teams place to dictate how anyone should do their job. They simply wanted to build a system that would empower the rest of the organization: the digital department, who ran the social media accounts and flooded your inbox with emails; the analytics team, who crunched the data to create scores for each and every voter on how likely she was to support the president and how likely she was to vote; and the field staff, made up largely of volunteers, who went door-to-door and worked the phones to encourage voters to turnout on election day.
Harper’s team developed tools for each. The call tool made dialing and connecting with voters easier, and allowed volunteers to talk to more voters per shift. Dashboard, a social platform, helped volunteers organize events. Facebook and Twitter Blaster allowed supports to message targeted friends and encourage them to vote. But most important of all, they would build Narwhal, the backend all of these tools were built on.
Historically, the tech work for presidential campaigns has been done by consultants who focus more on politics than tech. Historically, those campaigns have also failed to integrate their information. Efforts were always siloed. When a field office put together a canvassing packet for a volunteer, they didn’t know which doors to knock on and which to avoid. The canvasser didn’t know that a specific voter had a nice phone conversation with a volunteer three days ago and that their efforts today would yield more returns if they knocked on a voter’s door one block over.
Narwhal could change that. It would integrate all of the data the campaign gathered across the organization. But every connection point is also a possible point of failure, and they were building the most complicated tech system a presidential campaign ever had. And, unlike many engineering deadlines, there was no way to push back the date. They’d either be ready for the election or not.
“From the beginning of the project, it changes how you build software. Everything has to be done a month in advance,” Harper said. “You don’t allow for slippage. You cut features pretty surgically. The biggest thing that you can fuck up is missing your deadline of election day … or crashing.”
With four weeks until Election Day, the system Harper and his team built was feature-complete. They had their platform and were just making tweaks to their tools. That allowed time for Dylan Richard, Harper’s second-in-command, to run what they called Game Day.
“Game Day was this really fascinating experience of walking through and breaking all of your infrastructure to understand exactly how it breaks, how it fails, how it will survive, and what success is to us.”
“Dylan handed out a menu and said, ‘Here’s all the failures that we’ll have today, get ready.’ Then he just fucked it up.”
It was like a nightmare game of DevOps dungeon master. They roleplayed through each and every way things could break, probing the system for all dependencies and redundancies. It was stressful. Everything went wrong. But they learned how to make it right.
“Then we had our run book. If there was memory pressure on this database, then do this thing, we knew exactly how we’d go about fixing it. It taught us a lot about how to design exercises to understand resiliency.”
Four weeks later, on November 6th, 2012, 129 million voters cast their presidential ballot.
ORCA, the Romney campaign’s tool for getting out Republican voters, broke under the strain of election day. A spokesman for the Romney campaign said ORCA, named for the killer whales that hunted narwhals, had promised that it would blow the Obama campaign’s system out of the water. But it was unstable and failed at every level. Sources told Business Insider that “the product was never properly beta-tested.”
The Obama campaign’s tech stack felt the pressure of volunteers across the country getting out the vote, but they were prepared for it, and for all of the ways individual pieces could safely fail. And in the end the system held.
Shortly after 11PM on the East Coast, the networks began calling the election. President Obama had been elected to a second term of office.
The day after the election, Harper was back in at the campaign headquarters. “Maybe slept in a little,” he says, quite cavalier. “The President stopped by.”
“You know how some things in life end up being anti-climactic? This wasn’t one of them. We did all this hard work, and then it was just like, “Oh my God, we did it, thank you for everyone.”
The Atlantis shuttle is now retired. It sits on display at the Kennedy Space Center, now itself a piece of modern history. It will be remembered for all the things it accomplished. Traveling over 125 million miles, over 33 missions, carrying 207 passengers. Deploying satellites and docking with the International Space Station. All of which made possible by the less recognized efforts of those behind the scenes, working through each and every way that an aspect could fail, without complete failure.
And its mission lives on today in the Orion spacecraft that NASA is building and testing. In December of 2104 Orion completed its first test flight, and in 2018 it will orbit the moon. All building towards a crewed Asteroid Retrieval and Utilization mission in 2026 to rendezvous with an asteroid and retrieve samples.
And today the work that the 2012 Obama campaign advanced lives on in the 2016 campaigns. Tech staffers are building the tools to further the efforts of volunteers to drive Americans out to the ballot boxes in November and elect our the 45th President of the United States.