Two years ago, QuizUp became the fastest growing game in the history of the App Store. But to stay relevant and to maintain its users, they knew it had to become more. With data-driven development, QuizUp has grown into an app that brings its users back time and time again, transforming from a trivia game into a social platform.
Every week, somewhere in the ballpark of 15 thousand apps are submitted to the iOS App Store. About a quarter of those are games. A quick glance at the Featured screen shows a “Best New Games” list with a couple dozen games released in the past week or so. They’re from names as big as Lego, and as small as Futureplay, a five-person shop out of Helsinki releasing their first game. It’s a highly competitive landscape. And each can only dream of the success that Plain Vanilla Games had in the first week with QuizUp: over a million downloads.
Five months after launching, the trivia game QuizUp had 15 million registered users – the kind of success most mobile games can only dream of.
And yet, this success exposed an even bigger challenge: gaming is fickle, and most mobile games are quickly abandoned, once the novelty wears off. The cellar of the app store is jam-packed with dusty old games that were once the flavor of the week. For users, these games become neglected icons on our screens, soon deleted to make space for the next cool thing.
107 weeks later, however, QuizUp is still going strong. How? Because after figuring out how to acquire users, they focused on retaining those users, using careful data analysis to see who were the most engaged and passionate. In the process, they developed the app into something more than the world’s best trivia game and started to grow the product into a new kind of social platform.
The world’s best trivia game
When QuizUp surveyed users to see how they found the app, the results were overwhelming. Users heard about it by word of mouth from a friend. And usually not a friend on Twitter or on Facebook. But from a real life friend, standing in front of them, telling them, “Hey, download this and play me in this Game of Thrones quiz.”
That’s how I found the app. Looking over the shoulder of a co-worker as she played a Disney quiz. And a few weeks later, home for the holidays, bored in suburban Delaware, that’s how I got my little sister to download it and play me in U.S. History trivia.
“Because of the nature of the game, acquisition is very organic.”
The process repeated itself in different regions and groups. There’d be a trigger, like a League of Legends tournament in Chile, and suddenly there’d be a spike in users playing League of Legends trivia, and QuizUp would spread virally through the country by word of mouth.
Very quickly, the app’s goal of being “the best trivia game” was realized, fueled by the initial strategy of focusing on organic acquisition
Becoming more data-driven
“When I first started at QuizUp, decisions were made by gut-feeling,” says Stefanía Bjarney Ólafsdóttir, Head of Data Science at Plain Vanilla Games. “They had to be. We didn’t have data. We didn’t even have users yet.”
But as millions of users found the app and data flowed in, it’s been Stefanía’s task to make sense of it all. To build up their event tracking and data structure. To get her coworkers asking the right questions. And to collect the data to answer those questions. Simply put, it was time to make QuizUp more data-driven.
Because, while QuizUp was hot, that alone is not enough. You can’t survive long term merely from being the it-game. Even a game as hot as QuizUp. Afterall, who is still playing 2048? Acquisition is only the first factor in growth. The second, equally important factor is retention.
“Our biggest opportunity for growing QuizUp is increasing retention.”
Today, if there’s one metric Stefanía and the team at Plain Vanilla Games look at every day, it’s retention. One day retention. One week retention. One month retention. Which users are coming back? And why? And what can they do as they build out QuizUp to turn it into an app that more and more users come back to again and again?
The answer, it turns out, lay in a secondary component of the app: the social aspect. From its launch, you could invite and challenge and chat with friends. You could earn and share badges when you reached a new level or unlocked an achievement. In essence, the game did more than just surface fun questions–it used those questions to start building a sense of community.
When Stefanía and her team dug into the data to see which users were retaining and which users churned, they noticed something. It was the users who did more than just play a trivia game that came back.
“For a game, QuizUp already had really good retention. We realized that was because it was a hybrid of a game and a social network. And it was the people who used the social features that were more likely to return.”
People who took the extra step to chat about the topic were much more likely to be retained. These users weren’t just getting value out of the game; they also got value from the conversations that they’d have around the quiz topics. Not only that, but they also started making requests to QuizUp. The hundreds of general interest quizzes weren’t enough for them, they wanted to create their own.
With this set of insights, QuizUp now saw what their new vision would look like: instead of being the best trivia game in the world, they wanted to be the best platform for connecting people through shared interests.
Even given these insights, how many apps with the success of QuizUp would be willing to make substantial changes to their app? But that’s just what they did. In May of this year, QuizUp saw its first major revamp growing from a trivia game into a trivia social platform.
“QuizUp 2.0 isn’t just a trivia game, it’s a platform for connecting people through shared interests.”
QuizUp 2.0 introduced a bunch of new social features – including friend and topic feeds with sharing and commenting, expanded people profiles, and a smarter user search – to further engage users around the topics they are passionate about. Then they launched My QuizUp, a new way for users to create and submit their own quizzes.
By opening QuizUp to community-generated quizzes, the amount of topics went from over a thousand to orders of magnitude more. Today there are millions of questions on topics as strange and unique as Odd Historical Deaths, Metal Album Covers, and Name the Cloud.
“Somewhere along the way, we realized we weren’t building a game but instead a new way for people to interact with one another. That was an interesting revelation. We are now connecting people all over the world through their interests and passions.” – Thor Fridriksson, CEO of QuizUp, Finding a Smurf
With the revamp, QuizUp has become even more niche and yet, counterintuitively, more appealing and stickier with users. Today, instead of just challenging my sister to a general history trivia game, I can play her in Gilmore Girls trivia, or I can play some random person from Bradenton, Florida in Pittsburgh Pirates trivia. After which we can chat about the merits of re-signing Pedro Alvarez or how much we hate the two Wild Card playoff system.
The new QuizUp has been very specifically engineered to encourage social engagement, the type of behavior that Plain Vanilla Games, thanks to their focus on data, knows will make a user more likely to return.
“QuizUp is a very data-driven company. It’s a mindset. Everyone is using data on a day-to-day basis to develop QuizUp into a better app.”
Questions QuizUp asks during data-driven development
A large part of QuizUp’s mobile client code base was rewritten for the May 2015 QuizUp release. The data team used the opportunity to revamp the data structure, which dated back to pre-launch.
“We spent weeks and weeks mapping out how best to track how people would use the new QuizUp.”
It was an intense period where they went through the design for each and every feature of each and every scene.
“We mapped out what we would want to know about it. By iterating on that process, we’ve settled on guidelines for the process of designing a feature from the data perspective.”
It’s a process specifically designed to make sure that they know why they want to ship a given feature.
“What is the outcome we’re working towards? And are we equipped to answer whether we are meeting our goals. Doing this has helped our clarity immensely.”
At QuizUp it starts by sitting down with the group that’s responsible for the feature: data specialists, product managers, developers, and designers. They discuss and answer these three questions:
What are the goals of the new feature?
It might seem a bit simple, until you sit seven people down in a room and get seven very different answers. You need to have everyone working to develop the product in the same direction. Answering this question helps QuizUp make sure the feature they ship fit their overall vision.
“In our product development, we have seen how easy it is to decide on a feature just because at a glance, it sounds like it makes sense,” says Stefanía.
“Having a clear goal and a desired outcome of a feature is a prerequisite for designing key performance indicators, and thus being able to understand how successful it is.”
How can we measure how successfully we meet our goals?
What does it mean for a feature to be successful? Again, seven people might think of seven different ways to judge how a feature is performing.
“To be able to understand how successful we are, we need to find a sufficient measurement. To make sure that QuizUp, as a whole, does not regress, we monitor our engagement KPIs closely for most, if not all, changes. In addition to our engagement KPIs, we decide on a couple of metrics that are specific to the change.”
What data gathering changes do we need, to be able to answer how successful the feature is?
The best time to think about measuring a feature is when it’s being developed. Think about the types of questions you’ll want to answer, and make sure you’re collecting the right data to answer them.
“Despite our minds not wanting to think of data gathering as part of a feature, data gathering can add extreme complexity to code,” says Stefanía.
“Including tracking requirement changes before kicking of the feature implementation allows developers to bear the requirements in mind when designing their code.”
If the tracking requirements are surfaced after the feature has been implemented, the code may need to be altered to accommodate for the required data gathering, which delays shipping and creates frustration.
It is crucial that the developers are included in this conversation, because it’s important that they, who know the code base, surface concerns about complex data gathering changes.
“If any such concerns come up, we might revisit the second question, to see if we can come up with a simpler metric to help us estimate our success. In most cases that’s doable, in other cases we’ll go for complex data gathering.”
Solving the growth formula
It would have been easy to simply ride out the early success of QuizUp. But being data-driven helped grow QuizUp from a hot new game that could acquire users into a social platform that could keep them around. They worked to understand how people were using their app, and what actions lead a user to be retained. Then they worked as a team to develop QuizUp’s social features and solve the second half of the growth formula. In the process, by utilizing usage data throughout the development process, they created the thing that their customers had been telling them they wanted all along.