Kaplan Test Prep & mobile app conversions
Higher education students in the United States have to battle it out for seats at prestigious institutions whose admission officers, out of necessity, move through applications very quickly – sometimes in eight minutes or less. That makes standardized test scores even more critical. If an admissions officer scans a student’s application and sees a low score, it increases the chances that any student could end up in the reject pile, despite their other virtues.
The pressure on students to perfect every aspect of their application has also intensified the competition in the private education industry. To stay ahead of the hundreds of other online tests prep products available, Kaplan Test Prep, a clear leader in practice testing and tutoring for more than 100 standardized tests, works continuously to improve its digital practice environment. The Kaplan Test Prep team currently utilizes their own proprietary “Atom” platform technology to serve materials to students, empower educators, and provide content creators with robust feedback on what is or isn’t working.
Part of what has successfully differentiated Kaplan Test Prep’s Atom platform from other tools on the market is its product team’s deep investment in data, analytics, and the Atom platform mobile experience. Smartphones play a huge role in the lives of the students who make up Kaplan’s target market, and, as a result, mobile learning has rapidly become the industry norm. Still, when it comes to mobile experiences, many testing apps tend to miss the mark, frustrating students with clunky sign-in flows and confusing instructions.
The Kaplan Test Prep team predicted that if they could reduce friction and maximize positive interactions on both Android and iOS – from the very beginning of a student’s selection of Kaplan Test Prep resources to the actual delivery of a student’s first assignment – students would keep coming back to the Kaplan Mobile Prep app. But it wasn’t enough just for the test prep industry leader to create a mobile app: user analytics were needed in order to see the full picture of how students interacted with the platform’s content and other measures to evaluate student engagement. Mixpanel was selected as the primary data source for Kaplan Test Prep’s mobile app on iOS and Android, titled “Kaplan Mobile Prep.”
“We saw immediate value in Mixpanel after trying and implementing it on our app. Our effort in gathering analytics was immediately justified by bringing us insights that helped improve our mobile offerings,” said Jeff Sugar, Senior Direct of Product at Kaplan Test Prep.
Mixpanel has given the Kaplan Test Prep team a deeper understanding of how students engage with Kaplan Mobile Prep, so they can alter the product, content, and business strategy to improve customer experience. With A/B testing and granular insights, the Kaplan Test Prep team has replaced guesswork with data-informed decision-making that facilitates innovative, substantive change.
The results are significant. Since Mixpanel was implemented, Kaplan Test Prep’s mobile app has attracted thousands of new student prospects, and increased the student conversion rate by over 40%. Here’s how.
Zooming in (and out) on the customer journey
Mixpanel allows the Kaplan Test Prep mobile team to zoom in and out on Kaplan’s customer landscape and identify mainstream and niche user behavior patterns in the data. Jeff notes that the flexibility of Mixpanel enables his team to segment users at the specific user ID level, which allows Kaplan to see more granular user trends while still analyzing cohorts as an aggregate.
The Kaplan Test Prep team uses these insights to create hypotheses about how the product can be improved, and subsequently verify that these hypotheses result in a higher level of product quality and effectiveness. These improvements allow the Kaplan Test Prep team to maintain its focus on the student user experience and mitigate risk even while implementing changes.
Winning students and non-students
Improving the sign-up flow was at the top of the list for Jeff and his team, as that would enable individuals who didn’t have a Kaplan account to use the application. To improve this onboarding experience, they tested a new sign-up feature that would drive more conversions by addressing a major product gap in the sign-up flow that the team identified using Mixpanel.
A little background: in order for users to see Kaplan Test Prep’s exam-prep content, they have to sign up for an account. Once they have signed up and created a username and password, they can log in and gain access to all of Kaplan Test Prep’s substantial resources. However, the first iterations of the mobile app didn’t include a sign-up option at all. As a result, individuals who were not already enrolled in a Kaplan program and did not have an existing account would download the app and find they couldn’t log in, rendering them unable to take any action within the app.
The fix? The Kaplan team added a new selection to the flow which gave users a choice between logging in (if they already had a Kaplan account) or signing up (if they needed to create one). The decision to add the selection point was driven by the hypothesis that having a sign-up option would result in substantially more users. Jeff and his team were able to confirm this hypothesis by measuring the number of users who successfully completed the sign-in flow through Funnels reports. What they learned was that the mobile user acquisition rate jumped significantly within two months of releasing this update, and drove up total user counts by more than 20% in that same timeframe. “We’re thrilled because Mixpanel has given us a new avenue to quantify the number of students who are willing sign up for this test prep app.”
To measure the success of their app, Jeff and team also look at the percentage of non-students on the app – that is, the percentage of individuals that have never logged in, even if they’ve created an account. They measure this because if students perceive the app as valuable, the number of people who have signed in should be very close to the number of people who have downloaded it, and the percentage of non-students should be low.
“We were initially at 48 percent [of non-students] month-over-month, year-over-year, and have improved that number significantly since then.”
While analyzing non-students in Mixpanel, another insight emerged: a significant percentage of people who download the app are looking to prepare for exams that aren’t supported in the application today. This points out an obvious opportunity that the team is now exploring: building an richer library of content that supports more exams (and more students).
Hooked on learning
Kaplan Test Prep is seeking to increase mobile usage, and the team has been able to use Mixpanel to identify the new content that is likeliest to increase usage. As Jeff sees it, “There are some people who will download the app and determine that it doesn’t meet their specific needs. Not all of our programs or materials are supported natively right now, and Mixpanel helps us decide which exams to prioritize based on the number of would-be customers searching for them.”
On the learning side, Kaplan Test Prep has launched an adaptive quiz-maker – Qbank – that caters to mobile and web users alike. The Qbank allows students to request a series of questions on a specific subject, e.g. forty questions on biology. Unlike other quiz-makers, the Kaplan Adaptive Qbank adapts the difficulty of the questions to the student’s skill level, ensuring that they’re getting questions of the appropriate difficulty at all times.
“With Mixpanel, we’re able to track and validate student perception and usage, to see whether the number of questions reviewed by students is going up, for example.”
Kaplan Test Prep is also using Mixpanel data to eliminate features that don’t enhance student learning and engagement. A “Teach Me/Test Me” element was originally designed to help students navigate either to teaching resources such as videos and readings, or to tests. Comparing the number of unique clicks to the number of repeat clicks led Jeff to a clear conclusion: students were finding the instructional materials they wanted on their own. “We found out quickly that students didn’t actually use this feature as designed. The data told us it was just a distraction, so we were able to adapt and iterate accordingly.”
Insights like these give the Kaplan Test Prep team time back not only to expand their offerings in ways that benefit their user base at large, but also to customize experiences so that individual students get the most from the mobile app.
Communicating for completion, impact, and retention
Since its implementation in the Kaplan Mobile Prep product, Mixpanel has helped Kaplan Test Prep steer students away from content that they might not find engaging or that would turn them off of the app entirely. Using Mixpanel’s Insights and Explore functions, the Kaplan Test Prep team have been able to test theories and validate the connection between student engagement and optimization for the mobile experience. While students were opening tests and assignments on the mobile app at the same rate as they did on the web, they were not completing them on their mobile device with the same level of frequency. The data highlighted the fact that most of the assignment material pulled into the app from external sources through a webview was not optimized for mobile. Based on this data, the team has prioritized optimization of the mobile test experience to encourage more students to stay on the app and start completing tests and assignments on mobile.
The Test Prep team also uses Mixpanel’s automated in-app messaging feature in to give students the best user experience possible. When students don’t have the most recent build of an app, for example, they are automatically notified that a new build exists, and given an overview of its new features. Here, two of Mixpanel’s features work together. The automated messaging feature prompts students to upgrade, and then, if the conversion rate isn’t satisfactory, the Kaplan Test Prep team can use Mixpanel A/B testing to see which design, messaging, or timing will boost conversion rates.
The automated in-app messaging has also enabled the Kaplan team to address student confusion about which app to use. Kaplan Test Prep offers multiple apps, and often students who have enrolled for product A on the web will download product B from the app store by mistake. To address this, the Kaplan Test Prep team created a push notification to be sent instantly to students who made the mistake and eliminated the problem entirely.
Seeing the user journey ahead
The ability to identify and nip potentially significant problems in the bud is why Jeff has put a large organizational effort on data collection and analysis. “I want to be able to say, ‘This is the singular student profile, John Smith. And John Smith uses mobile every morning, but then switches to desktop for serious practice at 5 pm and then uses mobile again at bedtime.’ I can say, ’Show me all users that have done this thing this many times in this many days and that’ll indicate to me that they’re a power user.’ That way, I can quickly identify them what’s working and what’s isn’t with little effort.”
To extend Mixpanel usage throughout the organization, Jeff has prepared several videos to train his colleagues on key Mixpanel functions – also known as Sugar University. In these videos, he teaches teams how to pull in data, how to use key reports including Segmentation and Explore, how to perform A/B testing, funnel analysis, and more. This internal rollout where Jeff helps teach members across different organizations how to answer their own data questions, without analytics experience, is one pivotal way Kaplan Test Prep is building a data-driven culture. Now, teams can make more informed decisions based on user behavior insights, and this is a fundamental factor in making Kaplan Mobile Prep a success.
