Why tracking less got VSCO more

In a meeting room in the VSCO office, on the corner of Broadway in Oakland, Steven Tang and Matt Turner stood in front of a whiteboard that was covered in little neon colored squares.

Each Post-it Note had a phrase scribbled across it: “Picture taken,” “Filter applied,” “Image saved.” There were tons of them, accounting for each and every event that VSCO was tracking in their photography app. Some were vital to evaluate the success of the app, but many weren’t. It was an illustration of a problem they already knew: their data had gotten out of control.

Matt is a Product Manager at VSCO, an app that enables over 30 million active users to create, discover, and connect through images and words. He has been there for four years and is part of the core team driving growth and engagement. He is now responsible for metrics across the company.

Steven is an iOS Engineer at VSCO and a member of the growth team. He has been there for two years and focuses on building features that drive user engagement.

To continue improving the app – expanding existing features and adding new ones to meet a growing audience of photographers – they knew they needed to confront the analytics problem together. Even though they were collecting all this information about how people were using the app, employees weren’t consulting the numbers.

VSCO was logging hundreds of millions of events, but teams weren’t venturing into the data to inform decisions. The data was messy and confusing, and sometimes it was just wrong. And that put the company at risk of being left behind in a competitive space.

Thus the whiteboard and the neon notes. One by one, they went through each event.

That was about a year ago. Today, Matt, Steven, and the team at VSCO have cultivated a data-informed culture that has fueled the company’s continued growth. Their Slack rooms, where employees ask and answer quick questions, are chock full of funnel screenshots, numbers, and links to reports.

Simply implementing an analytics solution isn’t enough to foster that culture. It’s not a box to check – being data-driven takes time, thought, and effort.

“It took awhile to get everyone on board,” Matt says. “Our data culture wasn’t that strong before, which is probably how we got into that state of disarray to begin with.”

And it’s still an on-going battle, as the product continues to grow and change. Still, VSCO went from a point where few could make sense of their data to today, where half the company has a Mixpanel account, and can quickly consult data to make an informed decision.

Here’s how they lost control, and how they got it back.

When everything is important, nothing is important

Analytics are inherently simpler when you’re a small startup with six people in a room. You’re probably tracking a single app, with a couple of funnels and a dozen events. But as you grow, like VSCO did, tracking how users navigate a more robust app is much more difficult. There are so many paths, so many actions a user could take.

The intentions behind tracking an event are always good. Maybe someone will want to see this someday. Better to be safe than sorry, right?

“There was this thought that we needed to track anything someone could do,” says Matt. “Then, five months later, you realize no one is looking at it.”

Answering data questions meant climbing through mountains of data and required tribal knowledge, making it off-putting for new members joining the growing team.

Something had to be done.

One day in February 2015, Steven and Matt put all of those events up on the whiteboard. One by one, they went through each and every Post-it Note, asking themselves a simple question: “Why?”

“The first pass was just us asking if there was any use for this data? Can it tell us anything we can actually act on?” says Steven.

Was this an event part of a metric that the VSCO leadership team need to see? How about the product team? Engineering? If not, then why were they tracking it?

The most egregious example was “Setting toggled,” an event that was firing each and every time a setting in the user preferences menu was toggled. What exactly does that mean? When does a user trigger that event? What information is passed along with it? And who at VSCO is even looking at that event?

“We were looking at our most popular events and saw an event we didn’t even recognize,” says Steven, referring to “setting toggled”. “I’m thinking, ‘What the hell is this? Who approved this?’”

The idea, Matt and Steven think, was simply to know which settings were being changed most often, and to float those to the top of the list. Simple. But once that was figured out, there wasn’t any purpose in continuing to track it.

“The event ended up being triggered millions of times a month,” Matt recalls. “And we were tracking it for … what, six months?” He says looking to Steven.

“I don’t know, but it was an insane amount of time, and no one was looking at it.”

That was a particularly messy one: it had no benefits, and it was just cluttering up the data.

“No one missed it. Once we took it out it was, ‘Good riddance, we don’t have to worry about that thing.’ ”

“The second culling was consolidation,” Steven says. “What events could we condense down into simpler events?”

They were still tracking a good amount of events, but by consolidating they were able to build reports that were more about surfacing metrics.

“At one point an engineer needed to know how long it was taking for people to apply edits,” Matt remembers. “And so every time you adjusted a setting while photo editing, an event would fire.”

Engineering was concerned about how long the app took to reflect each edit, and they just wanted to know how long it was taking for users to edit photos. Then they could work to bring that down. But VSCO has many ways to edit a photo. Exposure, contrast, clarity, saturation, tint, temperature, on and on … and each one was firing a different tracking event.

“Then it occurred to us, why don’t we take averages in the app?” Matt recalls. “Why are we sending 50 events when we could be sending one?”

“Now the app takes the average and sends it as a single event when you’re completely done editing. That’s the number they were looking for anyway.”

If it’s really important, then it better be right

Cutting and consolidating the events that VSCO was tracking down to only those that were meaningful was an important first step. But there was still more work to do. It wasn’t just that the data was overwhelming and hard to navigate. There was also a concern about whether the data was trustworthy. And if people don’t trust the data, they won’t consult the data.

Tracking so many events had another negative byproduct. They hadn’t been meticulous enough in how the metrics were implemented. VSCO had grown fast, and along the way, the quality of their data fell by the wayside.

“Part of it was we had tried to track so much and in so little time,” Matt admits.

Before the great culling of events, the app had gone through a top-to-bottom rebuild. Popular features were expanded, new features were added.

“In the process, all that code was getting thrown out,” Steven remembers. “Along with all the new features, we rewrote our old features to make them better. But not all the tracking was transferred over correctly.”

The team faced a pretty tight cycle of getting the rebuild out the door. They had separate teams, one working on iOS and the other on Android. And precisely how events were being tracked wasn’t a top priority.

“We had two separate platforms that we’re churning on,” Matt says. “And then at the 11th hour, it’s like ‘Don’t forget the metrics!’”

Everyone was grinding away and no one was taking the time to govern the data. No one was checking to see if the new builds were tracking each event correctly and consistently on iOS and Android. A simple event, like “Picture taken,” could be tracked in so many different ways, with so many different properties.

“Picture taken. That’s obvious, right?” Matt says. “But whenever someone takes a photo, what properties do you need to send with that event? We added a bunch of custom camera features. Do those need to be captured? Should they be booleans? Should they be an array of things? How are you going to segment this later?”

Matt continues, “People just make assumptions, ‘Oh yeah, ‘Picture taken’ means this and needs these properties.’ Then the Android team does this and the iOS team does the other.”

Small things could set off a chain reaction. An event could have been spelled “filterApplied” in the Android app and “Filter applied” in the iOS app, which would fragment the data in a report and lead to confusion.

For VSCO’s data to be trustworthy, they had to go back and make sure each event was implemented correctly and consistently. When they ran into an edge case, they would reconvene, discuss how it should be treated, and then document it.

“We had to set aside engineer hours. We had a team of people just focused on metrics for months,” Matt says.

As a product and an organization grow, things become more complex. You can’t expect how things work to just be in one person’s head.

“At this point, we’re three years in. We’re not a handful of people tracking a handful of events. It’s important to document metrics as you grow so other people can understand and take over different bits and pieces.”

For data to make a difference, people need to use it

The Post-it Notes exercise brought their overgrown events back down to what was important for VSCO metrics. And the documentation process improved data quality to make their metrics trustworthy and a source of truth. But there was still an important step left: educating the teams at VSCO about digging into and gaining answers and insights from the data.

“It started with company-wide presentations from the product and revenue teams,” Matt says. “These people already cared about really getting into the metrics.”

“Eventually we realized that if we’re showing these metrics in an all-hands, we can do it right from Mixpanel to show everyone how it works.”

“Then suddenly everybody was like, ‘Wait … we can do that?’”

Now half of VSCO has a Mixpanel account so they can quickly and easily answer their data questions themselves.

“We don’t give anyone an account unless they ask for it. We don’t automatically sign anyone up,” Matt says.

They wait until someone has a question for one of them about the data. It might only take Matt or Steven two seconds to find the answer, but it’s worth it for them to take the two minutes to show the person asking the question how to do it. This way, their introduction into the analytics platform for answering questions isn’t just another email invitation to sign up for an account. Instead, each new user would see immediate value and was bought in from the start.

“The other day the VP of finance asked, ‘How many people who actually buy presets are signed in?’ That was an interesting question, and I didn’t know. So we get into Mixpanel, and then two minutes later we have our answer.”

They’ve found that answering questions one-on-one and in small groups is much more effective than a large formal training session.

“We tried to open it up to more people at once, but they didn’t feel comfortable asking questions. Maybe they thought it was interesting, but didn’t want to waste everyone’s time with their question.”

Now Matt frequently holds small, informal sessions with coworkers to walk through finding answers to their questions, right there on the spot.

“It’s important that they feel like this is just a tool for you to scratch an itch you have about data.” Even though small and informal training might feel laborious from the start, it ends up having much better impact than the large sessions.

The upside? Teams at VSCO can make decisions based on how people are actually using the product, instead of relying on intuition.

“They don’t need to know everything, they just need to be able to answer their question. And then they’ll keep going back into the data again and again.”

Getting back in control of your data

VSCO isn’t alone in experiencing these types of issues. Metric tracking quickly becomes complicated and prone to error for any growing product. It’s easy to lose control. But that’s also when trustworthy metrics become even more important. As VSCO grew, they needed high level insights into how people were using a more robust app. They needed to regain control of their data.

It would be great if analytics were always a priority, but realistically, it’s easy for tracking to fall behind when a company is growing quickly. But by going back and putting the time and effort in their metrics, they were able to make them more meaningful and create the type of environment where a data-driven culture could grow.