It’s Math, Not Magic: Four Lessons from Building a Machine Learning Team - Mixpanel

It’s Math, Not Magic: Four Lessons from Building a Machine Learning Team

Suhail Doshi

Machine learning insights for product leaders, from Mixpanel’s CEO

Everyone from chief executive officers to product managers to venture capitalists wants to understand machine learning better. They know it has the potential to take their software to the next level. They feel the excitement around it. They’ve read the TechCrunch or Fortune articles, and they’ve maybe even done a quick linear regression or two.

But the primary issue that many product leaders grapple with when it comes to machine learning is that they want programs that don’t just crunch the numbers, but can also think for them. They’ll say, “I want the algorithm to tell me why. Why are my customers not converting? Why isn’t this feature doing as well as I’d hoped? I want causation!”

Here’s the thing. The goal of machine learning isn’t to prove causation; it’s to make high-quality predictions on specific metrics, behaviors, or patterns. The algorithm’s sole job is to get you to your goal more efficiently and with greater accuracy, not to tell you why. That might seem a little disappointing on its face, but it’s actually liberating when properly harnessed.

Machine learning is creating a next-level data analysis evolution. The foundation being laid by machine learning today is helping engineers build better and more sophisticated software. In addition, smart companies are moving away from a growth-at-all-costs philosophy and embracing the promise of predictability. Predictability, it turns out, is the promise of machine learning.

But machine learning isn’t easy for an engineer; it’s not like picking up Python when you already know Ruby. None of our engineers, including myself, knew anything about machine learning even two years ago. I knew if we were going to build out product analytics using machine learning, we had to accept that we didn’t know what we didn’t know and understand that we were dealing with complex math, not magic.  


Our biggest challenges – and lessons learned

The thing that most interests me about machine learning is its ability to make software do more automatic things that were never before possible with just straight programming. As an engineer, this is exciting.

But as a product leader, machine learning presented me with some new challenges that I had never encountered before. Instead of trying to become a machine learning expert, I hired Jenny Finkel, PhD – an actual machine learning expert – to build out the kind of team that could successfully tackle the kind of machine learning challenges that would take Mixpanel to the next level.

I’m very proud of what we’ve built over the last two years, and even more proud of what we’ve learned along the way as we built Mixpanel’s machine learning team. Here are a few of the most important lessons:

1. You need experts. I realized very quickly that we needed real machine learning experts to build real machine learning products. A high percentage of top tier machine learning PhDs go to Google or Microsoft. To make up for that, people are staffing their companies with statisticians. This gets them close to a real machine learning team, but the level of complex math required for machine learning is not comparable to type of math statisticians do. I held out for actual machine learning experts, and it’s a decision that I’d make again.

2. It’s difficult to know where to start. While you might want to build crazy, complex algorithms right away, there are a number of foundational steps you need to take with your data to even begin to apply machine learning. You may not have the data in the first place, you may not have clean data that’s ready for modeling, and you may not have the models either. So, it’s not as simple as “just start applying machine learning to what we’re doing.” That’s a recipe for engineer burnout.

3. Scale always becomes an issue. Scale affects the success of machine learning in a much bigger way than pure statistics generally face. Engineers know that the volume of data as it relates to pure statistics doesn’t matter that much. This is why, for example, we can take random samples of e-commerce customers and get a fairly accurate model of how all customers will respond to an online promotion – we don’t need to know all the many combinations of products purchased or every possible potential outcome to understand the likely outcome.

Volume of the data matters significantly more, however, if you want to do machine learning. The more you understand every possible outcome, the better your prediction capabilities become. It’s not enough to merely sample a few past events or real-time actions; your models need to know every single case of something and converge on the right solution to confer a predictability advantage.

To continue the e-commerce example, let’s say an e-commerce company wants to go beyond predicting who will respond to a promotion and understand more complex purchasing patterns. Machine learning must learn how each one of an e-commerce company’s 2 million customers found the products they purchased, in every possible combination of paths to their purchases, to build the most accurate model of the how of purchasing patterns. In this way, future purchase patterns could be more accurately predicted with this huge increase in data granularity. It’s a little nuts to think about getting that level of detail from so much data, which is why you need a team of real machine learning engineers who can hit the ground running and build the best models.

4. Remember that predictability, not causation, is the goal. Businesses have always wanted to be able to do things like predict sales for the quarter, see how many customers they’ll gain thanks to a promotion, or know when to expect fluctuations in demand. Machine learning can majorly boost pattern identification, which simultaneously automates and drives predictability. This is why I knew I had to make a serious investment in the field.

None of these lessons should discourage you, of course. After all, the “why” of machine learning is simple for product teams: Machine learning can figure out what makes people sticky faster and more accurately than a human can – and then act on it, automatically. If you can set up a virtuous cycle of user feedback and better user experiences, then maybe you can get more people to use your product. From there, you can build this data network effect of getting more users, which gets you more data, which gets you better models, which gets you more users, and so on.

Your job as a product leader is to build great products and get them to your audience. From a product development perspective, machine learning is most exciting because it can automate away many of your other tasks and worries within analytics (i.e. anomaly detection). So, if you’re seriously considering implementing machine learning, then you should be seriously thinking about the problem you’re trying to solve.

Before you build a machine learning team, you have to know whether you have the infrastructure to match your ambitions. Realizing you might not is not a failing or proof of your being shortsighted; it’s just that machine learning is bleeding edge and everyone is figuring out how to best integrate its capabilities while reaping its benefits.


“Build vs. buy” is a false dichotomy

The bottom line is this: Computers are really, really good at memorizing a lot of data and then being able to find patterns and trends in all the data they memorized. Humans are terrible at both of those things. If you want to make a difference to your company’s competitiveness, you’ve got to push insights through great models that solve complex problems. And the only way to do it right now is through machine learning, with a highly skilled team that can figure out why patterns are what they are.

If you’re a smaller company then your engineers could build models on their laptops and that might be enough to get you started. But if you have a lot of data, a bootstrapped solution isn’t going to give you what you need from machine learning, and it’s not going to allow you to remain competitive.

More than anything, it’s about tradeoffs. Building a machine learning model can take a long time. Every minute your engineers spend on a non-core competency is a minute not spent on your core products and differentiators. In addition, most machine learning involves iteration at scale, since you have this large volume of data to collect, label, and repeatedly process. If you have to deal with that scale yourself, it’s an extra load on your engineering team, and tens of thousands of dollars.

You’ve got to decide what to build in-house and what you shouldn’t. We’re working hard at Mixpanel to bake in machine learning to product analytics in the right ways, not just “buzzy” ways.

Both Predict and Smart Alerts are just the beginning for us, because even if you’re just starting to track analytics, we want to help you focus on building the right products – not watching your numbers all the time. 

When machine learning is done right, it’s extraordinarily powerful. Machine learning generates accurate predictive models that can do really exciting things like create highly customized user experiences, classify thousands of objects in photographs, and generate a ton of groundbreaking and industry-specific results without being programmed to do so. The opportunity with machine learning is that you can build whole new civilizations – not just a single skyscraper with a view of an empty city.



Photos are from xdxd_vs_xdxdchrstphre campbell, and Sagie and are made available under a Creative Commons Attribution 2.0 Generic license.

Get the latest from Mixpanel
This field is required.