Neil Rahilly Q&A: teams & customer feedback

As Mixpanel’s VP of Product and Design, and our former Head of Engineering, Neil Rahilly brings a blend of strategy, hiring, management, and career advice to those who want to push innovation at their companies forward. Distilling knowledge from multiple disciplines, this AMA session from Rahilly helps turn amorphous concepts in product management into a tangible guide.

Product strategy begins with figuring out exactly what kind of user that you want to serve and what use case that you want to serve for them. And I think that, in a B2B setting, you need to figure out both  what kinds of companies that you’re going to serve and then what kind of users within those companies you’re going to serve. I think often people are always trying to expand what the possibilities could be for their product. But strategy is all about focus and finding ways to not compete. 

So what you’re looking for is actually the most narrow definition of those things, the most narrow definition of types of companies, and users, and problems that you want to go solve. That’s still big enough to get you your next increment of growth as a company and growth in your user base. And then it’s about figuring out how do you leverage what you already have in terms of technology and in terms of the user base to go get that.  

 For us, in the last round of strategy, we started segmenting all the different types of companies we serve and users we serve. And we were looking at the ones where we have the lowest cost of acquisition, where we have the highest retention rates, and how that guides us toward an understanding of who it is that we should go after. 

A useful brainstorming question for yourself is, if there’s only one company or three companies that should use your product, who are they? And then you can start to define who that is for you and move out from there. And then you want to go and really focus on those users and make sure that you’re in a position where you think you can out-serve them compared to any other company that’s out there. Like on a basic level you’re going to invest more energy, more R&D spend in making it awesome for them. And that’s the moment when those users are going to feel like, in your marketing and your product, wow, this company, this is something that’s for me. And you can really delight those users. 

For Mixpanel, we’re going after digital native companies and the EPDs in them to help them make decisions about how to build their products, and that’s where we ended up. And then from there it just goes into looking at all the feedback that you have from those users, and going and interviewing those users. And then it really starts to flow. You know that you’ve got a good strategy when it just starts to feel easier to make trade off decisions. . . . We have some customers that care about this and some that care about that, and you’ve gotten to a point where it’s kind of obvious because it’s like, no, that’s not what we’re trying to do and who we’re trying to do it for. And whatever bite you’ve taken feels like something that you actually have the resources to chew, and you start feeling the momentum of being the best for that segment of users. 

Right now, we have about 100 people in engineering, product and design at Mixpanel. They’re split into four groups: analysis, data, admin and growth. And they’re not all the same size. Analysis is the biggest—almost half the team is in analysis. And then there’s about a third in data and then admin and growth are smaller teams. The analysis group is focused on the core, Mixpanel product analytics experience for the end users that come in and are trying to answer questions about their product. The data team is focused on getting data in and out of Mixpanel, and integration, and data trust. So how you get a foundation in place so that analysis can happen where you’ve got the data you need, and it’s accurate and people trust it.

Admin is all about the administrators, and info sec, and compliance, and all the other enabling teams within our customers who are key in either getting Mixpanel integrated with directory, and single sign-on systems, or other admin systems so it can be rolled out in the company, as well as making sure that we’re providing them the tools they need to feel comfortable from an information security standpoint. And then growth—Mixpanel has a self-serve freemium model, so how do we get more people signing up? And those people who do sign up, how do we get them to be successful to set up their first accounts or service on Mixpanel? 

For the bigger groups in particular, they’re then subdivided into smaller teams. And I think one key principle to how do we things, which is taking a while to learn but works well, is the interface of autonomy. It’s like how the different levels in the org relate to one another. So David, who’s the VP of Engineering, and I think about what should the groups be? And what should the major areas of investment be? And we work with all the leaders across EPD on that, and then we can invest based on the staffing of those groups. And then we work with everyone and the leaders of those groups to figure out our all-up purpose and measures of success. But then once we pass this down to those groups, the  director level and product design leads in those groups, that group is really for them to organize and they can plan as they see fit to execute on the outcomes that group wants to get. 

And then the same  even on project teams or even more persistent teams. There’ll be tech leads and PMs that own those areas. And again, they have autonomy to go and execute on the goals there. That’s important because then decision-making is very distributed and resources are quite well defined in terms of what purpose they’re going to be put to. And so you don’t get a failure mode for a lot of EPD teams. One that we’ve had at times in the past is where you have too many decisions that have to go up and down the tree, and things move really slowly, and you have too much context switching and packing of resources across different teams. You get the best result when you’ve really done the work to figure out, “Okay what are the goals?” And then you actually let teams go execute and figure out their own path and stick together. 

The other thing I’ll say on that topic is, I think we’ve come through a bunch of different phases. Mixpanel at one point was organized mainly by engineering functions, so there was an infrastructure team, and a product team, front end team, and an ML team, and a mobile team. And then we moved it to be product-based, so there was a team that owned various parts of the product. The most recent one is persona-based, where the analysis group is for that analysis end user, the data team is for the engineers that implement Mixpanel, the admin is for groups for admins and IT and Info Sec, and growth is for new signups for self-serve. 

What we ended up with though in those groups is kind of a blend. I feel like we kept some good things about each time period. So what I mean by that is like the analysis groups, and admin, and data, and growth that we have now, are still engineering groups as well. And they own the services roughly that are within their area, and so the on-call rotation and that kind of thing is within their groups. And similarly, they own areas of the product and now they also have this mapping to personas.  

The last thing I’ll say I think is interesting when you’re organizing is to think about reactive work streams and proactive work streams. Those areas of ownership, you want them super clearly defined for reactive work. So reactive work is like, “Hey this system is down, this customer found a bug, we have a big deal and someone there has really deep questions about some part of the product, should they talk to you?” 

You need to know who’s going to own that and who to route that to. But just as much when you go to do projects, you want to. . . . Let’s say we’re going to change how sharing reports works in Mixpanel. That could be the growth team because sharing reports gets new users to see reports. Could be admin, because it’s kind of permissions and sharing. Could be the analysis experience group, because sharing reports is part of doing analysis. And that’s okay because we actually only have six to eight big projects in the year. And so it’s relatively easy to coordinate those and make sure that people aren’t working on the same thing. And sometimes we even staff a particular project with multiple groups. But that’s how it’s changed and that’s how it works right now. 

I was kind of cynical when I got my MBA. Like, is this ever going to be useful? And I think initially there were times where I wasn’t sure. And then over time I have more and more looked back, and there’s a lot of stuff that I learned there that’s been really, really useful in thinking about organizational design and marketing and segmentation and innovation and strategy.    

At the same time, I actually joined Mixpanel as a front-end engineer. I was a self-taught coder. I’m the founder of AtomicContacts, this terrible app that I built that went nowhere. But I learned how to code and then I actually discovered Mixpanel building that app and used it, and was like “this thing is so cool.” And that’s how I joined the company. When I interviewed, they were very skeptical because I didn’t know how to program that well. So I joined as the most junior front end coder on the team. I changed Mixpanel’s buttons from images to CSS, that was my first task. And then I worked there, and have grown with the company, and have seen a lot of different phases and stages, and found different people to be mentors, and learned different things. I think the learning from experience and just doing it was the more important part. 

But it was really awesome to have this reference back to things from school and go, “oh my God, that actually would be useful here.” And I’d go dig up that book or look it up. But I think I would recommend more going and working at a high-growth company. Because high growth companies, they always have needs to promote people faster and give people more responsibility than they’re ready for in many respects, which has happened to me a lot of times in Mixpanel. So that’s where you’re going to really get that hands-on and forced learning. At least for me, I feel like I am so much more motivated and get so much more out of, whether it’s books about programming, books about business, books about product, when I feel like these things actually have answers that I need right now for a problem I’m trying to solve. And there’s so much amazing material online now, you could probably just get in the job and then try to learn as it’s relevant to you.

At Mixpanel we have what we call the gap system. For us it’s built into Salesforce, which our whole go-to-market team uses for everything. And it lets anybody across all of Mixpanel log a gap on an opportunity or an account or just anything related to the product that the customer complained about, or requested, or an idea they had. Anything. And then, obviously, it’s connected to the accounts so we can see all the features of the account. We also have some key fields—one is whether this is nice to have, a work around (which just means you can work around it) or a deal breaker. So we have a level of severity. 

And then we also have a broad categorization. It’s like a peripheral nervous system for the company. Every contact that we’re having with a user or customer, we want to know, we want to pick up all the information that we’ve got and make sure it flows back to the right places. So we can route based on category, and the gaps just go to the right Slack channel for the teams that own that stuff. We also dump it all out into a big Google sheet where you can run all these pivot tables on it. And then that lets us answer a lot of really important questions, like, “in the last two quarters. . . what were the top things that for customers in this segment of this vertical were the deal breakers?” That kind of thing. And it’s really useful. 

I think there’s some other equally useful sources of input from customers. One is user research, like going out and just having them use the product and recording a video and just watching the truth of what it is to use the product. And the other is support tickets and also the little comments on our NPS responses. And we’d always just try and do a better job of collating all that information and making sure that it’s driving our product prioritization. Because as you get bigger and you have more customers, that’s a competitive advantage. It’s like one thing that we have that an analytics startup doesn’t have is tens of thousands of customers who are giving us that input all the time. And that’s proprietary information. Sometimes people will talk about competitors and people are like, “Oh I wonder what’s on their roadmap.” And I’m like, “I wonder what’s in their gaps.” Because that’s what we could go and build and have a better product than them. 

I think you always want to have your company/business KPIs, and then your user KPIs—and you want to connect the two. The company KPIs for product, are either going to be on the acquisition side or retention. And I think you want to go with retention if it’s not good first. Because what’s the point of filling up a leaky bucket?  

And then on the acquisition side, the one that we’ve found product is most directly on is win rate. So for us, we have a win rate in the segment that we care about most. You can break down win rate in a number of different ways, for example, by competitor. I think overall win rate tends to be too general but if you really try to do a sales push in a particular segment, or you’re going after core set of types of accounts that you want, you want to look at your win rate on those deals and then similarly on retention. 

Right now, we’re focused on the retention side of things and then that’s like your business KPIs. For us, the all-up metric is going to be net retention in EPD. We always typically have some kind of data center costs type; how much money are we spending on data center?  

But the other big one is users. How are your users doing? What’s the user health? How much value are users getting? This is something we’re actually iterating on right now. You want to have a sense of what the fundamental value that you create is. For Airbnb it’s a stay, for Uber it’s a trip. What is the core thing that you’re doing to create value in the world? And for an analytics product, that’s answers—we’re trying to help people answer questions. And so that’s our all-up thing. 

Now sometimes you can’t actually measure that. The problem for us is that people are running queries, but we’re not sure if we actually answer their question or if we fail to do that. The closest proxy we have is “asked a question” or “ran a query” and we look at it in conjunction with retention and we assume if people are asking a lot of questions and coming back and asking a lot of questions, we’re presumably giving them their answers.  

The other is reach within accounts, so when people ask us to come into their company and rollout Mixpanel, what they’re looking to do is effect an organizational change, to make the team more data-driven. We want everyone using Mixpanel to make decisions. Sometimes I think about democratizing data; one version of that is democratizing dashboards. But what we’re trying to do is democratize analysis. Give everyone the power not just to consume some KPI, but to actually dig into the data, build their own dashboards, own reports, and drill into the data and answer their own ad hoc questions. 

So the two sub-metrics of how many answers we provided to a particular account or overall as a company is, how many users? How much reach did we have within that account? And then in particular, what was the quality of that reach? And so we have different cohorts of types of users—from a consumer (someone just viewing reports), to a learner (someone actually analyzing the data), to a creator, (someone who’s creating their own reports and potentially sharing them). We’re really focused on how many of those higher levels of mixed value—in terms of the answers you’re getting out of Mixpanel—can we get users to?

In some ways it is [strongly defined], in some ways it’s intentionally disorganized. Probably the most critical thing for us is that engineering product and design are in really tight collaboration from start to finish, and that the people who are working on it work just on that from start to finish. Sometimes I call it single-threaded ownership. I think you make the best products when you work with a team of other people—like some engineers, a designer, PM—and you’re all in it together. Not like the designers doing a little design for this team, and little design for that team, and the PM has three different things on the go. It’s like that group comes together every day and there’s a whiteboard and six desks and they’re working on it together. And similarly it doesn’t work out as well when the PM just goes and writes a spec and then hands it off to design to do a design and hand it off to engineering to go and implement it. 

You really need everybody completely involved. Even at the stage of just understanding the problem, deeply understanding the problem and figuring out the scope and the goals. Because then the purpose and the constraints are all just clear to everyone. We don’t always get that right at all, but that’s what we’re going for. That’s even true within engineering; sometimes you’ll see us like, “the infra team is going to go off and build this thing according to this API and then the front-end team’s going to…” You wind up just making all these locally optimal decisions and you miss out on really critical ideas and input from the engineers, and the designers, and everyone at the outset. A big mistake we often see is where teams ask for feedback at the last minute, where they’re like, “Hey, we really want it to be collaborative, so here’s this 90% baked thing, and we just want your feedback.” 

At that point, either you’re debating the finer points of things or it’s just going to be frustrating because the team’s not going to really want to hear the feedback because there’s so much sunk cost in what they’ve done. If you’re going to get everyone who’s going to be involved downstream involved at any time, it should be at the beginning. Because it’s really frustrating that by the time it gets to engineering it’s like, “Hey we wanted to solve this problem, we went that way and it’s like why didn’t we go this way? That would have been so much better.” You want that input right at the start. 

Basically you avoid a lot of communication and coordination and other types of execution problems just by trying to create that environment. One way you could think about it is: you’re always trying to decompose the company into states where you’re basically simulating a five- or six-person startup. Because then that team can just go operate like a five- or six-person startup, the most productive organizations around.

If you’re starting out, I think the truth is learning is probably the most valuable thing you’re doing at work. And when you start out you always need mentors. Just pick somewhere where you think you can learn a lot. And the biggest gift you can have in some ways is knowing what you want because that’s a lot more possible than people think a lot of the time. It’s just you need to really want it and just go all in. When I started at Mixpanel, the CTO there, from an engineering knowledge and skill level, I just wanted to be able to build the things that he could build and know the things that he knew. And so I just focused on that and we became friends, and we hung out outside of work, and it was just like constantly soaking up what he knew. And that was the most valuable thing. 

As you go up, one other thing to consider is what you can bring to the organization. There was some point where I realized, “Okay, I actually kind of know how to build a good engineering team and I know how to recruit really good engineers.” And that’s a skill.  I remember thinking, “That would probably be the thing I could bring to another company to be super valuable.” Because usually with 10-, 15-person startups, if they’re really scaling, they’re like, “how do we go and hire the engineers that we need, and the designers that we need?” And that’s a bit of a mystery. 

So you can look for places where it’s like, “Hey, I could bring something that I’ve learned and I could keep learning. But I’m also coming in at a higher level, and so I also need to be able to bring something that’s new, and of value, and knowledge and experience that’s a value to that company.” When you’re in a company that’s scaling really fast, you see a lot of executives or mid-level people getting hired in. And they have to do that, otherwise, the people who get layered by them are like, “Why? I know more than this person.” And that’s not a good spot to be. And then exec, I don’t know, I’ve only been an exec for a year and a half. I don’t have advice on that one yet. 

 I think you just want to go somewhere where you feel you can learn a lot and you really like the people and you feel comfortable there, and it’s something you enjoy. And then it’s like anything in life, you get out of it what you put in. So you’re looking for the thing that seems like fun, and actually it’s like, “I can’t believe this is a job.” Coming out of my MBA, programming was this hobby where I thought, I’m not actually a programmer. Joining Mixpanel felt not like a job, and I think that’s been a big advantage. So I would look for that and then look for people you can learn from.