Sprig’s Kevin Mandich on a decade of building with ML and AI

I've worked in the ML space my entire professional career. I started out working at a legal tech company, moved on to a cybersecurity company, and spent some time at an AI consultancy before I ended up at Sprig.

I joined the industry at a pretty interesting time—it was 2014, when big neural networks specifically for image recognition were just beginning to launch. That was a really big year for neural networks in general, and I got the chance to start using these large models professionally in the years to come.

Up until then, what we call old-school ML—which are basically just statistical models—were pretty prevalent, and they still are at a lot of companies. So most of my career until I joined Sprig was working with relatively simple models to do relatively simple tasks.

In the beginning, still in the pre-OpenAI days, we rolled out all our own ML models. We constantly trained them, and it was painstaking and expensive. We had a team of annotators to look over data, review it, mark it if it’s incorrect, and stuff like that. That lasted about three years until OpenAI came around and just blew the top off of everything. It leveled the playing field across the industry.

Being in the industry, it was definitely a mix of both. You could see the trajectory of the technology as it was being developed. 

Image recognition, for example—2014 was a massive step up from 2013, but it was still pretty rudimentary. It would mix up a blackboard eraser and a mobile phone, right? It was cool, but it wasn't quite there yet. But it just kept getting better and better. It was obvious where it was going, but I think that the modern LLM movement took everybody by surprise.

"In the beginning ... we rolled out all our own ML models. We constantly trained them, and it was painstaking and expensive. We had a team of annotators to look over data, review it, mark it if it’s incorrect, and stuff like that. That lasted about three years until OpenAI came around and just blew the top off of everything. It leveled the playing field across the industry."

It's definitely a mixture of everything. I've seen the machine learning hype cycles over the last 10-plus years, and it's always the same. Something comes out and a bunch of articles get written and the hype goes up. Then people try to implement it and realize it’s wrong 40% of the time and get upset.

I think tech has driven a lot of the recent changes around how people create and package products, and I don't think we've seen the extent to which that will change with this new technology. I can see that product packaging piece catching up to the advancements very soon.

With AI, you just don't need the expertise that you needed five years ago to build products to automate these tasks. A few years ago, you would’ve had to do a lot more work—you had to make sure your inputs were perfectly formed to get the specific outputs that you wanted.

Now, you don’t have to do that anymore. You don't even have to know too much from a technical perspective because these models have so much context and memory stored about what they've seen people do on the internet or the data they're trained on. 

The barrier to entry is super low now, so AI is just another tool in the toolbox, something people can use to just build stuff faster. I don't think it really changes anything fundamental about the creativity needed to build good products or create value. That part is still the same as it's always been—it's just easier to do it now.

"The barrier to entry is super low now, so AI is just another tool in the toolbox, something people can use to just build stuff faster. I don't think it really changes anything fundamental about the creativity needed to build good products or create value."

We started by analyzing Open Text feedback, because that’s historically something product teams spend a lot of time on. 

That formed the foundation of our work today, which falls into two major areas. One is the analysis of different modes such as video recordings or session replays of what somebody did—or heat maps, because there’s a lot of data there. 

The other is branching out from just analysis and moving into the entire product research lifecycle to see what we can automate or improve. That second part is exciting because the focus on that whole lifecycle is part of our new company vision.

If you're a product builder, you have to figure out where to even start collecting product experience data. “Why should I collect it? How do I collect it? Once I collect it, what do I do with that information? What are the follow-ups?”

Take our study creation assistant feature—you’re the product owner and you don’t have a research team, but you want to get some information about how customers are using the product or what they want you to build. 

You have no idea how to survey your customers or what the best practices are. Sprig has a tool where you could just enter your goal and say, “Hey, I want to do this. This is my customer base. This is a description of my product. Show me an effective survey that adheres to research best practices,” and it’ll create that survey.

Then, once you've collected your data and Sprig has analyzed it for you, we can also provide next steps, like helping set up A/B testing for future changes and offering recommendations for code changes. That's the next big thing for us.

Yeah, we’re always trying to gather feedback. Every AI-driven feature has a feedback mechanism. So directly in the dashboard, users can give a thumbs up or thumb down, type in what they think about it, and so on. 

We’ve always had those kinds of feedback mechanisms so nothing's really changed there, and it gives us a sense of how well those AI features are helping our customers.

How much more difficult it is to determine whether that technology is working well. For example, the Open Text analysis feature I mentioned earlier—the whole point of this product is to automate the survey analysis process for our customers and to give them the answers without them having to put 10,000 surveys into a Google Sheet and spend weeks summarizing, marking, and aggregating responses.

That’s a very subjective exercise because you could look at a thousand responses and come up with many different ways to group them, and two people can do it very differently. So you’d expect to get a decent amount of feedback—if you look at the results from somebody else’s analysis, you'll likely have some opinions about how they could have changed it or how the analysis could’ve been different to suit your needs.

But surprisingly, we didn't see that, which I think means one of two things: maybe users didn’t have an example of what they expected to see from the results, or they just didn't have the time or resources to see if that raw data met their expectations. That's a new challenge that didn't exist before AI and a signal for us that Sprig is providing a valuable service.

"It’d be really interesting to see something [from AI] that can enable a single product manager to do the work of an entire team. Even then, I think that's a pretty conservative guess as to what we're going to see."

With this industry and this subset of products, I can easily see an agent-based approach that automates the entire thing for you. You could say, “I’m building a product and I need you to take over the entire product research cycle for me, so here's my code base. Here's access to everything you need.” Then you’d collect data and drop in surveys, and then maybe it'll use that data to run follow-up surveys, schedule interviews for you, and provide code changes or product suggestions. 

It’d be really interesting to see something that can enable a single product manager to do the work of an entire team. Even then, I think that's a pretty conservative guess as to what we're going to see.

I think people are realizing we're rapidly approaching a world where a lot of content is going to be AI-generated, and we're going to get to the point soon where legitimately good human-created content that's unique is going to come at a premium. 

It might mean that people who can create content that’s outside of AI’s knowledge bubble—who can truly push the boundaries—are going to be worth a lot more than today. The current state of AI is not something that can dream outside of what it already knows. It just sort of interpolates between things that it’s already seen, and it can't go outside of its knowledge bubble. 

So maybe we end up with human content creators being much more valuable in the future. Robotics is pretty cool, and so are the new applications with computer vision—it's also terrifying because we've all seen Black Mirror. Who knows where it can go?