Dots And Lines in Product Management, with AI Pencil | Çağrı Akgül

Welcome everyone. How many of you heard this one before? Please raise your hands. Quite a common one, right? Well, I'm a bit into quotes, a quote nerd. You will find it out in a bit. So how about coming up with our own quote from this room? What else is there that you can't manage if you can't? Let's try to fill in the blanks here. Just throw some ideas, suggestions. What you can see, what you can't understand. All right? I really like this one. It hits a painful - place in my heart and mind as an ex-data analytics person. Yeah, we have been following Peter Drucker's mantra - and have been trying to measure everything we can, right? At the same time, our capability to make sense of it, - to understand all that stuff, - I don't think that has been developing, - And that's what I want to talk about. You can quote me on this one. "You can't manage what you can't measure and understand." We are co-quoters. But who am I? I want to tell you a bit about my background - so you can understand where my perspective is coming from. Said it, software engineering. Came to Finland 20 years ago. Came for Nokia. Stayed for the weather, of course. Then I took my chances with doing a startup. The Mike Tyson connection comes from another quote. You might have heard it lately. "Everybody has a plan until they get punched in the face." I mean, it wasn't a bad story. A lot of learnings, - which I thought I can share as a consultant, you know, workshops, - lots of post-its. Then I thought, hey, let's go back to building stuff. I joined their massive digitalization effort, - lots of different industrial digitalization projects. And while I was there, I thought, hey, let's build a design system, - not just about UI, but also other things that the design covers. How you discover the problem, how you iterate on it, this and that. Then I joined the forest business unit running the data analytics team there. I've been churning out those dashboards you saw in the cartoon before. And then about three years ago, I joined Nest as a product manager. Working with the Nest app, serving hundreds of thousands of users in - Finland and Baltics. So lots of data points. There's the user feedback, there's the transaction drop-off rates, - all sorts of performance metrics, reliability metrics, - service metrics, this and that, - that we started to collect and a little bit started - to drown in it. But after all this - programming, designing, managing, and overall building products experience. I arrived at this three-leg stool - for successful product management. But I want to emphasize, this is about successful product management, - not about a successful product. That's a whole different story, as you know. Problem solution fit, product market fit, - and mostly luck. But you also need successful product management. So yeah, three-leg stool because, - you know, one or two legs can be super strong, - but if one of them is weak, you might not want to sit on that stool. At any given time, you need the agreed upon expected outcome, - the alignment, right? What are we aiming? What are we trying to achieve in one to three years? Next quarter, next increment, next week's release event. At any point, it needs to be clear what the outcome is you are expecting - from your effort so that you can have a focused effort towards that outcome. I think we have been talking about this - and doing pretty well in most product companies. But clear visibility and understanding of the current situation, - we are churning out all those outputs, but where are we? I mean, we can say, yeah, we did 80% of our to-do list, - backlog, whatever. But what does it mean when it comes to the outcome - you are after? That's why I ran into this beautiful framework, - North Star and Input Matrix, which you might be familiar with. But let's have a short refresher. The idea is you might have tons of different KPIs to look at - to understand how your product, business is doing. But if you had to choose one, the single strongest leading indicator as much as - possible, every indicator is a bit of a lagging one, - but the leading one, - for customer business value creation, what would that be? What would be that metric? Once you have that, what are the most contributing factors, - the input metrics for that one? So that's the gist of the idea. Some examples from the industry. For example, Airbnb, - I'm sure they are tracking how many accommodations are there in their - webpage on a given day. How many registered users, revenues, profit per accommodation unit, all that. But they are going after this, allegedly, - number of booked nights to understand if they are creating - customer and business value. Because those listings and users don't mean - anything if they are not taking action and booking those places. Yeah. And this is how the whole thing comes together. You do have... I don't know, yearly EBITDA goal, maybe a sales-driven goal. So that stays there. But the problem with such goals is, - let's say you aim to finish your next marathon under four hours. That's your goal. When will you know if you have achieved it or not? And you cross the finish line, right? And if you haven't, you haven't. If you have, congrats. But that's not really how you can measure while we are preparing - for that marathon. That's the thing with the end of the year - financial results and that kind of stuff. So you need something in between that's going to be a leading - indicator for whatever is that end lagging goal that you're after. The input matrix. And connecting your work, - the efforts, your to-do list to input metrics, - I think that's one key thing so that you understand we are doing this for - that, it should affect that. Okay, so all good. But we can do better with the premise scheme - that you are all invited to join. I promise you the riches and successes. Yeah, I mean, why not? Let's build a semantic hierarchy around this thing. You do have the North Star metric there at the top. Then maybe you look at what the users are doing. Growth, retention, churn, engagement. Then what do they think and say? Of course, this matters, but I have to say, what they do is more important - than what they say. They're not being dishonest. They're not trying to mislead you. It's just the way it is. You know... We might end up using things we don't enjoy that much, - but there's nothing better or it just does the job. So we go ahead for now. Okay, then, hey, without this, you don't have anything. You're not even in business, - if your product is not doing what it's supposed to do. But of course, reliability might change from time to time. You might want to track that too. Then the bedrock, - which we tend to neglect every now and then. The team health, the code base health. When was the last time you sat down with your code base and asked, - how are you feeling, dear? How are you today? How is your cyclomatic complexity? We don't do that, although it has a major impact on whatever we are after. Connecting the work items. You have to do this refactoring exercise, - work item number three, because it should help with our - code-based health. You have to do those connections so you understand - how all those efforts are coming together. All right, you did that. You collect all the key metrics. They are there beautifully visualized on a dashboard that you are following. So what? What now? I mean, I did promise you success with the Pyramid Scheme. Am I scamming you? Here's the thing, the power of going back. Such a powerful thing, going back to the drawing board, - which we don't use enough. Why? I don't know, human nature. We don't want to admit if we did the wrong things. The way our upbringing, school, education, even workplaces, you know, - at the end of the day, you are still expected to come up - with the right answers, right actions. That's what you get paid for. So even if you know you have made a mistake, - or even if everyone knows you have made a mistake, - it feels like it is in everyone's best interest to, - hush hush, let's not put it out there and so on. But no, that's wrong. You have to go back and - fix things up. So let's zoom out for a minute. The key questions we are going after here, - what do we want to achieve? What are the contributing factors? And how do you understand if you are getting closer or further from that? How does it look? What happened last month? So you have the descriptive analytics covered, right? Then this part, why? What caused those casualty analytics? Well, that's where things get a bit muddy. If you leave it to people's opinions, insights, - we will be all over the place. Engineers, managers, designers. You can get three of them, ask a simple yes-no question, - and you will get five different answers, at least. Luckily, casualty analysis has been a high interest area for scientists - and academicians for a long time. For many decades, people have been trying to solve this puzzle. And then we have the LLMs entering the stage with the fireworks. So smart people started to look into that also. And according to this study, it's not there. If in the training data, if there is explicitly identified, - described, casualty, - you know, cause-effect descriptions, - the LLM will do a fine job mimicking that, - parroting that, relaying that to you. But if you ask her to do her own casual reasoning, - not gonna work. Which is kind of expected? After all, reasoning is one of the core, key - skills of the human mind. But yeah, some other group of scientists took another shot at - this question from a different angle. They arrived at the same conclusion. No. But they also did this. They took this BERT model. They fine-tuned that specifically for this casual inference skill. Then they used the same vocabulary that is in the training data, - in the queries and prompts, - and they saw quite whooping, almost 95% success rate. That's quite promising but there are all the guardrails in place. And they were testing with limited test data. But still, this tells us that this might be possible. So just like someone at the beach, - a surfer, you know, you are waiting for the waves. But if you don't have your surfboard with you, - it doesn't mean you will get to surf when the waves are there. You need to, you know, build the surfboard, - polish it, make sure it works well. So when the waves are there, you can jump in. There are also things that we can start doing. Intent, content, context. These are the things we can already start working at. And these things are actually... They would be the ingredients for a successful AI model - that can tell us about casual inference - in our decisions and the outputs and outcomes. But even if that doesn't work out, - I think these are good ingredients for human intelligence as well. You want to build your intent. What is it that you are trying to achieve? I think having a hierarchy like this also would make the model and us - work easier with that. You might change layers, - of course, but having a structure to start with is always a good idea. Then the content that you are producing. The software, the release notes. You have to put some clear descriptions that other humans - understand and maybe eventually LLM's too. And then the outside, rather. Because, - we all know things are happening around our product - that affects our product performance. Seasonality, competitors dropping the price, - micro, whatever, economic indexes, all this stuff. But we just say, yeah, it is what it is. We can do better than that. We can try to build - casualty relations. So there's your black box, product decision ops AI model, - which eventually might give us this cause and effect relations - for your past product performance. This is what I think of the most reasonable ingredients - that we would be putting into that soup, the intent, - content, context, fit. What are we trying to achieve? What have we achieved so far? What have we been doing for it? The efforts, - actions, deliveries. It can be marketing campaigns. It can be that feature, publishment. Of course, you need the metadata there also. The dates and platforms, countries rolled out and all that. And then, yeah. Such a fine-tuned model. And this, well, did I build this thing and try it out? No, not yet. Because we are still busy coming up with the, - at least the outside. Talking about the Nest app, we have been pretty good connecting all this, - our intent and the data that we get out of our product, - the usage performance metrics. But there are so many things affecting us - when it comes to the energy business that are not in our control. They impact our product. So we have to build that context, a smart context rather, - that filters the events from the outside world into our decision ops. But once we are there, hopefully, - we will be able to build this kind of a product decision ops AI model. And it will have to be a custom one, at least according to the latest research. You have to fine-tune it specifically for your product and domain. At this point, you cannot just get whatever is the latest best - performing foundation model and start asking about your - product decision performance. So where we are today? We are looking at the usage performance metrics. Engagement, reliability, feedbacks, reviews, - MPS, CSAT, whatever. And then we try to make our decisions, - the roadmap actions according to our best ability as humans. But I believe we are more and more going towards - product decision performance metrics. So we will be deciding our next actions accordingly to our, - we'll still be there, but also our model's best ability. You got to put an ops at the end of something if you want it to get - taken seriously, right? So we have the DevOps, I think it's time for the decision ops. And a way to build that might be this. Am I 100 per cent confident on this one? I don't know. Maybe. That was it from me. Thank you. [outro music]