Patrick Debois and AI in SDLC

[Patrick] (0:06 - 0:10) Now that's the biggest risk, to trust something that we know is unreliable.

[Darren] (0:17 - 0:25) Welcome to the DevOps Sauna, the podcast where we deep dive into the world of DevOps, platform engineering, security, and more as we explore the future of development.

[Pinja] (0:25 - 0:35) Join us as we dive into the heart of DevOps, one story at a time. Whether you're a seasoned practitioner or only starting your DevOps journey, we're happy to welcome you into the DevOps Sauna.

[Darren] (0:48 - 1:04) Welcome back to the DevOps Sauna. We're very excited to have a special guest with us today joining us after his appearance at the Future of Software in London, we have Patrick Debois. Welcome, Patrick.

Yeah, it's always a pleasure to be here. So, and of course, we have Pinja, as always.

[Pinja] (1:05 - 1:25) Hey, and Patrick, welcome to the DevOps Sauna. We're really excited about this. And I'm really sure that many of our listeners notice and recognize the name, but let's just do a very quick introduction of you.

And we couldn't give a, let's say, grander introduction, because you are being called the father of DevOps. Isn't that so?

[Patrick] (1:25 - 1:28) Yeah, I think that's what people say. It feels weird.

[Darren] (1:30 - 1:36) I actually wanted to ask, does that title bother you? Like, everyone's calling it to you. I don't think anyone's asked if you like it.

[Patrick] (1:36 - 1:49) It's a compliment, right? It has moved into being the grandfather, and I don't know what will be the next phase. It just means I'm getting old.

I was there when something happened. So, definitely there.

[Darren] (1:49 - 2:06) Okay. So, in London, you talked about the software development lifecycle and AI, and it was kind of interesting. And I think there was a great response from the crowd.

You were, from what I saw, you were basically surrounded by people most of the day asking you questions about it. So.

[Patrick] (2:06 - 2:30) Yeah, it's definitely a hot topic. A lot of people are trying to figure it out. And sometimes it definitely reminds me of the early days of DevOps, where it was like, what's this crazy thing that we're doing?

How is this going to evolve? So, definitely some similarities. People with a lot of questions, which is great, right?

Because that's how we can kind of improve our craft is start answering some of those questions and see where it evolves to. So, it's a great moment in time, again, to be alive.

[Pinja] (2:31 - 2:52) It's really nice having those conversations going on, for example, as you say. It's not just you answering questions, but people are actually engaging in those discussions. And one of our other speakers, also Kelsey Hightower, who's really been there since the cloud native movement started.

So, it was you two also having the conversation along the day. But it's really nice to see that people are engaging and that's how the community grows.

[Patrick] (2:53 - 3:00) Yeah. And it was all about learning in the craft and improving the craft. And yeah, that's part of the journey indeed, as it is.

[Darren] (3:00 - 3:24) It's exciting to see that it's, as you say, like the starting days of DevOps, it was, it's very much that AI is both something new and nothing new in that it's just, it's the newest tool and people are excited. People have questions, people are struggling for understanding and having this open communication is, again, it just becomes critical. Nothing has changed in that way, I think.

[Patrick] (3:24 - 4:21) Well, I wouldn't say nothing has changed, but it's like I made a comparison when we were moving to the cloud, we were copying our old ways of configuring machines. It's like, okay, let's go for a VLAN and configure this in the cloud. And then what?

There's no VLANs? How do I do this? So, okay, you know, that kind of became another thing.

But that was almost like mimicking what we're already doing. But then due to the fact that we were in the cloud and we could do parallel things, we started doing microservices. It's not that we couldn't do this before, but because of the new environment and the new tools, we started adopting new practices.

And that's kind of like what I say here as well within AI. It's not that, you know, unlikely that, well, that's not true in AI. Technology still moves so fast, but in general, a lot of the technologies are there and we just like figuring out how to use it differently and being more efficient in the new ways of working.

[Darren] (4:21 - 4:36) Okay. So let's take a step towards this. You were talking about this idea of working towards full autonomy in software development life cycles.

And you had this concept of AI at the moment being kind of slapped on and what we need is to drag it towards the center.

[Patrick] (4:36 - 6:58) Yeah. Using a new technology in the way that you had your past technology, that's slapping it on. And often when people start with the AI journey, they're thinking about, oh, let's do a chat because, you know, kind of everybody's doing a chat and that's kind of like probably the hell world in kind of AI land to do things.

But as we added more complexity to that thing and we added concepts like RAG and so on, these tools kind of got better. Now the problem usually is if you just tag something in your product that is, hey, you can use a chat now in my documents. Great.

You're like, you know, everybody can do that right now. It's not going to be the differentiator anymore, but maybe you'll have to rethink how your data feeds into that chat or how you're going to curate your data or kind of where you store all the knowledge of people asking questions. So that's kind of like rewiring things in your product to become more efficient.

Now, for example, people have, you know, a lot of data lakes that have a lot of data, what people like, what they don't like. But one of the hard things right now is to get data from the data lake in real time to actually be used as knowledge while in the chat. So again, those are pieces that we'll have to kind of rewire to make them more efficient.

And it all started maybe with that simple chat, but that's why your architecture and your product as well need to evolve. So on the product side, for example, we've been saying, well, customer first, which is great. You know, UX, UI, you know, make it user friendly, make human friendly.

You know, 10 or 15 years ago, we said, well, there's another persona. It's the SEO that we have to do. So the bot's coming to our side and we need to optimize those pieces as well.

And now what you see is, well, that's all good if I can read HTML and that stuff. But what if an agent comes across that site and needs to consume the site and all the knowledge of that? You know, you would say, oh, we're going to get into a genetic experience or how do we actually make the site more kind of useful for that new persona?

So that's why rewiring, rethinking, going deeper and not just slapping a chat bot onto your site is where the real value is going to be in the differentiating factor, like with your business, more succeeding in this AI world than kind of going for the hello world example.

[Darren] (6:58 - 7:17) This data lake concept is kind of fascinating. I hadn't really considered it because when we talk about retrieving more augmented generation, we're typically talking about document bases. But yeah, trying to pull database data into those in real time, that's actually kind of a fascinating problem.

I don't know how people are solving that currently.

[Patrick] (7:17 - 9:55) I think it's somewhere within an intermediate cache. So either it comes through the analytics and kind of the feedback flow into the data lake, and then they expose this almost like they would do with reporting, but they expose like a database or with this. But we already know that that flow is going to be too slow, even if you get it like, you know, down to a week, a day, an hour.

The user wants their history from the previous chat or the previous thing they typed. So what you start seeing is that there's kind of a merging of that data lake, intermediate cache and a knowledge base store or kind of more of a memory store that you see like hybrid things coming together in that world. So it's like there's definitely a gap that needs closing.

And like a lot of companies are working towards making that more closer to real time, which is a challenge because of all the data. But we do know that if you want to make it personalized, you want to have the agents have the latest information, you need to kind of solve those problems. And it's almost like your continuous deployment of data that you want to do as fast as possible to get that to close to real time there as well.

Because I think gathering knowledge and information about your customers is probably almost like the ultimate goal. If everything can be automated, everything can be done by the AI, like that curation of that knowledge is going to be your ultimate job once we reach that kind of level in that way. Now, we're not there yet.

And you made a kind of a reference to maybe the different levels of automation. In DevOps, we had the same thing. Oh, you can have a CICD and then oh, we can kind of do a deployment, which is great.

But then we wanted to have more continuous deployments. And then it was more about continuous delivery, feedback from the end user. So kind of that automation cycle and that maturity level is definitely something that we want to go with AI as well compared to DevOps automation.

Now, we learned that what you really want is if you want to go faster, you need better brakes. And kind of that capturing of the testing, that's why kind of continuous integration was so important to have that test framework. And we're still working out a little bit of the kinks with how that looks with AI, because it can produce a lot more stuff.

But that brings kind of the burden on us to do more reviewing. And that's not scalable either, right? It's like having somebody generate loads and loads of codes, whether that's a junior or senior even, you still have to kind of do that review thing as well.

And it puts the pressure on that process currently with the AI code generation there as well.

[Pinja] (9:55 - 10:29) And I've been thinking about this quite a lot from the cognitive load perspective, because the word of developer experience has been on the rise, rightfully so, in the past couple of years, even more now. So what do we think of that? And what is your stand on that?

How can we balance between this where the rise of the importance of maybe, let's say, code reviews and actually putting in the effort of trying to find different places and different threads is on the rise at the moment. So how does that play with the developer experience and the cognitive load in your mind?

[Patrick] (10:29 - 14:41) Yeah, it's a great question. And so a lot of the articles you'll see are about faster code generation, but the review process, that is the bottleneck for humans right now. So I usually compare this to if you have a lot of PRs from your peers that send you, like what can you do to improve your life on that?

Now, AI has helped us with summarizing some of that work. So it explains what has been changed. Now, we probably still have to do some of the code reviewing in those pieces.

Now, more and more, those code review tools actually also check things, like they do linting of the stuff, they make sure it's according to certain guidelines. So they do kind of almost like the first steps of automation, like the first barrier in our way to production, if they can help us. Well, this failed.

Maybe I can just do a fix. Oh, this is not a secure thing. Let me kind of fix that.

So we're seeing the first steps to kind of improve that part as well. Now, that's kind of almost like mimicking our tasks and kind of like dealing with it. Then there's the synthesis part, which is like, this has changed.

What do you want to do with this? I saw an interesting example in the early Cursor Composer, where the usual, there's a couple of ways you review code. Here's the diff, right?

Compare the two things. What do you think is useful? Now, that's great if it's one file, but if it's 15 files, you're like, ah, really annoying.

So imagine it would give comments like, this has really changed. This is the thing that does. So it takes that diff almost and makes it a better version of that diff, right?

So more annotated, more kind of readable, human consumable, that helps. Some might say, put it into a chat and the chat explains everything. I personally find that too verbose.

I don't have the time to listen to a chat system that does it. I've seen another system within Jules and Claude. It actually creates a podcast using Notebook LLM and it just tells you what went on and that gives you like situational awareness.

So we're kind of seeing those smaller hints on kind of improving things and synthesizing stuff for us. Now, the third thing that you could do is imagine you have to do a review of some kind of Terraform, right? I could have a look at the code, which is great, but you've imagined this would be a diagram or a visual representation of your infrastructure.

And it says like, here's the diagram. This is what happened. This is the impact.

So it creates an alternate kind of visualization or way of looking at the product. Now, this is one example, but imagine you're doing a change and there's a business impact. It shows you a diagram.

It helps you kind of go through that as well. So it doesn't have to be text. There could be different representations.

There's a concept that is called the moldable development environment, where actually your IDE adapts itself to what the task is that you're reviewing. So that image was maybe one kind of very simple thing, but imagine that your reviewing system is optimized for whatever thing that you're doing. And so that is an involvement from your IDE, which was used to kind of you typing the text to become a system that does a lot more reviewing and adapts itself to the environment.

And we can extend this also in production, where we could say the moldable development environment was for reviewing what the code has done. And we say, go to production. But imagine there's an incident on this.

I haven't written the code. I want situational awareness. So now I have my logs and my observability and I dive in and kind of look at all that stuff.

But imagine that would also adapt itself to a moldable exception environment that shows me in a different way for me to understand the problem at hand during an incident. So kind of that is some of the techniques and kind of hints that I see people experiment with to kind of reduce some of the cognitive load, either assisted or different representations or kind of different tooling that is being created on the spot for you to do certain of those reviews.

[Darren] (14:42 - 14:49) So it's basically all about applying context, really, just to give rapid context to people so that they don't have to hunt it themselves.

[Patrick] (14:49 - 15:02) Yeah. And now with the automation part that we kind of do for the code gen, we can also start automating that as well. So we get better results and help us to do that review in that case.

[Pinja] (15:02 - 15:30) There have been discussions on how the job of developers is changing. And I guess this is also something that is shifting a little bit because we have the code assistance. We are more moving towards actually trying to read the context as you just said.

Do you see the role of product management in this in setting the direction so that not let's say we take a company of 200 developers and not everybody is creating their own context. So where does the context come from to set the direction?

[Patrick] (15:31 - 18:43) So I wrote an article about the four patterns of AI native development. And what we've the first one we kind of already touched was you're going to go from kind of a producer of your own code and the code to becoming in the manager of reviewing all the code and in production kind of also dealing with all the problems that it creates. You become the manager now to express what the coding agent should be building, that's typically the role of the product manager.

They define kind of the specifications, they define kind of the tasks and they break this down. So what you actually see as the second kind of pattern is that we're moving from focusing on that implementation like oh this is the exact call that we need to do to where it's hey let me type a prompt and then you as a tool create that code for me. And instead of then doing this more prompting and keeping typing let's write this to a markdown file and kind of give that to you so that's a longer prompt.

And then if I take this further I could say these are all the features that I want to build. These are all the technical requirements that you need to build. This is the security requirements that you have to build.

So you start the build up of more specifications. In some contexts this is also called the product requirements documents and so you start like writing those things and then you feed that into the agents who break that down into tasks and kind of then start implementing smaller bits of the features much like a product manager will do. So your role is shifting it from a developer who says like okay this is the exact code to I just tell it how I want it. My directory structure needs to be like this.

I like to use these three tools. So you don't have to retype this over and over again in your prompts. It just keeps that as a specification as the context and kind of improves the generation more consistently.

And you can put a lot of things in there whether that's how you like your naming your commenting. I want to have everything that's being generated needs to have tests. My tests need to be running like this.

So we have so much domain knowledge in our head both technically and then when we work a long time with a product like what the product actually does that this is a lot of useful context that we should kind of start writing more as requirements and then have the coding agents kind of implement that. And we're getting more and more closer to that and you can see progress being made every day. Now you might not see this if you're in github copilot right now.

It's not natively that well supported. Yes you can say at spec.md and it gets into the prompt but more and more there's tools that say here's a spec. I'm going to break this down for you.

These are like five subspecs and then kind of like oh this is the next task or the task needs these specs to kind of run and kind of go in that world. So it is constantly evolving but it's definitely a trend and now all of a sudden we become product owners that have to know exactly and specify exactly what needs to be done which is a little bit new. Although I'm pretty sure if you're a senior developer you have opinions on what the product should do.

[Darren] (18:43 - 19:14) So it's kind of changed the perspective in a way that maybe actually might not be that helpful but it changes the perspective to needing a very clear vision right from the start. I just as you were talking I was wondering how that's then going to impact like junior coders newcomers to companies who perhaps don't have visibility on anything. And I think interestingly more than anything it's going to increase the requirement for clear and concise documentation because the vision needs to happen as soon as possible.

[Patrick] (19:14 - 21:25) Yeah well there's two questions that you ask like what is it for kind of more juniors right so they don't know the domain which is fair they don't know coding yet as well right so that's great that's true. Now I often hear this remark from senior people that they worry about the junior people. The juniors consistently tell me we can become a senior way faster thanks to AI.

So think of this as there's a you're new even if you're senior you're new on a job and you have a code base pretty complex and you come in and it's like okay well I can dig into the code base and if I'm lucky it's readable there's a lot of documentation if not you're on your own or I have to bug my colleagues. Now thanks to AI you can start asking questions about your code base like how will we do our authentication here like what's kind of the standard way find me the classes that do this so there's an exploratory thing how you see products like shoulder.dev take this into well let me just generate lessons from your code base right so new developer comes in and says like okay just tell me here take this like 15 lessons and learn better to understand your code base.

So this is definitely kind of helping that level up now the domain that's something different they'll have to learn that's maybe harder but again we can apply lessons there as well but that automation is useful and the great way is that if you start implementing that into oh let's take that information and knowledge put that actually into the readme have that readme in sync keep the documentation in sync so it's a lot more used in because if you feed better documentation your AI will be more powerful so there's a flywheel happening on kind of documenting the way you do things and the way around this. Now that is indeed a third thing like I can say here's my technical requirements here's my business requirements and usually there's a third piece this is how we do things around here so that that kind of are three pieces of kind of knowledge that you can feed into the documentation that actually help the developers on that piece.

[Darren] (21:25 - 21:44) So do you think there's some danger in that we've heard the phrase before I believe that hallucination is not a bug it's a feature so having maybe junior coders asking questions about their code base and being responded to by AI without human verification might lead them down the wrong path.

[Patrick] (21:44 - 24:19) I think that is a remark that makes sense in a way that I don't want to have somebody just randomly put stuff in my code base that gets kind of deployed towards production. Now we had that same discussion in CICD and I'm just going to push it is the test run are we going to deploy this to end users yes or no that was even before AI that was somebody copying things from Stack Overflow kind of moving things in so what we did was actually we moved to the PR and the review process to kind of like you know stop that we added tests and a test suite to kind of make sure that we had like that safety level that when a junior came in we are able to detect what is happening and the impact is there you know there were stories about like juniors get deploy access you know from day one only if you have a test harness don't do that if you don't that so it's the funny thing is it's actually driving us towards the good practices of having more tests and dealing with those pieces of work. Now I'm gonna say something similar there was always that discussion is as much as you automate in DevOps right it's great now I can delete all instances at the same time if I've mixed up right so it's the same thing like safety and thinking about like guardrails of kind of making sure that like one mistake does not have a full impact on all the others and that's the same thing now with juniors but if you give them guidelines and I think that also helps reduce hallucinations because it's grounded in documentation it's not something that was you know randomly trained so kind of that grounding helps is it perfect no I've seen kind of when you ask it to build something the coding agents build a simulation of what you're about to be building so yes there's weird stuff but it's definitely improving and I think the error rate of the hallucinations is going down now that's the biggest risk to trust something that we know is unreliable and that is kind of the paradox of the AI is the more we're going to automate that like if you don't find a good way to automate the review and the testing and kind of that I'm not sure if it can go faster now it all has to do also with the risk appetite you're not going to have your junior touch your database system it might be a feature somewhere in the feature branch after a kind of release flag so this is very similar right so you you kind of have to think about that as well how do you what's the risk do I need to review is it okay to accept it and kind of that level is important to look at like when you deploy those tools as well.

[Pinja] (24:19 - 24:36) And there is the the notion that we need to know I guess more now and also in the future what does good look like so that we can actually see when we get the recommendations and suggestions that is this okay is this something that we're striving for is not just that we have the vision but what is good quality and what is good and in our standards.

[Patrick] (24:36 - 26:05) Yeah that definitely stays important I think the the challenge is will we keep up with the volume if there's a hundred things we're gonna get tired of reviewing everything so that is not a cognitive load but there's like a review fatigue that you just say yes there has been a study that the acceptance rates of code being suggested by code completion is higher in the weekend I wonder what that is right so it is kind of just say well you know looks good to me let's go but that is the challenge of of this more explosion but you know I learned that the answer is usually if there's a problem with AI you know put more AI to it and maybe that's kind of where we are at but understanding good looks like it's still a valid skill learn the basics learning all the components learn why it's doing this ask those questions as well but you can't like fight stupid if you just go and go to production and always say yes yes yes and you put it in YOLO mode you know hey you know you can be lucky but it's you know it's very similar to well I'm a year into production and I have zero tests okay you know but then still probably you have a manual review if you do it in smaller chunks that you say okay you know that's good that's not good but there's definitely the risk appetite is different for different people and different companies to say well let's just ship it right let's see what happens different phases of your company as well.

[Darren] (26:05 - 26:54) Let's for a moment focus on something you mentioned or we've kind of been skirting around here and this is the human layer because over the last few days I don't know if people have been catching it on LinkedIn something interesting happened that members of Apple put out a paper stating basically saying that AI was not capable of critical thinking that's the TLDR of it and then there was an immediate response of an AI generated rebuttal to that where it critically thought about all the flaws in this paper so I guess what I want to ask is where do you see the human layer of judgment ending up in this process because right now we're talking about the human element being important but how do we think that will translate as time goes on and AI gets more advanced.

[Patrick] (26:54 - 29:07) Yeah I usually bring up the analogy with self-driving cars there's different levels of autonomy in a vehicle okay you know assist lane kind of changing or something like that or completely autonomous driving similar in DevOps we learned that we can automate for example the application deploy but only few of them do deploy of database update schemas right so it's all about kind of like taking that risk as I mentioned before like we know this is a known kind of restricted case and that's where we're going to apply maybe like more automation which is safe which we got guardrails around and that's good and other pieces will kind of have to do keep doing manual things so I think because the technology of Gen AI is inherently flawed and we cannot express the perfect thing that will always have a certain level of human interaction now a lot of people believe because we're throwing so much money within the AI world that you know much like the autonomous vehicles that we're going to get there right but it's definitely not the state where we are at and the risk we are doing and even if the models get a lot better there will always be the risk that it's not okay so I think that that urges us much to simmer in the DevOps world where you say it is CICD we're building all the stuff but then we started thinking about resilience engineering which made sure that whatever was the impact of an error actually that the system was changed or the architecture would be able to deal with it if you couldn't do that you probably you know can we say that's a case where I can actually deal with it without human intervention because the system is has a fail safe to be built so that's kind of the journey that we're on how much automation how much not but I think the technology for me right now is not something that we can trust like 100% without a human to do it even if it gets better you know up to 99% is always going to be that one thing but the question is what's your risk appetite to that?

[Darren] (29:07 - 29:20) I think in light of the recent Tesla developments and the model why just speeding past school buses I think your autonomous vehicle analogy kind of nails him.

[Pinja] (29:20 - 29:57) So I guess there is the conversation of which are the tasks in each job that we can complement which we can substitute with AI and is really critical I've had this conversation with many organizations like do you have do you have an AI plan do you have an AI risk appetite for example and it started with having no AI policy to having an AI policy that says do not use AI tools to now having an AI policy that says it's okay but I guess the AI risk appetites are still it's something that not the the great maturity doesn't have them perhaps yet so much.

[Patrick] (29:57 - 30:54) Yeah and it was very similar to how we separated things that needed to be audited by socks right so yes in this contained way this can be approved this is a process this is something we do and kind of deal with those pieces as well now it is getting increasingly interesting in the way that I think when I gave the talk in March it was all about okay the pipeline we do the coding the completion agents were beginning to become part of this and the agents kind of like gave a result I think the agents now go into a loop which is interesting the new part is actually why go into one loop if I can do seven loops of the same thing and then select the best answer so that's another way of dealing with this like fan out find multiple solutions and then I have a critique that actually selects the best so I know even more burden on us reviewing but kind of that is another way to kind of have come up with multiple solutions and then say oh well that's actually a lot better than the one shot that I got or something perfect.

[Darren] (30:54 - 31:10) I think we're starting to run out of time so I think we should leave it there but thank you Patrick for joining us it was great to have you here and extremely interesting your ideas.

[Patrick] (31:10 - 31:40) Yeah thanks for having me uh if you're interested in following me you can go to ai native dev.io where I write blog posts I also curate for people interested in tools they can go to landscape.ai native.io I have now 350 tools in the space it's like crazy to keep track of them all but you know if you want to try them if you want to play it within a certain domain testing development product requirements there's tools now for everything much like the CNCF landscape is just exploding and we'd love to get your feedback on that.

[Pinja] (31:40 - 31:44) That's amazing. Thank you so much Patrick for joining us. It was an absolute pleasure having you here.

[Darren] (31:44 - 31:49) And of course thank you Pinya for joining me once again.

[Pinja] (31:49 - 32:04) Thank you and to you thank you as well and we hope you join us next time in the sauna. We'll now give our guest a chance to introduce himself and tell you a little bit about who we are.

[Patrick] (32:04 - 32:16) So I'm Patrick Debois known as the father of DevOps but I got bored of DevOps and now I'm working around the intersection of Gen AI and the whole software delivery lifecycle as a curator in the AI native dev community.

[Darren] (32:16 - 32:26) I'm Darren Richardson, security consultant at Eficode.

[Pinja] (32:26 - 33:04) I'm Pinja Kujola. I specialize in agile and portfolio management topics at Eficode.

[Darren] (33:04 - 33:15)