AI safety nets: DevOps strategies for risk management

Marc (0:00:21): Welcome to DevOps Sauna Season 4, the podcast where technology meets culture and security is the bridge that connects them. Welcome back to the sauna. We have a really interesting episode today. I'm super excited to have a couple of kind of fresh but familiar Efficodians in the house. We're going to talk about AI and DevOps as a safety net. Today we've got Kalle Mäkelä and Henri Terho. Hi guys. 

Kalle (0:01:02): Hello. 

Henri (0:01:02): Hello, hello. How are you doing? 

Marc (0:01:04): Awesome. Judging by the 'moro', I guess you're from Tampere. 

Kalle (0:01:09): Of course. From where else? 

Henri (0:01:11): I guess we both are. 

Marc (0:01:12): The greatest city in the world.

Henri (0:01:14): Yeah. We both are from the greatest city of the world, Tampere, but I think Kalle is more from the center of Tampere, as he likes to call it, from the forest. 

Kalle (0:01:22): Yeah. Geographical center, meaning that I live in the happiest place in the world. 

Marc (0:01:28): Right. And we have a recent immigrant to Tampere, my usual and absolutely amazing cohort, Mr. Darren Richardson. Hello, Darren. 

Darren (0:01:37): Morning, Marc. 

Marc (0:01:38): All right. We're going to talk about AI today and DevOps as a safety net for AI. And this feels really familiar when we've had some other technologies come up. And we're still looking at companies that are struggling with DevOps, and now these same companies are wanting to introduce AI into their organizations. This is huge right now. But the thing that we're going to talk about today is the risk and the safety and security of your company when you start to roll out AI there. So move fast and break things, unless you're working with AI. But guys, what are companies trying to do? 

Henri (0:02:17): I guess this is the thing now that everybody wants to move really fast and break things with AI. But when you end up breaking things with AI, stuff just evolves so quickly or goes into some direction so quickly that the risks there are quite big on it. But then also, if you get it right, you get a huge amount of benefits from that. People are trying to put AI into customer support, content creation, language translation, localization, education, cybersecurity, a lot of the different fields.

Kalle (0:02:44): Yeah. I have heard sentences like, we are all in AI already. This is something that is coming and companies are actually seeing a huge potential for business growth when they adopt these things. And it's kind of scary, but I really emphasize that people start small or start iteratively working on it, because we really need to make this happen. 

Henri (0:03:06): And one of the parts that's really happening now is many companies are trying to build systems where AI is helping people to actually understand computers, be it in customer support or coding. You've probably heard a lot about copilots and all of these other kinds of tools to help you code better. But then also, that's a lot about communication where we have a lot of problems. It might even be that we can kind of go into this mythical man made problem of communication and fix it in AI. 

Kalle (0:03:33): Yeah. The copilot has gained the most, of course, like stage time, but in reality, companies are actually humans communicating to each other and not writing code. And this is where the ultimate gains are actually made. 

Darren (0:03:49): Okay. So when we're talking about AI, we're talking about this kind of large-scale thing, but I think that most people's experiences with AI are with large language models. So let's focus in on those and let's talk about the advantages of large language models, because I think those are what companies are mostly trying to implement at this point. Would you agree with that? 

Kalle (0:04:12): Sure. Let me take first. So first of all, when people use so-called large language models, they actually use retrieval augmented generation systems, meaning that you have the large language model which has been trained with some data, but then you have also actually the company or whatever data like static data within the system. And meaning that we embed something more than your only prompt there. When you ask something, you can ask, okay, what is our company vacation policy? Meaning that then we can go and search it and then the LLM will have that in the context and it will output an answer for you. So that's the basically very, very simple architecture of what they are actually using. 

Marc (0:04:54): So let me replay that back for our listeners. We've talked about this a lot. So essentially, you have an LLM, you've probably gotten this from elsewhere, a pre-trained model. Maybe you have used a service or trained one yourself, but generally speaking, you have an LLM and then what RAG is doing is you are having a database of all the information in your company and then you are creating a prompt based upon querying that database, feeding it a bunch of stuff and then asking the LLM the question. So it could be that you've dumped all of your intranet there. And then when you ask the question on the vacation policy, the first thing we do is we query the database for vacation policy and we get all of that information. We stuff that into the prompt and then we say, what is the vacation policy? 

Henri (0:05:43): Exactly. Not just vacation policies. You can use a lot of different company data for this. Like your intranet, your Slack, your discussion or your code base. And then where it gets interesting is when you start adding in email or CRM into your AI access, because then you got to start thinking about this kind of stuff. Hey, do we actually want to have visibility into the board emails for everybody? And where do these lines go? And what's the risk if we actually show everybody everything? And now we're coming into this. How do we shape up the AI to work in the ways that we do that? There are reasons to do information controlling companies. There's a laissez-faire type of thing where everything is accessible. It's really noble. But in true companies, you have NDAs and other stuff limiting this. So you cannot really do that. 

Kalle (0:06:32): And consultancy companies, they have multiple projects that cannot mix the data, for example. But I would go back a little bit still on the RAG and why the RAG is important. It is exactly these reasons. You cannot have this kind of data governance and control on the data within the LLM itself. So the only way to do these kind of controls is actually implementing an RAG system. So there you can have role-based access control, and then projects have their own data tables, et cetera. Sorry, the table is maybe a wrong word here, but yeah. 

Henri (0:07:09): Yeah. And I think this is the two-way interaction that we have with LLMs is actually that LLMs read through what we do and mirror the architecture that we put into information and all of that. But at the same time, LLMs make humans and computers understand each other a lot better because that understanding is also based on the data that we have, and it also gives visibility to the data that we have already created. So it's totally reflecting what we do.

Marc (0:07:32): You know, there's kind of an interesting idea here that I'm not sure that everybody has thought of, especially if you haven't done a lot of prompt engineering or talking to an LLM. You know, when you query something on Google, for example, you search for something and you get some results and you kind of go through those and you pick the one that seems the most relevant, and then you go with that. When you talk to an LLM, you get something that seems really, really confident, but you still have to go through exactly the same kind of, you know, it's like having a conversation with a human. It's like if I ask Henri a question and he starts to answer, and then he thinks about it a little bit, and then he answers a little bit better, and then we challenge each other, and then we end up with a much better answer together. You have to do the same. 

Henri (0:08:16): Yeah, and talking with an LLM is like talking with an overconfident four-year-old with all the knowledge of the world. He will tell you that this is how it's going to be, even though you know that that's not the truth. So you have to check those LLMs, and the more structure you can build into your systems, the more elder, the more knowledgeable maybe the LLM is on what's the context, and what are we talking about, and not just do the… You probably interacted with four-year-olds. They'll say with conviction everything that they say, and they believe everything that they say. 

Kalle (0:08:44): Yeah, and in the end, actually, this is interesting stuff. So like now you can, when you use a RAG system, you can actually see the citations, what data was used, when the prompt actually was made. So meaning from the actual organizational work context, those need to be gathered when these kind of prompts are made, and the output is stored somewhere, for example, when you use it, for example, generating code. So you need to have there some sort of a metadata layer where you can see and access what kind of citations were basically used. For example, manuals, I think of one use case, if you have a maintenance person, for example, and he or she is doing some maintenance with some machinery, for example, an elevator, let's say it like that. Of course, you need to know what data was used when these decisions were made, or like we order a part, for example, for the machine, then we need to see it. 

Marc (0:09:36): And there's an interesting thing here that comes to mind, which is that confidence in the response. So you might ask for a part number for an elevator, for example, and the RAG system will go off, and it will query the manuals, and it will pump the information through the LLM, and the LLM will very confidently say that it's part number XYZ. But what version of that manual, what model of that elevator, what time frame were any of these kind of things created in? And then once we start to understand those outputs and realize that we're getting it wrong, if we don't have the traceability of exactly how that query worked, that returned the wrong answer, we don't have the ability to improve it. And this is one of the basis, I believe, for the DevOps as a safety net for AI, is that you have to build in this level of traceability from the beginning. Otherwise, you're going to be talking to a very confident four-year-old with all the information in the world. 

Henri (0:10:39): And exactly, this is also the reason that it typically also, you discover that some of your company data is not managed in the way that it can be actually ingested by an AI. So you thought you had a lot of data, you thought you had it nicely organized and all of that, but then when you feed it to the AI and you see where it actually cites data, it might uncover a lot of data issues that you have. This is, again, a good thing that you uncovered them through these tools that you can then improve on them so you can code much faster with AI. 

Kalle (0:11:06): Yeah, but how many systems in a company we actually could have that has some data for the company? I have heard numbers that you have a 10-people company, and you can have 20 different systems for that company. So it's an extremely difficult problem. It's going to be huge in the next couple of years. There will be large products actually built around this. And this is something that really needs to be solved. 

Henri (0:11:35): Yeah, when you think about it, when you start combining data from, as I said, all of these different data sources that you have, you have your ticketing systems, you have feedback systems. We've touched a little bit on board emails. We have that one as well. CRM and then Slack, a lot of places where you can get that information. These were really ticketing manuals, a lot of other places that get you all of the data. But then again, for example, in Finland, there was this case that you have to then again do data governance on your data. There were cases that somebody had put into a CRM that this client is bad. And then through that, AI spread pretty much everywhere. And then you suddenly have a totally different data governance problem. So it's not always that AI is false, that the data is false. You might have had a bad actor putting something that's not supposed to be in a CRM in there. And again, you get a lot of this kind of stuff that you need a safety net for to make sure that it doesn't end up on the client's desk. 

Kalle (0:12:28): Yeah, I explained it one time to a customer that you can think about like, you have four people in the room, and you want to access some system, but you don't have this access. Is it sold actually this way that you get an AI as a fifth person in the room and ask from AI, okay, that I need this data, and then it's okay. It cannot be like that. You know, you really cannot have a joker like this kind of superhero, that will break all of the rules, all of the data security, like within the same room. And you can like freely ask a person that, okay, please tell me everything about this confidential customer, for example. So yeah, so we are there basically now. This is the situation now normally, but that's the starting point. 

Darren (0:13:13): But correct me if I'm wrong. What you're describing is not new. What you're describing is basically an AI bill of materials, like we've been seeing software bill of materials for a decade already. So here we have AI bills of materials where you're just collecting this kind of data of where you're picking the data up from. So we're just seeing an extension of something that's already been put in place, or we're seeing people put into place because a lot of people are still struggling with it. And this is why we have so many supply chain vulnerabilities, but it just feels like it's taking one step further. 

Kalle (0:13:51): It is like infinitely taking steps further in my mind, like because like everybody can now access that, like have that capability. And it's like, it really needs to be handled with automation meaning, because like for the humans, you're going to have human based security. But for like for AI, it is basically infinitely scalable output engine, and meaning that then you need to have an automated based solution. 

Marc (0:14:17): You know, it's easy to think of the temptation to be able to, you know, create a super knowledge base based upon all of the tickets that you've ever received, all of the feedback, all of the requirements, specifications, the documentation, if it exists. And, you know, we've had companies creating these data lakes for, you know, the last, I think, five years or so it's been uptrending. And they're like, well, we don't know what we're going to do with it, but we're dumping everything into the lake. And now you've got a lake full of completely ungoverned stuff. And you need to understand exactly what of those things is being used at any given point so that you can govern it. It's difficult to govern if you can't see. 

Henri (0:15:01): Yeah, this has been the typical, even with like talking about automated testing and all of that, automated testing doesn't bring you the quality, it just gives you visibility to what's actually going on in your organization with testing and quality and code. But then we need the same kind of systems for AI, because when you think about it, exactly the superpower is that you have all of this data, you have it in some kind of managed layering as well. And on top of that, you now have AI systems, which can access all the different stuff you have. It gives you visibility to your data. But then when you think about it, and you can go from there to the fact that every time you ask an AI question, you get the AI bill of materials or the path. Where did you get this? You get a citation for every single piece of fact that you ask someone. That's really the power as well in there. You are not just asking Kalle about, hey, Kalle, what do you think about this stuff? But you are asking Kalle, hey, what do you think about this stuff? You'll get the answer and you'll get the 10 different citations on why it's so. So I think that's also the big power that we'll get from AI. 

Kalle (0:15:57): And also, if we start recording every meeting where we are, we talk about a lot of non-contextual for the meeting, for example. So we transcribe the meetings. But when you use an LLM to actually abstract the data out, what was the most important part from customer point of view? What were actually the action points decided in the meeting? So you filter out a lot of that kind of data that you don't actually want into any knowledge space. That is also one thing. 

Henri (0:16:26): Yeah, but this is also the scary thing that you have a lot of this data, you have a lot of access. But when you do this, you have all your complete data in a data lake and you make a nice customer-facing, customer service LLM. And then you haven't done really proper data governance on that or not really done that kind of thing. For example, there was this case about, I think it was about GM, that somebody managed to prompt engineer their first deployments of a job to basically answer him that, hey, I'm actually giving you a legal contract. GM will sell you a car for like $1. This is a totally new area that what actually happens at that point that somebody has made an AI promise you that the company will sell you a $1 car with a legal contract. So these are the kind of risks that you're also looking at an AI without the proper data governance and proper access controls to that data on what's actually going on in there. And this is the risk that you want to mitigate and solve that you can actually take the AI into use. 

Marc (0:17:21): Yeah, there's an elephant in the room, guys. And we talked about this a little bit in the pregame. And when I listened to the needs here and understand that, okay, we need to have clear governance of our data. We've been trying to do this with GDPR and whatnot for ages already. Companies should have it in order, but many times they may not. When we talk about the transparency and, you know, like I love that Darren brought up this AI bill of materials. We're supposed to have a software bill of materials as well that shows absolutely everything that goes into the software that we're producing. In this case, we extend that to all of the data and citations of the data. So doing AI properly essentially means doing good DevOps practices today as we know them at not only the state of the art, almost the commodity of the art. Do these things properly and then you have the ability to govern. 

Henri (0:18:16): Yeah, AI is just faster IT. 

Kalle (0:18:18): Yeah. And like my personal, I believe, we have been talking with Henri about this a lot. So we have even talked about specification driven development, meaning that actually you can even build the system using the DevOps practices, meaning like Gherkin written pseudocode and specifications. You just input that to the system and then you just put energy in the system and wait until the DevOps pipelines, CI/CD pipelines pass. So really it is like that, like what Henri said. It is very fast IT. 

Marc (0:18:54): So I'm going to replay that one as well. So we talk a lot about acceptance test-driven development or behavioral-driven development or just test-driven development. But essentially what we're talking about is like closing the V model of engineering. Let's get the requirements in terms of testable, actionable items. Let's get those into development value streams very, very early. Let's use that as the guidelines for whether or not things are done. So what we're talking about is once again, let's just get this practice that is well-known, well-documented and well-understood into the front of your AI development pipelines now before you get into trouble. 

Kalle (0:19:40): Exactly. We have even a couple of years ago with our ex-colleague, we invented this or thought of this new V8 model basically. So it is actually that closing up the V. Yeah, yeah, like Marc loves that at least idea. 

Henri (0:19:59): Yeah, yeah, yeah. The V model kind of like exactly doing this, that you have two decoupled loops where you can do the specification on top and having the right really, really for like ultra secure environments or stuff like that. Decoupling that from having that kind of a verification loop on top but letting the bottom loop do all the development in the DevOps way as fast as possible. You always have a release candidate in there but then having the verification loop doing the V8. And of course, this was automotive companies also branding on point. 

Marc (0:20:30): That's so awesome and amazing and obvious at the same time.

Kalle (0:20:34): One more, like DevOps as a safety net, okay? So that's like retroactive, like, you know, like I just like want to reiterate this. So DevOps as a safety net, it's not only reactive. It's actually that proactive what we just liked. So it's not only safety net, it is the driving force of your software engineering. So want to say that one also out loud. 

Henri (0:20:58): Yeah, and exactly in this kind of way that this gives you the visibility. The DevOps gives you a lot of the processes and a lot of the ways that you are actually like codifying your culture into your automation when you do any kind of DevOps or any kind of like automation solutions. That's something that you take the responsibility of certain ways of working from the humans and give it to the DevOps toolkit that you have. And through that, you start building safer and better ways to work that everybody kind of adheres to in your company. And this building up these kind of DevOps safety nets helps you again with IT, and it helps you with faster IT, aka AI. So the same kind of processes apply doing proper data governance, building it into automation, doing nice ways of actually working with all the data, doing verification, doing validation. And if you have this kind of like DevOps-based loop already in your company, you are in a lot better position building on top of AI at that point. 

Marc (0:21:51): Excellent. If I think in terms of value streams, and we talk a lot about operational value stream, that's how you sell your product and service your product and maintain your product. And then from a customer-facing perspective, and then you have your development value stream where you are building the capabilities to be able to sell, service, maintain. So can we split out a little bit how DevOps safety nets, it actually serves the operational value stream as well, doesn't it? 

Kalle (0:22:22): For sure, for sure. Of course, like the first quick wins that we have seen, like the customer chatbots, for example, those are perfect examples of operational AI adoption. I always think it like this. We in DevOps, we, of course, have the automation-driven everything, and we monitor everything. Okay, but now you can imagine, like you have the operational value stream and you don't have system-specific monitoring, like tooling, for example, but you have underneath or like you go very deep and you can then actually talk to your operational value stream wherever it is in the phase from development point of view. So that is something very fundamental and new for me as a capability to actually like create the most effective development processes, because then you can go and ask that, okay, what is our problem in the operational value stream in, for example, in manufacturing part or whatever? So you can really go in in the operational value stream, but then the customer-facing applications, they are like growing like crazy, but there is actually the most difficult thing is the evaluation of the models and evaluation of the system, because it is a totally new science. Really, it's very in the early phases of adoption. 

Henri (0:23:44): And exactly on this kind of level that we have, we talked a little bit about the GM chat bot that we don't have the governance for that model that somebody meant to prompt engineer it. These are the first type of things that's going to happen. We have the same kind of stuff that happened with databases, SQL injections, stuff like that. We are going to have this problem. That's a given. And these are going to be mitigated in the future. And then you're going to see more and more LLMs plugged into different kinds of value streams. As I said, I think many were surprised that the first stuff that was taken by the AI was the marketing department's basic art generation, content generation for blogs, maybe even podcasts. Let's see. But then chat bots in there as well. And then we have the development value stream, which is then full of copilots. Everybody's giving everybody a copilot. But that's also a really small, narrow part of the whole value stream that is there. Then you have all the traceabilities, test automation, requirements, design, a lot of other stuff as well. We're still kind of like touching on small points of it. I think we're going to see a lot of innovation in the development. I think the operational value stream is actually more, on more parts of operational values team we have AI than actually in the development value stream. 

Kalle (0:24:54): Yeah, yeah, exactly. 

Marc (0:24:55): Yeah. And the funny thing about copilot is we don't really need developers to write more code faster. We talk about this a lot that it's the ability to have all of the things that are outside, which are of just writing code efficiently, which are kind of the bigger problems to solve. But it's a bit of a digression. Let's talk about the solution, DevOps as a safety net. Would one of you guys like to describe that? 

Kalle (0:25:26): Henri threw me under the bus. We have talked about this already in this podcast. So DevOps is two words, development and operations. And make any kind of delta, which is coming in from the customer need, safely to the operations and fast as possible. And now when we have this generative AI capability to write those lines there faster, or infinitely scaling the development possibilities, you need to have control. You need to have some sort of break system there, like automated breaking system. Like when you really accelerate with AI, you don't like run out of the road in the end. So safety net, it's like a term for me, means that you can be sure when we have problems, then we can capture those problems. And we don't fail as a business. And I'm just like for the previous point of that, okay, operational actually, operations have adopted get-gen AI tooling more. And that's, for me, it's quite interesting. Like why? Because it's like customer facing, you don't have that safety net. But in development phase, when you change your systems, you have the old industries, like we are already like 30 years in into agile development, et cetera. And test automation and 70 years in the validation. So it's kind of like, for me, it would be more wisely to actually start from the development values because there is the safety net. 

Marc (0:27:00): So I'll break it down into a few different parts here. So knowing all the data that is going in, this AI bill of materials, I think that's absolutely key. That's really freaking cool. Then basic DevOps, even before that, when we had build management and things like this, always being able to reproduce building a model and knowing all the things that went into it, the reproducibility or the AI bill of materials. I like also, Kalle or Henri, I forget which one of you made the point, but the references in the future. I think that knowing that AI has been a part of the chatbot that you're having a conversation with, or maybe even the advertising campaign that is hitting you or a photo or a deep fake video or something that you see. Knowing that AI was involved is something that I think it might end up going the way of California with the Prop 25 label known to the state of California to cause cancer, birth defects, and other reproductive harm. So AI was used in creating this and here's the references, the data that we use. So the transparency in how the models were created. And then we also see the traceability that's necessary, like Henri talked a lot about the operational value stream, what were the prompts that were used in order to get the guy a binding contract for a $1 truck from General Motors. When somebody is gaming the system, we not only want to be able to trace that, but it'd be nice to be able to monitor it and say, hey guys, something weird is going on. People are starting to find a hole in our chatbot and it's leaking our internal intranet policies or something like that, or even worse, maybe customer data. But it's the traceability so that you're not a blind person driving a Lamborghini. 

Henri (0:28:50): Yeah, and exactly kind of this. And again, I think one of the problems is that people are trying to scram all of the AI into one AI. And exactly the easiest way to mitigate leaking customer data is having different chatbots facing customers and different chatbots for internal use and different chatbots for different contexts of data. So you are actually avoiding that leakage that you have and having this kind of a tiered system or something like that that we have in there is something that we need. And again, on this format, talking about just one AI model, I don't think it's something that we do now, but we are going to go into a future where we can dynamically build new models, we can create or rebuild, or we have to be, in the same way as DevOps, we have to be able to rebuild all the AI models based on our data and base models. And then not custom make them, but have them dynamically made for any of the contexts or data contexts. And this enables freely, again, scalability in your AI without the fear of leaking data. And then correct benchmarking of AI is actually quite bad. If you actually dive into many of the data sets, HelloSwag, MMLU, they are really bad at actually validating AI because they are typically contextual, very cultural based and not too well done. They forced AI into answering multiple choice questions as A, B, or C, and you don't actually get natural text output. So this is something that gets a lot of, again, improvements, going to get a lot of improvements in the future as well, is the AI development pipeline, which is going to become something like DevOps AI, AI DevOps to have visibility into multiple tiers of this system as well to get that context. We are still now starting on it, but very picking up speed. 

Kalle (0:30:36): Yeah, and I want to reiterate also that point, what Henri just said. So that is the whole, that is going to be the new field of research, new field of applied sciences when you are developing these AI models. But as I see it, DevOps as a safety net, it's for the software. And actually, what is AI? It's only software. Don't think about it as this big unknown kind of thing. Just think of it as a software asset that does things and it creates actually randomness into your system. And my point of the DevOps safety net is not for the AI models themselves. It is actually when you integrate those models into your procedural code systems and making sure that the end-to-end is handled correctly. And so even from DevOps point of view, you don't even need to know almost about AI. It's running in the system when you have a very well-made DevOps safety net. 

Marc (0:31:34): I mean, it's just running statistics plus a randomizer on not only the data that you have, but all the data the model was trained on. Plus the randomizer, because if it wasn't for the randomizer, certain words would always lead to certain other words. So that's one of the joys. I was at GitHub Galaxy last week as of our time of recording, and there was some really amazing looking stuff coming on co-pilot in the future. But then there was one demo and we've done a lot of these AI demos and you never know what you're going to get. And I think part of the joy of demoing AI is showing how you work with the prompts in order to get the response. And that's like some of the coolest stuff to see. It's like, you know, if you try to just reproduce something, that's not what it's about. What it's about is showing how to get the best result when you don't necessarily know what you're going to get, but you have the power to get amazing things. 

Kalle (0:32:29): Yeah, tool is still a tool and you need to learn how to use it. And I see this adoption of the LLM systems and you will have them. You will have tens or even hundreds of LLM systems within your company. And it's not like this, like specific AI houses cannot actually build these systems so the LLM services themselves so that you can safely adopt them. It is a you problem. They will not fix it for you. When you take these ones into account, exactly like Marc, you had this GitHub example, it's when you apply it into your company operation and development value stream, those are the points where the good practices, you know, and like intelligent decisions, let's say it like that. Don't run around without your head. Just like, think about it, adopt it iteratively, build it, audit it, and do it like how good, like any kind of good software practices are anyway done. 

Marc (0:33:26): All right. Let's pull a summary together, guys. This has been fantastic. So what is the advantage of AI available today for companies? 

Henri (0:33:36): Here we go. What's better, faster, stronger? What's the Daft Punk song about that? But pretty much as I said, I think the main things come from the communication. All code is communication between humans. All applications are pretty much communication and ways of automating stuff that we could do with humans and all of us. What AI really enables us to do is talk to computers with natural language and really augment that discussion as well and get the computers much better into the discussion loop. Help you with a lot of the communications as well between humans. Communication, I think, is one of the biggest key takes here. 

Marc (0:34:11): All right. So one of the most powerful ways to use LLMs today is RAG or Retrieval Automated Generation. Would one of you like to say what is RAG for our listeners? 

Kalle (0:34:21): It's Retrieval Augmented Generation. Sorry, Marc. It is a combination of two things. It is the data which is basically embedded to a vector database and then you have that LLM model itself. And you use those two to basically talk to your data. It's simply put like that. 

Henri (0:34:42): Yeah, so the data is the database, yes. And the LLM is the same as a guy accessing your database and going through what you want from there. So LLM is just doing the interpretation and output of your database and what you have put in there as a company.

Kalle (0:35:01): Why actually, Henri, we use vector databases with LLMs? 

Henri (0:35:05): Yeah, I said everything's just math. Everything's just a math vector in that database. And you have to take all of your data, put it into a vector database to get the best results with LLMs. So you need to vectorize it. But yeah, that then creates the system that you have. 

Kalle (0:35:20): Yeah, and it's like technologically compliant with the LLM way that they also handle the data. 

Marc (0:35:25): All right, what does this enable companies to do today? 

Henri (0:35:29): Today, as I said, focus is on communication. So it helps with external communication, external being these marketing chatbots. Customer service is going to be big in this. And there's been a lot of companies doing it even before LLMs and getting good results. But now with LLMs, that's going to get supercharged. Communication inside companies, digging out data, hey, how do we do this? Even just creating new stuff like PowerPoint presentation and all of this, commercially available LLMs is also something. Anything that you need to create data quickly, you need to create something quickly. AI is going to go in there and help you do it. But it's not going to take your job. It's going to take your tasks. And I think that's the most powerful thing about AI. It's going to automate with all of these boring tasks like filling in your hourly sheets, which I have to do today as time of recording. But stuff like this can be automated and really help you. 

Kalle (0:36:19): And for example, everybody is asking GitHub a return of investment. But actually what it brings is developer experience. It's much more easier to develop code with that. And it creates, although it can create some bugs or not working code, but anyway, you have always basically a system what you can ask, okay, let's start something. And let me run actually through one use case where this RAG system can be used is that you have a customer that has a problem. Or he or she calls to your customer service. You transcribe the phone call. You ask from the RAG system that, okay, hey, who has had this problem before? Like what are the 10 most important feedbacks now within the one week? Okay, now you basically have created some sort of textual context about the calls. Then you put it to your development RAG system, meaning that you will get specification changes against your existing specifications, for example. Then you ask it that, okay, let's create API documentations here. Like what API needs to change? How you would change it? Okay, now you have that. Then you go to a co-pilot. Then you ask, create a code for me or create actually first API tests and then create me a unit test. And because copilot actually works better when it knows the test automation context and what kind of tests you have, it creates better code for it. And then you iterate on that. And then in the end, you will create a pull request. You will create release notes. And then you put it in the master or main, and then you push it to production. And you could even create phone calls automatically to the customer that actually called you. And you will call them and say that, okay, now it's okay. And that's like the time to market for that can be hours. And currently it can be even months. And if you do it safe way, you can imagine the benefits. 

Henri (0:38:18): And to summarize Kalle, a safe way here is exactly what we've been talking about the DevOps safety net, that you have a lot of these different things that you can put into the whole pipeline. It's about your culture. It's about how you proactively develop your whole knowledge, corporate culture. You develop all the different phases and have automation and checks in the right place. Also, this system gives you all of the transparency that we've been already talking about AI. Because you have all the checks and balances in place inside of your company. And you can plug in AI at that and really help you manage the risk and quality of your AI systems. Exactly the same way as you do software artifacts with DevOps as well. This is no different than before. AI is just faster IT. AI just lowers all of your lead times. 

Marc (0:38:59): Amazing. Thanks guys. This has been really fantastic. It's like I'm in the middle of this stuff all the time, but I'm still learning every day. And I think our listeners are gonna get a tremendous amount. Thank you, Henri and Kalle. 

Henri (0:39:15): Thank you. 

Kalle (0:39:15): Thank you for having us.

Marc (0:39:18): All right. And thank you, Darren. 

Darren (0:39:20): Thanks, Marc. Pleasure as always. 

Marc (0:39:22): Okay. This has been another episode of the DevOps Sauna. Thanks. And we'll see you back in the sauna next time. We'll now give our guest an opportunity to introduce himself and tell you a little bit about who we are. 

Henri (0:39:38): My name is Henri Terho. And I have background in software development. I'm actually a biologist by training. And this is why AI is a super interesting field now that I've done a lot of this simulation for neurons and stuff like that. And now I'm just hoping that AI finishes my PhD of software engineering before I do it. 

Kalle  (0:39:56): Hey, my name is Kalle Mäkelä. 20 years of IT background, being all over the place in the software development world. And basically now with AI, my preaching of DevOps is gonna come to a nice ending and very eager to talk about AI. 

Marc (0:40:11): Hi, I'm Marc Dillon, lead consultant at Eficode in the advisory and coaching team. And I specialize in enterprise transformations. 

Darren (0:40:19): Hey, I'm Darren Richardson, security architect at Eficode. And I work to ensure the security of our managed services offerings. 

Marc (0:40:26): If you like what you hear, please like, rate, and subscribe on your favorite podcast platform. It means the world to us.