We sometimes see strong differing viewpoints about the means by which our customers should help themselves, and these differences extend to our customer base. To express these alternative points of view, we decided to host a debate between a few esteemed specialists. We discussed what is keeping software enterprises from scaling up, and touched upon some possible solutions.
Welcome to DevOps Sauna. My name is Lauri and I am the Chief Marketing Officer of Eficode. We at Eficode value results. As we work with our customers, we see that sharing our experience and expertise spins them forward. However, it wouldn't mean that there was only one truth that helped everyone. We sometimes see strong different viewpoints about the means by which our customers should help themselves and these differences extend to our customer base.
To express these alternative points of views, we decided to host a debate between a few esteemed specialists. We discussed what is keeping software enterprises from scaling up and touched on some points and possible solutions. If you get excited about things like Agile, continuous everything, Devops, value stream mapping or theory of constraints, I'm sure you will enjoy this conversation.
The discussion is attended by Eficodians Kalle Mäkelä and Szilard Szell from Finland and Henrik Hoegh from Denmark. Kalle delivers us the opening speech and after that, it's truly a free flow, so let's tune in
Let me start by saying that why I'm here actually recording this with you guys. My main professional reason for getting up actually in the morning is to see our customers get a-ha and eureka moments when they learn new and better ways of working. After my work is done in the customer side, any CXO, middle management, manager person and a knowledge worker understands the best way of them to collaborate on that common goal and satisfying their end customer with the best products and services.
In my point of view, we are here talking about the reasons why my daily work sometimes seems to be so incredibly hard. Even if we seem to have all the necessary frameworks, catch the magic fix in our pockets, to get our customers to be the best that they can be. Or even better. To state out loud officially, we at Eficode do not subscribe to just one methodology or principle. We recognize that there is Agile, of course in the basement as the mindset. Then there is safe and then there is less. We also acknowledge that there isn't sometimes anything at all when trying to scale up Agile. To add more to the point, we see that our customers have equally wide array of opinions on these subjects.
We will debate these options to help you decide what approach is best for you. And what do you want to think when you are doing that.
To set the scene for this panel discussion, let's start from the beginning of this journey when an Eficodian wants to help his or her customer to achieve great things. My personal favorite is this objective versus subjective truths. I see it at the basement or setting the scene for the transformation journey. Defining the objective truth is the key enabler when you build trust within the organization structures. And also to say out loud, my favorite nightmare is when we have subjective truths in this asynchronous communication environment. That's my first pain point in the customer that I want to solve, so what is yours?
Hi, I'm Henrik, I'm from Denmark. I agree with Kalle, that it's very interesting when you get to go to customers as a consultant and you get very different tasks in how to help them. You can't really help every customer in the same way. There are some things that are, how do you say, some things that are the same basically, but the implementation might differ. I'm also lucky to know people who work in small unicorns and see what they do. I am also part of the Cloud Native Nordic, Cloud Native Aarhus admin group co-organizer and see what's happening in that space.
To me, Agile is something that was made by developers for developers. Great people meeting in a cabin or a hut or whatever it was, over a weekend and agreeing on some values and some principles. It doesn't really, to me, make sense literally to say, we want to scale Agile, because it's a manifest. It's a mindset. But I think what people mean is that you broaden Agile to other areas of the business.
The first thing I think where we witnessed this was when Patrick DuBois created his first conference called Devopsdays. It was supposed to be called Agile for System Administrators. But he didn't like the name. And I agree. That's actually when DevOps was born, the term. People have asked him, "Why didn't you create a manifest?" And he said, "Well, first of all nobody reads the manifest. Second of all, I want it to be able to evolve." So to me, this is the closest you can get to scaling Agile, is to take the principles and the values and see how can we apply that to our department. Whether it's software development, Ops, HR or whatever it is. And I think we've seen a lot of great stuff in the industry when this happens.
Now when you don't write a manifest, you still get the problem that people read the word and they say, "Oh I know what this is." They put two words together so we should just bring Dev and Ops into the same room. That's not what it's about. The good thing about DevOps is that it has evolved a lot since the first Devopsdays. So it's not just about bringing Agile into Ops, it's also testing and security and all that.And it's also much more than just Agile.
And then I think to me, there's no point in trying to scale an organization if your architecture doesn't scale. So if you have a big monolith, if you try to scale the organization, you'll just have a lot of teams depending on each other and it will be a nightmare. I've seen that at customers, no matter what framework they use.
Whereas if you look at the open source community, like Kubernetes, Prometheus, all that stuff, they are building huge sets of software and platforms, and they're not even hired at the same company. They work at different companies, work at night, not in the same room, and they make it work. They create huge sets of platforms and software and it works for them. They don't have a framework. But what they do is that they have a really good architecture that is decoupled. So people working on Grafana, they don't put Post-it Notes on Kubernetes people, sports, that we are blocked by this or we are depending on this. They create an issue, and if they want it, they can actually implement it themselves if they want to. It's not like, "Oh I work in this department, I'm depending on that department, I can't do that." Of course you can.
To me cloud native might be the thing that DevOps missed because the whole Configuration as Code era is dying out, because it didn't scale. And now we're as configuration as data with Yamo and Kubernetes stuff. It scales much better. I think that's, to me, the road to scaling your organization if you want, because of Conway, of course. It goes both ways.
Thanks Henrik. Nice opening. Szilard, do you want to continue?
Yeah, actually a lot of areas to jump on, and being a tester, feeling my point is to say that what might not be the case, or at least I need to challenge some of the things. Not like I disagree, but to structure what you're talking about, and when we're talking about scaling. So my understanding was that, or let me translate what you said in a different way. You said that you can scale things in a different way. Would I understand correctly, that scaling horizontally equals DevOps? So when we want to have more, you mentioned, quality assurance and operations and scaling agility to those teams. So we are scaling throughout the lifecycle of the product development. Scaling agility involving them in the Agile way of working. You would call that DevOps.
But then what happens with the other way, when we want to scale vertically? We want to scale the business end? We want to scale to hundreds of teams? We want to scale to 10 continents, because the company is so big? And you mentioned cloud native, you mentioned open source, and this is great, but I sometimes call these the cloud native butterflies, because you can do really whatever you want.
In open source you don't like the frame, but you just go to the next project. You as a person, you just leave, or you just leave the product behind. You have no contracts that you need to maintain that stuff for the next 10 years. You have a lot of freedom there, right? But what happens when those companies who are big in size who have a very complex system, who has 20 years of legacy? How can they start on their digital transformation? What should be for them the route to go? And they say Agile, and they say DevOps, but does that include scaling to the business level, scaling to middle management, scaling to the hundreds of teams and to the multiple continents they might work on?
Are we able to give an answer to them? Be it simple saying DevOps and cloud native, or we need a little bit more framework and structure for that?
To me, DevOps is more about just scaling horizontally. But it's part of it. I think what DevOps evolved into besides just Agile is to look at things from Lean, like value stream mapping and optimizing it. I really like the DevOps report Accelerate, the book, it's really awesome. What it tells us is that lead time is really important. One of the factors that you want. I think the problem is that when you talk about big, huge companies, like they're really a different thing. I don't know exactly how many people contributed to Kubernetes, but I don't think we can find a company with that many software developers in the world. It will be hard, I think. So Kubernetes, and that's just the platform.
Then you have Service Message, then you have Prometheus, and all the other stuff. So it is extremely big. And it works. And I think the problem is that you want to take big companies, enterprise companies, want to keep their structure. They want to keep their middle managers and other management and all that stuff. Cloud native environment doesn't have that. And it works.
I don't think they want to keep it. Sorry to jump on that. I don't think they want to keep it but they don't know what else to do. And that worked for the last hundred and fifty years in big companies and the last 50 years in softer companies, maybe 40. But they have no idea what to change and how to change. And then we can say yeah, DevOps and cloud native and look they have no middle management. Yeah, you just kicked, I don't know, half of the headcount out of the company.
I'm thinking of this like what actually is open source, that business layer? Why open source exists, because of course you can be a researcher to be really on the front line researching some new stuff. Is the business then the research? Is the business itself the research? But then Kubernetes, for me, for example, I believe the drive comes from somewhere. And of course the opens source community members, their daily work is revolving around abstracting of the complexity in their organization, with technology, because that's what they can do. That's why they create value to the world. Do the open source community have? What is their business organization
There are companies that create open source projects and then they have like pro version or whatever.
Yeah, of course, like the biggest companies in the world, like Red Hat.
Yes. They sell licenses. Not software
What is driving the open source community? What is the end customer? And I believe it's in them, so that's my... Well how do you feel about it, because the business is the community leaders, by themselves, actually. In my mind.
The way Kubernetes scaled is that they have SIG groups, special interest groups, and you can be part of it. It's not like you have a shirt and a tie and then you're upper management and then you decide how this should be, because Agile goes against that. So the developers know, they should be in touch with the customer, the end user. And they should decide architecture, and everything, because that's where the best solution comes from. It's in the Agile manifest
The great thing about open source is that they begin small, and then someone else say, "Hey I can use that. I have a problem, I'll create an issue." And someone else in the world will say, "I have the same problem, but I can actually fix it." We had a funny thing at the company I'm helping right now, where we're actually using a scaled Agile framework, SAFe. The great thing was that we actually decided between teams that they should help us implement All Thing because they need it. That would be the same as someone from Prometheus needed something new in Kubernetes, created an issue, and then actually implementing it. This is unheard of in enterprise companies, because you're hired to be here, and your manager doesn't get bonuses for helping other teams and stuff like that.
There's a lot of constraints there that's stopping collaboration between teams. But we are extremely lucky in the company I'm at right now, that management and the architects actually know this, and they understand it, and they say, "Go about helping people, because it makes sense." Communication is extremely important, and that's also part of the Agile manifest, right? To me, they need to look at the unicorns, because they have a very flat structure. I know a guy called Kasper Nissen from LUNAR, they are beating the good parts out of their competition, the big banks, they are a new fintech online bank. They are racing with 500 kilometers per hours.
Cloud native butterflies.
Yeah. I think the problem for big enterprises is that they have this very old, rigid structure. They have old legacy software that doesn't scale. Then you try to solve that problem by introducing processes. That goes against the Agile manifest. You should solve the problem at the root. Start with the software
That's why I talked in the intro about this subjective truth, and in my customer cases that comes in the form of the old processes. This worked before and I feel that this is the only way I can contribute in the organization. And my personal goal, or the target in the first month when I'm at customer site, I always relate to that. I want to know where he or she comes from, and ask the questions around that and know where he or she is now standing, and then show on that context, this has been proven to work, and with objectivity.
It is not possible to grow something that is not fundamentally proven working in that context. You need to show the way, basically, that's what I wanted to say. That's why I wanted to start with that because if you don't know why a fact something, you always assume and that is subjective based on your beliefs and your experiences, and how do you think. That is in my mind a fundamental problem that if you don't know the technology, and this is when we try to bring in the technology to the business heart, and there is a huge mismatch when you have oil as a business and water as technology. Or not the way, I don't know. But it's still in the same bottle but it's never going to mix. I don't want to say that it's going to never mix, but I want to see that the knowledge workers can in the future, talk in the same language and grow based on proven things that actually work.
The technology is, in my mind, the enabler, number one in this sense because it is something tangible. It is something that you can use. If you don't have technology then the world will be full of different business models where the company is solely relying on a person's knowledge, on how something has been done before. But technology is the basement that grows from the bottom and gives the same possibilities to all of the companies. Then they can use that asset and I believe that's the main motivation for this whole digital business transformation. It's going to be a long road, still. I see that I'm going to be getting to pension before my job here is done. How do you feel about that
Probably. Originally I'm a software tester, and with automation and DevOps and CI/CD and I've been told that I will be out of a job. Soon. There is no tester needed anymore. And I said, "Yeah, I'm even happy to work for that." And I know that I will have a profession for quite a long time. Still.
Without testing there is no software. That's why...
Yeah. They are all assumptions until they are tested.
You just need to build in quality. And everything's solved.
Because in security they are built in quality. Nice.
I think for me, the important thing is to do things in the way that works for you. So I don't remember where I heard this, but there's one person who said that they have this situation where they had some developers who tested their stuff, and they had X amount of errors. And then they added a test department, a silo. And the funny thing is the amount of errors in production actually was higher. And then they added more testers, and they actually got more errors. That was because they didn't talk together. They didn't integrate.
I think you have these different, how do you say, generations of practices, like testers. You have the old version, then you have the newer, then you have the newest.
But here, let's go back to management. Who made the decision to add more? It was someone doing this based on old practices. Beforehand, how it worked, if I had a problem, I added more resources and it solved the problem. But these people will make the same decisions when we talk about let's scale Agile. Let's do technology. Let's go with cloud native. Let's go with open source. How do they know how to do that? And these people are in very important roles, decision-making roles, now. So that means they are the ones who can make a decision. So they need to be educated. How? How do you show them the picture? Will you say, "We will do what that is good for you?" Now, how do you show that? What is your road map, what's the future, what's the vision that you show, when you say it will be good. But how? And that's what I'm missing.
Yeah. Do we need to wait for millennials go to top management, because those are grown from self-transformation?
I don't want to wait.
Yeah, I don't want to either, but Generation X is on the way. So how do we train the middle man? I also want to talk about this.
Well I don't like this topic, sorry, because we talk about inclusion, right? Include everybody. And we talk about balance and we talk about have everybody in. And you now just excluded quite a lot of people based on age. Sorry.
That was a little bit joke, here Szilard.
Well, be careful.
But yeah, jokes comes always from some sort of truth. Yeah, but that relates to my previous point that the sole purpose of middle management, what I see, is to make the knowledge worker's life easier. And if they don't know the context where they are working, if they don't know the technology, how do you utilize technology to achieve more? How then, actually, they can achieve anything? Why middle management then exists?
So how can you help them?
Yeah. That's why I also want to talk about this technology and showing I don't-
But is it only technology, we've been talking about processes and now we are talking only about technology. Will technology solve me my organization?
It's something that technology if you go on Wiki page, you can see that it's basically everything in the world. So it's something that you can create from science and show as tangible thing, an asset. If you think about technology transformation, it is actually how do you use these assets to grow more? To get more value out? Technology is not Kupernetes, it's all a process around it.
But technology solves problems, right?
It creates value more. It is not only solving problems. The bigger picture in technology transformation is actually giving more to the society. Without technology, we wouldn't here be speaking.
Okay, it's disruptive, of course. That drives innovation. But on the other hand, you are solving problems. How do you know what problems to solve? How technology solves my process problem? How it helps to have the objective truth, as you said? How technology helps with collaboration? Making my team members working together? These all happens, right? How to give information to management, to make right decisions if they need to do decisions or how the decision can be delegated to the right place where the information, the knowledge is? But how technology helps there, and is it enough to say "technology?"
No, you need to put the sociological aspect on it, so together they are more than one plus one. It's like three. Technology is rather science of craft. So it's a sum of technology skills, methods, and processes to combine together. That is actually one thing that is also one of our problems, people don't... If you are a business, Master of, sorry, what is this? MBA, are they teaching in the schools... in this panel here, do we have an MBA? Yes, we have one.
I had mini-school of that, just part. One quarter of the MBA, but back in 2006, and it was already that time full of digital education. Digital marketing for example.
But still, it's about actually economical view, still is it? It is not like that, MBA will not search for this kind of first principle point of view. If we're going to break everything then bring together with technology, we create more business. I believe there is in the world, there are people that can see it, and can utilize the both sides, technology and business to build something more fundamentally I see that in the classrooms they don't teach this enough.
Well Kalle, then let me challenge a bit. Can you be a good CEO by knowing the Kubernetes and working on open source? Does it make you a good decision-maker?
I say that if you have used Kubernetes to solve your problems, to get more end user value, then you are a good CEO. If you know something about Kubernetes that it's there, it doesn't help you. But if you know how to utilize...
It will help you with profitability questions, it will tell you where to invest, it will tell you what other company you buy, because you know Kubernetes.
No, I just said that it will help if you know how to utilize technology to achieve end user value.
But still you need to understand your subject matter. And you need to understand how technology helps in that area.
So you need to understand profitability questions and all the rest, right?
Yes. Of course.
Henrik, you look like you want to say something, but before I give the floor to you, there's something that I want to add to this conversation from this Masters of Business Administration perspective. Of course, a very radical simplification of that, but there was one hypothesis, which is to solve problems. And then another one which is to create value. And I would put out radically that the role of business administration is to take those two and make more money. I think that makes a big difference between whether you want to go and solve a problem and you get reward for solving a problem. And another one is you turn the opportunity and say okay if we do it this way, we actually improve the society or we improve our audience's well-being at large, which is creating value. Consequentially, if you do that commercially, then you will make more money. And if you go with this staggering simplification, which is that without, I think...
I'm not deep into the open source, and I would be happy to admit that, but I would say that in many respects open source objective is not to make more money.
And therefore, there is less need for MBAs in there, because of those three options, they want to solve problems and create value, and then it's somebody else's problem to make money out of it.
But now Henrik, you have for too long time looked like you want to say something, so we give the floor to you.
I have to go back a little bit. I think it was Edwards Deming who said, or maybe Toyoda who said, they're basically raising an army of scientists. And that's how I think I would agree with Szilard that Kubernetes in itself won't solve all your problems, or whatever technology you would bring in. You would need to have people who question things and see how can we make things better? I'm not just going to attend work at 8:00 and go home at 4:00 and do what I was told and then forget about everything. I need to be progressive, I need to test things out.
But also my problem with Lean is that Lean at Toyota assumes that the constraint is in the market and not at Toyota. And that's just not the case for other companies. So when people implement Lean, they fail horribly. In Lean, you say-
Software companies, or factories?
You said when companies implement Lean, are you talking about software companies or hardware, like factories?
It could be both. There are many companies...
So you that those factories, like Toyota, Mercedes, and others, have failed miserably implementing-
Not Toyota. Toyota hasn't. I didn't say that. Toyota is at a place where they have been having this mindset for many years, and they've improved stuff over many years in a very rigorous way. And they are actually at a point where the constraint is in the market and not at Toyota. It takes 82 hours from raw metal to a car, or something. It's incredible.
The problem is when you then try to do the same thing at a company where the constraint is in your company. Then it actually doesn't make sense to try and just optimize everything. And this is where Goldratt with The Goal is very important I think. You should figure out how do I find the constraint and how do I actually remove it? Or solve it.
And then it will move around, and eventually it will move into the market. But only Toyota, very few companies, at a place where the constraint is in the market. So you can't just implement Lean and think that then you'll end up like Toyota. You'll break your neck before you do that.
And then when you said that a developer, if they would become a good CEO or someone who knows Kubernetes, I think the truth is somewhere in between of what you said, Szilard, and what I could counter with is that well, Mark Zuckerberg was a developer, Elon Musk is basically an engineer, the two Google people, who's not that active anymore, they were developers.
Also, what you said about open source, is that they don't need MBAs, but they make value. And in order to sell something you need value. If you're digging out diamonds, then digging out diamonds is the most important thing of your business. It shouldn't be, "how do I sell these diamonds?" And blocking people from "Oh you shouldn't dig now, I need to make you stop digging for diamonds because we have this thing in the way." You should dig out diamonds 24/7, right? And then you can sell it afterwards.
It’s time for an accessibility tip! Do you know what a skipnav is? Here are some http://DOs.Read more about our accessibility services: https://hubs.ly/H0xtBb30
#skipnav #accessibleservices #ux #uxdesign #digitalaccessibility #accessibility
The problem I see at big enterprise companies is that they do the opposite. They have a security department that says, "You can't put to production because we found these errors. We're stopping you." And this is where DevOps try and tell you well maybe we should build that in. Maybe you should scan us stuff every time we commit. So we get small errors and we get small changes and we get to market faster again.
But actually for that you need to transform your organization to be a value streamed organization. Right?
So everybody who is working on that value stream, working on digging out diamonds, they are together in a kind of one organization. And then you can apply the DevOps principles and have the CI/CD pipeline helping there do all the tooling and all the processes and all the communication channels included in that. But for that you need to do this transformation to value streams. We started to talk about scaled Agile frameworks as well. And I believe that's where those could help telling you that one of the most important thing is to re-organize yourself from this silo-based organization into value stream-based organization model. Following the value, your organization should be able to deliver end to end value. Then security is not a separate organization. Then security knowledge will be embedded in that value stream.
It might be my conception, or my interpretation of this is different, but every company in the world has a value stream.
It might be chaotic, it might be extremely slow. I think that the important thing about value stream is to first of all, measure them. Get to a point where you can measure. How much time does it actually take from raw metal to a car? Then after that, how can we make it faster? How can we remove handovers? And DevOps is all about removing handovers. Because we eliminate or we automate infrastructure, ops, test, security. So now we actually have the product owner handing over something to developers and they hand it over to an automated thing, and at the end, value pops out.
That's how DevOps tried to, how you say, optimize, use or be inspired by Lean, all that stuff.
Okay, I totally understand, DevOps will work, and it's a solution within your value stream-based organization. I didn't say you need to have a value stream. You're right, it's there. You need to have an organization around that. And that is probably part of DevOps as well. So nothing new. But having measurable flow of value through the CI/CD pipeline, for that you need to have everybody around that, right?
And my big question is, going back to the scaling, how do we do that with a thousand people? Or just, how do we that with a hundred people? And that's where we need something, and that's where I would say that we know DevOps is a solution, but we also know organization change is a solution. We can already propose something, we don't need to say, start measuring everything and then you will see what's your problem. Because we kind of know already what to change, what to start with. And this is the mean point where we disagree, I think.
I would rather measure than assume.
Let's talk through an example. There was this Battery Day from Tesla. I don't know if you checked it. The main invention that actually Tesla did is that they have been now buying the battery cells from different manufacturers: Panasonic, LG, and this one, CATL from China. The basic idea where they said that we see that the actual, the cell of kilowatt hours per dollar per kilowatt hours. We cannot go below certain amount. If we don't cut corners, if we don't make our value exchange better, and faster, and basically build the machine that actually builds the machine.
They did innovations in different areas in the packaging of the cells to the cars, they basically removed all of the shelling around it, and then most importantly they saw that the Panasonic had said to the other manufacturers, their business is to dig the diamonds. Their business is to do the cells. And then only do the cells for the customers, for Tesla, to use. And there is this handoff. There is specification of the cells, and then they're manufacturing them. But Tesla did it so that there is huge manufacturing processes where Panasonic is wasting money in revenue. A lot.
They don't want to actually optimize it because the margins are larger for them if they have huger processes. They can ask more money from Tesla. So they basically do their own manufacturing process where they innovated the whole anode and cathode part of the battery. So it doesn't take any more two weeks to try out the anode. You can just put the powder now into this machine and it wraps the coil, which makes the cell. They basically removed two weeks of manufacturing time. Costs are half of the cost and also the kilowatt hours per liter will go 50% up. I think that is a beautiful example of vertical integrating into the value stream.
In our IT organizations, I see that we need to see what is happening over the other side, to optimize it to our side. I don't know if it's through measurement, or interviewing, or just going there and see it, how they do it. And then basically science the heck out of the value stream. We need to science it out.
That's my example of, it's a huge change that actually everybody who is listening in to this call, and on this panel, will be affected in what they are doing now. I see that this is a perfect opportunity to see that in action. It's not only software. It's software and hardware together. And the science of the chemistry also, everything.
I think in my opinion, if you want to scale a software company, let's say you have 500 employees or something like that. I think we could learn a lot from the things that are being developed with microservices, and decoupled architectures and stuff like that. Because we don't really need that many processes, and we don't need a lot of requirements from others, because we're decoupled, but you need something to decouple it with. I meet people who want to build monoliths. I also meet people who say, "Just decouple it." Yeah but I need something to decouple it to. The most used ones in software now is either a queue or a API gateway. Kubernetes, for instance uses an API. This is why it scales so well in development, because you know the API. The Kubernetes API server is actually now being used for other things than Kubernetes, because it's a really powerful API.
It's also why people use queues like RabbitMQ or Nets, or Kafka, or something like that, because you don't really need to know what other people do or if they respond or anything like that. You just need to listen for stuff and then do some stuff, and put some other stuff back, right?
If we take the inverse Conway law, we can apply that to a business. And this is where I think some of the scaled frameworks fail, because they don't have this mindset of decoupling, from microservices and all that stuff. It's more like a monolithic where everybody has to deliver at the same time, the same tech time, and everything is removed from the people who actually do the work. I feel, at least, it's an old way of thinking, and trying to solve the same problem.
So for instance, some people do microservices, and what they do is they don't decouple it. That's actually the worst thing you can do, because then you get a distributed monolith. So you get no benefits, and you get the worst thing from both of them. You need to decouple it.
For instance, at a customer you could create a source of truth, and you could have a team that runs this, like the API server is in the cloud native. The entire cloud native landscape depends on that. It could be your queue, right? At some companies they have something that resembles that, but then they don't have an API, and they have manual processes, and they can't change stuff, because they're way too buried in work.
I think it's important that you take things in the right order. You need to figure out who's swamped and then figure out how do you scale that, so that they're not swamped. Only then can you scale the rest of the organization. Otherwise you'll just swamp them even more. This is the Theory of Constraints from The Goal by Goldratt
Yes. I'm just thinking, taking back to the previous example, you said, adding more testers or having the security team in a separate silo, so we see they are the constraints. What would be the management decision? We see this is the constraint, we have read the book from Goldratt, so let's solve it. Let's add more people to the security organization-
No, no no no.
Because that's how we know how to solve problems.
This is where DevOps come in and tell you there's another way of solving it. There's a much better way of solving it. That's building it in. Basically taking the Agile values and practices into security. Just like Ops did.
But that also means you change the organization around value, right?
No, I mean they just need to work in a different way
You need to move the people, right?
No, no no. They just need to know what CI/CD is. And they need to inject their stuff in the right places before we give it to the customer. So every time I do a commit, there's a scan. It's not like I wait three months and develop, and then hand it over to them, and then they check it with their keyboards and stuff, and find oh, there's a security patch here, you need to redo most of your stuff.
So you need to bring everybody to the same CI/CD platform first? How do you know who is everybody
This is where value stream is important. Value stream mapping is important. You need to know your value stream. And a lot of enterprise companies doesn't even know that
Actually, I totally agree that we need to redesign the architecture and go probably to the microservice architecture. Not because it's so nice, but because you can decouple and then because of that you remove the dependency problems, right? Hopefully you remove the dependency problem between your organization as well, right? So we need to change both. You need to change the organization as well as you need to change the architecture of your product.
You need to change the mindset at least.
And the mindset as well. And you change, you said, just the queuing mechanism and APIs, in your product. And you need to have similar things in your organization. How teams collaborate, communicate to each other, what channel they use, what language, template, whatever they use. What queuing mechanism, how do they queue work? And then we go back to Lean, working process limits, how do you do that, how do you handle that? We could say, yeah, DevOps does it, but how? How do you do that with a thousand people? So again my question is how do we scale all this?
If you take limit work in progress, Henry Ford solved it by minimizing the space between work centers. You've got no space left, you couldn't produce any more.
Where Lean, they shrunk the batch sizes, because then you had more overhead of handovers. That would then slow down the entire value stream. They also had the Andon Cord, and all that stuff to control the pace. And then comes Goldratt, with Drum Buffer Rope, basically making it a pull system. In many cases in traditional IT, infrastructure is the constraint. They are swamped with tasks. One thing we got from Kanban and Scrum is work in progress limits. I'm a huge believer in putting that on the backlog. And say if we can handle 10 tasks a week, then it should probably be set to 20 or something. And if someone come and say, "I need number 21," and we'll say, "Well, sorry, you need to re-prioritize," instead of creating a two year worth of backlog that you cannot manage or reshuffle or be agile about.
But with factory it was easy because you had space, right? Space was the limit. The inventory between the two work stations was your work in process limit. In Agile, I just add another thousand tasks! Great! No limit!
The funny thing is, in production you have this small sticker on each batch size. The name of that was a Kanban. Today in IT that Kanban is your issue in Dura. So if you put a limit on your backlog on a team, it's the same way of saying, you cannot put more work to us until there's space in our backlog, because we don't want to be swamped. And every time you fail at adding an issue, it's actually a mechanism to tell you where the constraint is, because if you don't have it, it will just keep growing.
Is that a limit in backlog, or is that a limit in work in progress? And I'm genuinely trying to understand that.
I would put it on the backlog. Because otherwise you're...
As James Whittaker stated that, It's good to describe processes, but if you want them to be kept, you need to make the process unavoidable. Meaning that it's done by your tool. Your tool cannot let you go around, right? So, if we talk about process limits, that means your backlog will need to reject it, like physically reject it. You cannot add more. So we need the mindset, we need the understanding what we need to automate and build in to the tooling, right, to implement our process into the tooling.
I guess we are almost out of time, but one thing that I wanted to discuss also is this, when DevOps is talked about in an organization, okay, this will help you in the long run, etc, They are labeled as these system teams in many of these frameworks. How do you actually do that? Okay, the system in the long run. If you have a system team employee or knowledge worker, and they are the bottleneck always, or usually, even more when they want to change the whole value stream, because they are very much needed to build that scalable through technology. So every other people can then utilize those APIs and assets and tools, etc.
So how do you transform, in reality, these team to those value streams and teach it in these cases? Because I see that that's a bigger fix, because we don't have 24 hours in a day, we only have two people, or usually five or 10 people there. How do we optimize the use of them in this transformation?
I've seen actually, that big verification organizations has been renamed to System Team as such, and then nothing has changed. And that's actually not what you want.
Or the other thing was, we said the Automation Team should be the System Team, but we are expecting them to automate test cases. So the testers are running them manually somewhere figuring out, and then they give it over to the System Team to automate. That's again, not the case. So for me, and maybe it's not the answer to your question, but what is very important, automation as a service. On demand. Those people need to create the platform so others can do their own work without actually talking to them. And this platform should grow and improve and communication platform, CI/CD platform, test automation platform, I would even say regression test as a service. So I can run it for me, but someone else defined hundreds of test cases in it, for example. And security test, as a service just to come back to that one. No functional test for four months la la la la, whatever, as a service.
I believe that's important to understand that they should work towards that, and then I would be able to transformation road map for them as well. Of course work on that plan somehow, and go into that vision, of course always changing the direction when needed.
Henrik, your last words? Then we give the very last words to Kalle, and then that's it.
I think it's very interesting how we can scale enterprises, and how we can make them more efficient. I don't think anyone have a silver bullet yet. I think people need to be careful not just to adopt the Spotify model, that's the best example I think. But steal, get inspired by Lean, get inspired by TOC, by Spotify, take what works for you. Always, try to keep in mind that your software needs to scale if you want your organization to scale. Those two things go hand in hand. Conway was quite clever with his quote, so I think that's important. And not to underestimate the constraint. I think it's important that we have people who hunt constraints down and actually remove them. I had my share.
I just want to finalize that it is the whole system, it is the whole organization, it's the technology together anyway, that makes or breaks it. I have seen it in action. I have done transformation in bigger organization. It is described in Gartner's Bimodal and also the whole organization, I have seen both. It's truly a fundamentally fun thing to see when people actually are not anymore these kind of line workers, in an old factory line, rather they are in this beautiful manufacturing line with knowledge work. And they can actually live on that one, but it still gives them that kind of process needs to enforce its rules. It's not like an anarchy kind of thing. I don't know if there's... Okay Szilard wants to say something, but I am kind of done. I don't know if this was very helpful, but we have a lot more to talk about in the future I believe.
Yes, I just wanted to add that too controversial on the other hand I believe these big frameworks might help. I agree you need to understand which is better for you, but I think you need to understand what the big frameworks could give you, how they can help you and they typically have a lot of good knowledge backed in a logical way. So it's good to understand how they might help in your organization.
I would say you are not alone and you can start with those frameworks as well, but I agree that none of them should be religiously followed by the word. But you need to understand why. And even those frameworks tell the same. You need to understand what is good for you.
You need to always learn. That's my motivation, it is always to learn every day. That is some hard thing to change if people don't want to learn. Maybe one thing that we can call agree on.
I think it goes very well in line with the objective of this conversation, is that people start adding stuff in this conversation after the final words.
There's no project, you know. That's a lie.
There's no root cause, there's just an entering point where you start getting deeper and deeper, and I think the closing words is one thing like that.
Well, nevertheless I need to thank you Kalle for preparing your opening speech. Thank you Henrik for participating on such a short notice, and thank you Szilard for participating as well. It was very interesting and I hope it also gave you some ideas for the forthcoming episode. Keep them coming and we'll probably reconvene maybe with a different setup or with slightly same setup. You never know.
Thank you very much, Lauri.
Thank you so much.
I have to confess that before the recording I decided to try and stay silent as much as possible, but when the debate was on, I couldn't quite keep up with my promises.
Going forward, we'd be more than happy to bring you into a debate or listen to you for ideas. Reach out to us at Eficode via Twitter, LinkedIn or Facebook and let us hear about you. Until then, stay safe, and interested in continuous learning.