Stories about how eBPF is used today

Bill (00:05): People are going to want things with eBPF inside, not because they know exactly what eBPF is, but because of what it unlocks and enables them to do. Marc (00:19): This season, Andy and Marc are back with a fantastic group of guests. Andy (00:26): I've bet the depths remain classified. And Marc keeps his head in the clouds. With our combined experience in the industry, we can go from the bare metal to the boardroom. Enjoy your time in the DevOps Sauna. Marc (00:45): Hello, we are back in the sauna. I am super excited and a bit humbled to have the opportunity to have a conversation with Bill Mulligan the Community Pollinator from Isovalent. Hi, Bill, how are you? Bill (01:01): Good. How are you? Marc (01:02): It's really fantastic. And it's been nice we had a little bit of pregame warmup to speak and you just drew me in with every single thing that you had to say. I have my usual cohort here Andy Allred. Andy (01:13): Hello-hello. Marc (01:14): When we were talking before and I've been on a few conferences on the circuit with Liz Rice. We we had her here on the DevOps Sauna podcast before, and we're still educating. It seems like a lot of people don't know that much about the work that Isovalent is doing and eBPF in general although you are hugely across the world. Could you tell a little bit of stories, Bill, about the work and the impact that you guys are doing? Bill (01:43): Sure. Isovalent is founded around eBPF. And the founders made the bet a long time ago that eBPF would be the future of how we do things within the Linux kernel and in the cloud native world. And so, I think fast forward to today, like about seven years later, we've really seen that long-term bet really starting to come true. And Isovalent focuses around the Cilium project, which is a host of different projects covering things in the cloud native world like networking, observability, and security. And the way that we're doing things a bit differently is doing those with eBPF from the ground up. And that unlocks a really new way of doing things and is what really excites me about being at Isovalent. Marc (02:29): Excellent. I think just to give a sense of the scale, there was a couple of data points that I believe that you had a couple of stories about how eBPF is used today. Bill (02:41): Yeah, eBPF is, I guess, really transformational technology because it allows us to unlock innovation in the Linux kernel, which is getting to be over 30 years old now. There's a lot of code in there. It's deployed across millions, billions of devices. And actually, changing something in the Linux kernel is starting to be quite difficult. And so, because it needs to work across all these different places, you can't introduce breaking changes. And so eBPF allows us to add programmability and flexibility into the Linux kernel. And we're seeing a lot of really interesting real-world use cases that are impacting how people are doing things in production today, I think two of my favorite ones that almost everyone can relate to are around the stuff that Google and that Meta or Facebook are doing. On the Google side, main developer of Android and every single Android device in the world. And actually, my phone too because it's a Google Pixel uses eBPF. And so, you can think of the deployment scale of that. And what can use eBPF, right? Phones are quite different than datacenter servers or other applications of Linux kernel, but they want to be able to pull different metrics and data on how the phone is using the actual network. And with eBPF, they're able to add that functionality without needing to upstream it to the Linux kernel, right? They can get information that they need for a specific device without having to wait years for the kernel development cycle. And that's quite interesting. And then on the other side, on the Meta or Facebook side, they've actually been using eBPF to process every single packet coming into one of their data centers since 2017. And so once again, the scale, the bandwidth, and the performance improvements that they're giving them is pretty awesome. I know I use Facebook Messenger and WhatsApp all the time to communicate with my family. And it's interesting to see the foundational technology behind that is actually something that I'm working on in my day job too. Marc (04:42): So every single cat video posted on Facebook goes through eBPF? Bill (04:49): Exactly. Marc (04:50): Really, really cool. It's inexplicable when like we have a conference and we do a poll and we ask the developers in the audience, do you know about eBPF? And most of them they say no today. Bill (05:06): Yeah, I think eBPF is an interesting technology. And actually, the way that I look at it is in some ways, it's actually not a bad thing. I remember growing up, maybe you do too, the whole Intel Inside thing. I can still remember the little jingle behind it. And everybody wanted like yeah, I want Intel inside, like this is this is great. Besides people actually working on the hardware understand what Intel Inside actually meant? And I think we're going to be in actually quite a similar place with eBPF. People are going to want things with eBPF inside, not because they know exactly what eBPF is, but because of what it unlocks and enables them to do. It allows them to have things like faster networking, better observability, more secure systems, and those are the things that developers are actually going to care about. And it's going to impact the work that they're doing. The actual implementation details of eBPF under the cover isn't so important. And so, I think what we're actually going to move towards a world is everybody's going to be using and leveraging eBPF. But not everybody else. Everybody needs to know or develop or actually write the code for it. I think that's quite powerful and quite cool. People don't need to be writing C code, but they can leverage higher level tools like Cilium, like tetragon, do you have better networking, better observability, better security? And that's, I think, quite a powerful concept. Marc (06:39): Quite wonderful. We realized on the observability track; I think it was one of the last conferences when we were talking in the halls. That is it that EBPF defies quantum physics because it gives you observability without actually changing the result? Bill (06:56): Haha, so we need to be running towards eBPF. Marc (06:59): Interesting. All right, Andy, Andy (07:01): I was just thinking about this Schrodinger as eBPF. And there was an example that I brought up a while ago, we had a bit of telco code. And it was basically processing connections between two telco components and whatnot. And we had this weird bug. And when we turned the debugging on, the bug didn't manifest itself. And when we turned the debugging off, we had the bug all the time. And eventually, we figured out it was a bit of a timing issue. I love, generally, how eBPF programs work is they're not really changing what's happening inside the kernel. They're just being notified when something else is trying to make a process and acting on it, usually in a notify or deny but notification. This debugging stuff, it's like instead of having to change your code and change the timing, you just say when my code triggers this kernel process, I want to know about it. It really just changes the way we could do observability, which is just awesome. kilometry Bill (08:11): Yeah, I was actually having a similar conversation at KubeCon. I was also talking with a telecommunications vendor. And I was talking with some people on their 5G observability team. And their problem is they're deploying out to thousands of locations, they want to be able to have observability into those systems, how they're working, how they're running, this is really important to understand the performance of network to make sure we actually have this call right now. And the problem that they were running into is they needed a consistent way to do that. They didn't have 50 different systems, and it creates this operational nightmare. But then say they want to go for an industry-standard way of doing it, something like open telemetry. Well, then they have to go to each of their vendors and convince them to add open telemetry into their actual code base that they can draw that out. And so, when you're trying to convince all of your different vendors to add something to their code, and support it and do things over the long run, like that's a really difficult conversation and can take a really long time. But eBPF since it's already part of the Linux kernel, it doesn't actually require any code changes. It gives you that instant observability and now they don't need to go back to their vendors. They can automatically just have that observability. And yes, it's quite cool for them. Andy (09:25): Yeah, I spent a couple of decades working in telecoms and then I moved over to more traditional IT consulting and whatnot then. And when I talk with telecom people, my old colleagues and my old mindset is telecom is so special because we have all these special things. And now I'm realizing that it's really not that different. The biggest difference is there is such a mixed match of different technologies at different levels of maturity, and we have to be able to integrate with all of them all at the same time. Then something is you just described that instead of going to every single vendor and every single hardware type we have, and saying, we need to make this change, say, just keep running the way you're running. And we're going to notice when the kernel calls go through and trigger on that, then it just makes so much better user story for us to install that observability and set up observability tools, without really having to change what's actually doing the kernel calls, just watching the kernel calls. Bill (10:30): Yeah, I think you nailed it exactly. Especially in the industry like telecommunications, where they're going from the copper wire all the way up to Kubernetes. And maybe even Wasum. How do you create a consistent experience across all of that? And you can't unless you're using something that's available everywhere like the Linux kernel, like eBPF. Marc (10:48): I think this is an interesting segue into platforms. One of the things that we've known for years and if you look at like Simon Wardley's work, all the telcos are basically just becoming more commodity and more commodity all the time. And 5G was one little bump in the cycle that everybody thinks if they're going to get there faster, that if all the standards and all the tools were open, if the platforms were more open, then they would be able to leverage one another and get there faster. And then still have to do something else to differentiate, but this platform revolution is something that's quite interesting going on. What are your takes on how you see the platform revolution coming up? Bill (11:29): Yeah. I think platform is a funny word. It's new and it isn't new. We've seen this whole evolution from sysadmin, to DevOps, to SRE, to platform. It isn't new. But I think platform is trying to capture is what you're creating is this foundational layer that other teams are able to leverage. You want to create these easy paths for developer and give them the right building blocks to build on top of. And I think if we're thinking about how platforms have evolved. What it says to me is that there's been things before platforms, and there'll be things afterwards. Technology keeps on changing and evolving. But what I think we should take away from this, I guess, platform era, is what are the foundational technologies that will enable you to do your job today, and also for the future, for tomorrow, and whatever is going to be coming after platform. And so, when you're thinking about how you should build your platform, I think people should be looking at what is our use cases today, but how may they evolve in the future. And that will help them inform their tooling decisions or what they want to focus on. And I think that's why a lot of people I see right now are actually moving to Cilium, and fall in love with it. And I've done a lot of case studies and user stories. And I think the thing that always sticks out to me is people say I adopted Cilium because I did networking layer for my platform or for my Kubernetes clusters or whatever they're going to call it. And then once I had it installed, I realized that there's these great additional features that solve my problems. And my use cases as my scale, my complexity, my compliance requirements changed. And I think if you're able to have a technology that moves with the pace of your business, that allows you from getting stuck in this treadmill of our requirements have changed. Now we need a new tool to cover that instead of adding new things and figure out all the integration, it's just turning on this new feature. And so, people start with networking, then they want to do something like observability, well, there's just helpful for that. They start with one Kubernetes clusters, and now they have two. Now you can mash those together with cluster mesh. And so, when you're building out your platform, I would think about what technologies are setting you up for the future to because technology is going to keep on changing. We're still going to have jobs because of that. I guess, Eficode is going to help companies continue to transform it. I think you're probably pretty safe. And I think that's what's cool is when you can help companies choose technologies that are going to set them up for now in the future. Marc (14:07): It's interesting you use words platform revolution. Actually, when we talked before the podcast, and then we've talked a lot about platform engineering. It's one of the trends that we've talked about all year. Is there a connection between Cilium and developer platforms? Or the kind of platform engineering as we look at it and the DevOps space is creating those enablers? Do you see it as like an important part of an internal developer platform? Bill (14:37): Yeah. This is actually quite a funny thing for me because I think there's actually multiple layers of the platform. A lot of people talk about platforms. I was actually thinking of the Developer Focus aspect. Things like backstage, what's your internal developer platform, IDP? But there's another part of it too. There's the actual what is it made up of? What are the foundational building blocks for that? It's great when the developers can be like, here's my coding environment, here's how we integrate with the rest of the applications. But there's these building blocks that need to be in place for the developers to be able to do that. I think when you're thinking about networking, and observability, or security, these are the foundational building blocks. And that's where Cilium comes into play. I think Cilium in a lot of way is technology like eBPF. It's at its best when it's completely transparent to the user. And so, when developers don't know about Cilium, in some ways, I don't think that's a bad thing. It's a technology that keeps on working. It's the same thing if I go to the faucet right now and I turn it on. I'm not surprised when water comes out. I'm surprised when water doesn't come out. Or the same thing when I flip on the light switch. We're surprised when things don't work. And I want Cilium to become such a ubiquitous and transparent technology that people are surprised when it doesn't work because they just assume that I'll be there and it'll work. And I think that's what platforms set you up for is having a consistent experience that allows you to do your job without having to think about all these implementation details. Marc (16:14): It's funny because then I keep thinking open source versus enterprise partners. One thing that I'm curious is, are their companies building sophisticated observability tools or something? I know there's some things with eBPF, but based upon Cilium or are there some partners that you work closely with maybe or open-source projects? Can you can you tell me a little bit about the ecosystem, I think is the word I'm looking for.