Open source vs. commercial: The toolchain dilemma

Darren (00:07): The more technical a tool gets, the more likely it is to have open source versions that are high quality. Marc (00:21): Welcome to DevOps Sauna Season 4, the podcast where technology meets culture and security is the bridge that connects them. Hello, we are back in the sauna. Hi, Darren. How are you today? Darren (00:42): Hey, Marc. I'm doing pretty good. Marc (00:45): Not bad for a Monday. We have a neat topic today, and it's, can you put together a fully open source toolchain for your software development, and is it any good? Darren (00:58): Yeah, this came out of Atlassian's gift to us all just after Valentine's Day when they retired Jira and Confluence Server versions. And obviously, if that's part of their trying to get more people to buy into cloud, that's up to them, obviously, but it's left kind of a gap in the market for a lot of tools. And I started looking into whether we can find any good open source replacements, and then it led to this whole avalanche cascading problem of can I build the entire toolchain just using open source technology? So that's where we are now. And I think we can talk about what happened and how the results worked. Marc (01:38): It's interesting for me as well because one of the things that I think about here is my first touch of enterprise version control or software configuration management was a thing called Continuous CM Synergy. Eventually, it was renamed, and then it became Rational Synergy and there's a few people out there that remember that. And somewhere around the time that this was taken into use in the enterprise where I was working, there was also people moving from CVS into Subversion, which was the one of the early open source version control systems. And then, of course, this is before we all went into Git anyway. But I've also seen there's an interesting thing here as well where we end up with there may be enterprise tools, but we also find that developers will have preferences and maybe even have some of their own kind of shadow things from the open source world. Darren (02:30): Yeah, I actually think it's surprising how much of the open source stuff sets an industry standard in places we haven't really expected or fully appreciated. So hopefully we'll be able to shine a light on some of that as well. Marc (02:43): Let's talk for a moment about what open source means. So there's this very, very important, clear definitions. We're not following things completely exactly in this podcast. There is, of course, free beer versus free speech or gratis versus libre. And we're not getting into the exact definitions of free open source software here, but we are aiming to talk about free and open source software while keeping in mind that some of these softwares have enterprise licensing models behind them. So you can start out essentially for free and then move up the tier into having something that is fully supported and, of course, paid for as well. Fortunately, we're not going to get into those distinctions today, but we are going to talk about what you can essentially get for free off of the Internet and do an awful lot of work forward on. Darren (03:35): And I also think we do have to mention quality here because I'm willing to admit I have a complicated relationship with open source projects in that the theory behind them is always great, and the results are less so. And there's been this kind of stigma in the IT world of the open source projects are sometimes lesser quality than commercial projects. So I think it's important that the tools we're discussing need to be considered from that angle of whether they are able to stand up to the commercial tools or whether they are just a bit lacking behind. Marc (04:12): And it's interesting. I have experienced quite a bit with open source software. And to me, nearly all the open source software that I was familiar with was built and maintained by software professionals working for a variety of different organizations and sometimes even, you know, collaborating across different organizations in the way that they contribute and maintain those components. Darren (04:32): Yeah, that's quite often the case. And it's not to say that like open source software is bad, but open source software is not really as vulnerable to end user requirements as we might expect commercial software to be. So it can end up being a case of a lack of consideration for user experience, for example. So let's say it's an interesting problem, and it was an interesting thing to dive into. Marc (04:58): So we'll go with open source software is software with source code that anyone can inspect, modify, and enhance as a working definition today. Darren (05:07): Yeah. Marc (05:08): All right. So let's start with version control. And if we start with version control, I think it's worth mentioning that there's several, I mean, you can get, you know, open source, get out of the box as well. But there's several systems out there that give a pretty end to end out of the box, the open source box, they give a pretty out of the box end to end experience with not only version control, but also things like CI/CD and storage, compliance, even task management and documentation. Darren (05:42): So we have two tools for this. We have both Gitea and GitLab, and I think everyone will be familiar with GitLab. It's one of the big contenders. And the fact that it kind of runs alongside GitHub for dominance in this industry is kind of impressive. So I actually want to take a step away from GitLab and talk about Gitea mostly. This is the one I use because it's kind of an underrated tool in my opinion. It basically has everything you need. So it's a fully featured Git platform. It's version control that includes things like CI/CD. It includes projects, it includes basically whatever you need or whatever you expect from your version control system. CI/CD is actually called Gitea Actions, and it's compliant with GitHub Actions. So it kind of borrows the syntax from that and turns it open source. And it's kind of interesting in the way that it has an extremely small footprint and a very, very easy maintenance, I would say. Of most of the systems I tried, it's probably one of the lowest maintenance systems. So GitLab is great but can be a bit of a resource hog, whereas Gitea will run on practically nothing, and it will run reasonably well there. So when starting this experiment out, I did build my version control with Gitea, and it was essentially just install and forget because it was such a fluid experience. Marc (07:12): Nice, nice. On the other hand, I've had a lot of organizations where, you know, I've talked about this in some previous podcasts where somewhere in the organization there is GitLab running, and oftentimes this has been a developer's favorite in many places that I've worked, and it provides a clear upgrade path towards enterprise support and has all the security and compliance tools when you get to the ultimate level there. And, you know, both of these are, Gitea and GitLab are really good foundations for SMEs, startups, and sometimes even a bit larger. We have some massive corporate customers, finance, and other sectors running GitLab Ultimate, but yeah, it's one of the things I always like to think about is when you have a tool that can support anything from a small to a very large organization to me, that's very interesting, and you can certainly do that with GitLab. And I'm not sure about Gitea, how long or how big an organization that has supported. Darren (08:14): It's curious because they do also have a commercial model. So I'm thinking they have the ability to scale it at least somewhat. I'm not sure how well it would interact with like a huge enterprise, but it is at least scalable past SME level. But you do raise a good point because if we're talking about company size, then I think we do need to have the discussion of should you be deploying this kind of thing by yourself, or should you be going towards software as a service? Cause this seems like a good time to talk about that because we jumped straight into the toolchain, and we didn't actually consider whether this is a good idea at all. Marc (08:52): Really key at the moment is why would you host any of this stuff yourself instead of going to something that is software as a service that you can buy just a subscription model and that is fully supported in terms of backup and uptime and really well-productized configuration out of the box. Darren (09:12): That's true. But for example, it's extremely easy for us at Eficode, at least in my business unit as a managed service provider, to say, yeah, definitely buy it as a service. But I think there's a level of independence by hosting these things. And I think the thing to look out for here is to be cautious about the idea of trying to save money by going down this path. Yeah, you won't pay anything for the software here, but it is a level of technical buy-in required to install and maintain these systems. So yeah, I feel like the people most tempted to dive into something like this to save money would also be the people who aren't going to save money through it. They're going to be the people who end up tripping over into technical debt because they don't have the money for a team to support the infrastructure. Marc (10:04): Absolutely. Free is not always free. Darren (10:07): Yet free speech, not free beer. Marc (10:10): So what about, let's look at some other tool types. So compliancy. Darren (10:15): Yeah. Compliancy is one of the interesting areas where we have one of the industry standards, SonarQube, actually as like a code quality scanning tool is actually open source. But again, we have this kind of pricing model where it's open source for a lot of the most common languages, except for a couple of kind of key languages missing from it. And two that caught my eye are C and C++. And anyone who has heard me talk, knows at some point, I will talk about IOT and talk about the security concerns in IOT. And it's something I continue to bring up because it continues to be a problem. And, like, I'm not saying this is a problem, but for like a problem that SonarQube is responsible for. But what I'm saying is having industry-standard security tools, which are free to access, kept out of the hands of a specific kind of developer, and that's a C developer who then has to pay for a enterprise version or a professional coder version, then obviously the easiest thing for this coder to do is just go, yeah, I'm not going to check my code quality instead of paying for licenses. So it's kind of this interesting idea that maybe this kind of enterprise licensing or this kind of gradual licensing that has important features removed could actually cause security risks. Just a kind of chain of thought I ended up on when I realized that every language I was writing in, except C, was available to be checked by SonarQube’s community edition. Marc (11:49): And keeping in mind that on the 27th of February, the president Joe Biden's administration wants software developers to use memory-safe programming languages and ditch vulnerable ones like C and C++ from the White House office of national cyber director. And so we've got the U.S. government saying that C and C++ are more vulnerable to attacks. And one of the most common open source tools, SonarQube, doesn't scan C and C++. Darren (12:22): But I mean, that's the thing with C, you're not going to remove C from the IOT sphere. It doesn't matter who says C is insecure. It's the best way of getting the most out of your hardware that you can. And so people running on limited hardware, like in the IOT space, they will consistently end up running C. Marc (12:44): And the S in IOT is for security. Darren (12:48): I like that one. I'll have to remember it. But there are other compliance tools I think we should talk about. So there are a huge amount of freeware tools by this one group called Aqua Security. And I want to take a minute to talk about those just because Aqua Security kind of, I want to fanboy over it a bit because Aqua Security put out these great tools that I talk about a lot. These things like Trivy and Kube-bench and they are like these minimal command line tools that you can insert directly into your pipelines. They don't require, or they typically don't require like a third party installation. They don't require anything except the ability to a lot of times pull Docker images, or you can pull them as like scripts and just run them directly in your pipelines. And you can usually take three or four of these tools, add them to your pipeline as one line tools, and increase the security of your pipelines, the compliance of your pipelines tenfold. And they are great tools and they all exist under these open source licenses. All the source code is all available on GitHub. I don't actually know if they have a enterprise model because they are one of the few tools that didn't beat me over the head with it every time I looked at them. They just quietly sat there and did their jobs with no nag where it was no kind of bloating, nothing saying, “Hey, would you perhaps be interested in buying an enterprise version of this software?” So yeah, the Aqua Security tools, they are just no matter where you are in the software development system, they are worth a look because they are so simple to use and they are extremely powerful. Marc (14:27): All right. There's been a lot of talk of securing the software supply chain, and I think there's some really interesting ones here. OWASP ZAP, Dependency Check, LunaTrace, and I think that many companies are still allowing developers kind of pretty much ad hoc to bring in other open source packages and then are looking at, maybe at the most, they're looking at licenses after the fact, in order to understand, make sure the developers haven't brought something with a copyleft or something like that. But there's a tremendous amount of things that can go wrong in the software supply chain. Darren (15:04): And it's also one of the more interesting areas because of how GitHub is pushing Dependabot. So Dependabot is part of GitHub Advanced Security. And what this does is it just works. It just sits there at the push of a button and tracks your dependencies. So in order to combat that or not combat it, to compete with it, OWASP tools like the AttackProxy, ZAP, and the Dependency Check are extremely seamless. Now, I probably wouldn't put the OWASP ZAP directly into a pipeline because it will be quite a resource hog. But Dependency Check is something you can immediately insert and just keep track of this. What you want is the SBOM, the Software Bill of Materials, because with the amount of open source libraries going into every bit of code, no one's really, I think we can say that no one's really coding everything by themselves from scratch because that's a bad idea because the work's been done. But with the amount, it's important to be able to track a release down to which components came into it because in security situations, you may actually be required to find out, okay, which component of this caused the problem. So, yeah, Dependency Check. There's a tool called D-Track. I actually don't know if it's open source. I think it is. And then LunaTrace. These are great tools for managing your software supply chain and the security of it. And it's actually kind of interesting. One thing I noted here was that the more technical a tool gets, the more likely it is to have open source versions that are high quality. Marc (16:38): You mentioned something that's really, really important that we're talking about a lot with our customers, which is the Software Bill of Materials and having all of the dependencies represented, having the checksums for those are a really important part of, I suppose this goes back into incident response planning as well that we had talked about in a previous episode. Darren (16:58): Yeah, but it's actually going to get a lot more important over the next couple of years as we talk about AI Bill of Materials. It's where we know exactly which iteration of which particular AI model is being used at any given time and know the training data and sources that have gone into that AI. Marc (17:18): I haven't really seen a lot in the more tort societies like United States about how the lawsuits are going to creatively ambulance chase the AI community and look for things like discrimination or things like this. And things like having an AI Bill of Materials, I think will also be a really interesting topic coming forward. Darren (17:40): It's something we're not going to be able to avoid, but perhaps that's a topic for another day. Marc (17:44): Yep. I think it is as well. How about secure storage or secrets management, I think as well? Darren (17:52): Well, then we actually get to probably the most interesting iteration of the open source model. So HashiCorp have HashiCorp Vault, and HashiCorp are kind of distinct in the space because, I mean, we can't beat around the bush. HashiCorp was worth $5 billion, and their two most common tools are Vault and Terraform. And these tools are completely open source, no kind of nag that I've seen, no advertising. It's just this is the tool. And then they sell enterprises and they become a company worth $5 billion on that. So they actually set the industry standard with HashiCorp Vault while being completely open source. So for me, I would say the search actually stopped there, knowing that the best tool, in my opinion, for this is open source and has no kind of problems, nothing questionable, nothing that would worry me, no kind of nagging every time I log in that, “Hey, we'd kind of like you to buy an enterprise license” because they've got to the understanding where they know their customers are buying the enterprise license. They know who their customers are, and they know who their customers aren't. Marc (19:08): I don't know how many people here think about business models on a day-to-day basis, but I bet there's a lot of people listening that have heard, “Well, how are we going to make any money if it's open source?” And that's always a wonderful example to pull out. Darren (19:23): And also from the developer side, anytime you want to make something open source, especially if it's been developed with company time, you'll always receive this pushback of, well, why would we give this to our competitors? Why don't we sell this? And they're kind of fair questions, but we have great examples here of how money can be made by making something open source, by making it of such high quality and making it the industry standard, and then the people who need to buy it will buy it. Marc (19:55): So I think there's a problem area that we've not really have a good solution for these days, which is where are we putting our documentation and keeping that up to date? Now, of course, I think, you know, oftentimes when I'm talking about documentation today, I'm thinking in terms of tech docs and readmes, which are stored along with the code for developer use and APIs. But the replacing confluence for internal use, intranets, wikis, and things like that that's not very easy these days. Darren (20:27): So yeah, the idea that you're saying of tech docs, these kind of readme files bundled along with the code, they make sense for coders, but for example, in building a pipeline like this, we are creating something that needs to be documented and sure we have infrastructure as code, so we can push these things out as code, but having a separate documentation platform is in my opinion, kind of crucial. And this caused the problem with Confluence because Confluence was extremely good at what it was designed for. And it still is extremely good. You can still use Confluence. You just have to be a large enough client to use the data center version or go into the cloud. So finding a replacement for Confluence was extremely difficult. I tried several options. The first I tried was XWiki, which was kind of unintuitive. It didn't really make any compromises for user experience. So extremely simple things like putting new colors in, changing a banner. These were extremely difficult things. It didn't even start with an admin control panel. That was an add-on you had to install to administrate your own website. So that was rough. Then the next option is MediaWiki, which most people may know is actually the backbone of Wikipedia. And so, yeah, what we have with MediaWiki is this platform that's designed to run one of the largest stores of information on the planet. So while it might be free open source software, you are not going to run MediaWiki well without a team of at least a few people dedicated solely to managing that software. It's an interesting exercise to try and get it running, but it's kind of unwieldy for the purpose. Marc (22:22): And some of the statistics that we look at when we talk about transformations or enterprises, we're already looking at sometimes developers spending 40% of their time on toolchains, and I think it's a really important takeaway from this discussion that having a small team that is dedicated to your development platform, even be it open source components, is really important. And the alternative is you might have a lot of developers that are maybe not necessarily as well coordinated that are spending time on maintaining the toolchains and the tools rather than focusing on what they should be doing, which is using their domain-specific knowledge for value creation in your company. Darren (23:06): And that's actually the cornerstone of platform engineering and where that comes from, the idea of having the small dedicated team to managing the platform, to managing the experience of the developers throughout, and you can do that with open source or commercial tools. Marc (23:21): And then, I think finally, one of the most important ones, task management. Now, I was a huge Bugzilla proponent for development for years. And one of the things that to me was really interesting with Bugzilla was that the way that you could structure work, the way that you could link things and dependencies was really powerful. And I think it's something that even most of the task management softwares that I see these days have never really gotten close to, like Jira, you've got a whole bunch of different kinds of relationships that you can use in order to link tasks together, but Bugzilla was the king of free open source bug management and error management and tracking for a long time. Many, many enterprises used it. Intel and Nokia and all kinds of big companies. But then people didn't really translate it very well for, even though it would be really powerful for things like requirements management, but now, we're kind of moving into newer eras where there's a lot of free services available for task management, work management, requirement management, but there's not a lot of very good open source ones, I think. Darren (24:31): And on the subject of Bugzilla. So I think the main selling point of Bugzilla or the main problem with it was that it was designed specifically for developers, whereas a tool like Jira is something that obviously had development in mind but has kind of grown and expanded and shifted to kind of encompass everyone's work in a way. And I would say, yeah, it's extremely difficult to find something that works as an open source replacement for Jira to serve for task management. So the ones I tried, one was Open Project, which was, again, it was relatively good, but it does this kind of open source to premium model. So it locks what I would consider vital features behind a paywall. So things like boards, you can have a basic board, but if you want anything more advanced, you have to build it yourself. So things like Kanban, they're locked. So it kind of feels more like a demo software to say that this is something that could fit your purposes if you want to buy it, but then I tried this other model, it's relatively new. It's called Plane, and so far, it tests well. It does have the same kind of model, but the enterprise-locked features are not as obvious. It's not as crippling if you don't have the enterprise version. So Plane is actually a potential contender, but it's currently not where Jira was, but we'll see; time will tell if that improves. Marc (26:06): All right. So open source toolchain, Darren, can you actually do it? Darren (26:13): Yes, you can. And if you put the time in, it can actually become pretty good. Marc (26:18): Okay. Darren, should someone actually do this? Darren (26:22): Probably not. This is the kind of thing that is a technical buy-in. And for smaller businesses, getting it as a service is better. And for larger enterprises, it probably makes less difference if you're doing open source or commercial tools. Marc (26:39): All right. If someone goes down the open source toolchain path, where are they going to struggle the most? Darren (26:46): Oddly, they're going to struggle the most the closer the technology is to people. So if it works quietly in the back end of things, it's going to work seamlessly and silently. And the closer it gets to people and the more advertising opportunity, the more difficulty you're going to have getting something that's as good as the commercial software. Marc (27:04): All right. So ultimately, if you're going to go down this path, should you go modular or use one platform? Darren (27:12): I think using one platform is a trend we've seen, and it's a trend we're still going to see. So you're going to end up using something like GitLab. I mean, obviously, you can have Git separately and use Jenkins for your pipelines, but you are going to end up wanting everything in one place. Marc (27:28): And to be frank, nobody would start a new project with Jenkins today. Darren (27:32): Well, that's what we hope with our fingers crossed. And yet we somehow know it's not true. Marc (27:36): All right. Thank you, Darren. Not only has this been insightful, but it's also been kind of nice to revisit some of the histories of how we got here. And to see, going forward, there are so many good choices available for simple online software as a service that I'm not sure why going back to maintaining our own open source-based toolchains would make any sense anymore. Darren (28:02): Yeah. It's starting to feel a little bit like masochism at this point, just for the sake of wanting that experience. But yeah, the software-as-a-service era is definitely the way to go, in my opinion. And I even like maintaining platforms like these. Marc (28:20): All right. We're hoping, folks, that you'll turn all of your pets into cattle and use software as a service. Thank you, Darren. Darren (28:28): Thanks, Marc. Marc (28:28): All right. Thank you for listening to the DevOps Sauna podcast once again, and we'll see you next time. Goodbye. We'll now tell you a little bit about who we are. Hi, I'm Marc Dillon, lead consultant at Eficode in the advisory and coaching team. And I specialize in enterprise transformations. Darren (28:50): Hey, I'm Darren Richardson, security architect at Eficode, and I work to ensure the security of our managed services offerings.