Talk: Practical DevSecOps using Security Instrumentation by Jeff Williams, Contrast Security

Lauri (00:04):

Welcome to DevOps Sauna. My name is Lauri, and I am the Chief Marketing Officer of Eficode. Not too long ago, we held a hugely popular two day DevOps 2020 event. In the event, we had awesome speakers from around the world telling stories about DevOps tools and culture for over a thousand people online. Since then, we have made this record and it's available at the Eficode website, and due to the popularity of the speeches, we have now made them also available in our podcast. You can find the links to the video recording and to the materials referred in the speech in the show notes. We are also keen to feature topics that you find interesting in the area of DevOps. Do let us hear about you in our Twitter, Facebook or LinkedIn pages.

Lauri (00:47):

Today, we have an exciting speech from Jeff Williams from Contrast Security. He's the co-founder and CTO of the company, and he takes us through the security instrumentation with a hands on touch.

Jeff (01:02):

Good morning, everybody. My name's Jeff Williams. I am super excited to talk with you today about the magic of security instrumentation. I'm the co-founder and CTO at Contrast Security. We offer a complete DevSecOps platform that covers vulnerability testing, analyzing libraries, and also runtime protection. And today we're going to dig into a little bit of the underlying technology into security instrumentation, and what it can do to help you get really good at DevSecOps.

Jeff (01:30):

I've spent 25 years as both the developer and security expert, and despite lots of well-intentioned efforts over those years, we've really made very little progress in security. If you look here, you can see the average web application or web API has 26.7 security problems, on average. This is measured off tens of thousands of applications that we've looked at. That number hasn't changed in the last 20 years. That's really embarrassing. It shows that what we're doing isn't working.

Jeff (02:04):

It's not like we haven't been trying. There have been dozens of well-funded efforts to try to get better at application security. It's confusing because these security vulnerabilities aren't that complicated. So something's going wrong. To me, security code just isn't making it into all the right places in the application reliably. If you imagine that your application is a car that you're building, we're really good at getting good driving features into the car. It's got wheels and an engine and brakes and so on, but the security features aren't really making it into that car. I would say most of the security activities that we use to try to make the security code end up in the application don't really work very well.

Jeff (02:48):

How many of you have a security requirements document that you don't read or a secure coding guideline in some documentation portal that nobody looks at? Or maybe you have secure coding training that you slept through. I'm not blaming anybody here, but look, if it was going to work, it would have worked by now. And again, I'm not trying to blame anybody. I've worked in both development and security, and I have tremendous respect for both. But I think the root cause here is generally security has operated outside development.

Jeff (03:19):

The last talk was about shifting left. Well, this is a perfect opportunity to shift security left and make it part of development. Because they've been outside, I believe that security hasn't had a reliable path into production. So then they're forced into these very destructive guerrilla tactics to try to get the changes that they want made to an application into production. It's very counterproductive from a culture perspective.

Jeff (03:45):

So people talk a lot about DevSecOps. Some people might say we don't need the term DevSecOps because DevOps already includes security. I think it's a useful term because it allows us to focus on the security aspects of DevSecOps. It's a way of saying, okay, we're going to talk about what we need to do to accomplish security goals within the DevSecOps process, but it's not diminishing DevOps by any means. So we'll talk a little more about that.

Jeff (04:12):

I think too often when people say the word DevSecOps, what they're really saying is that we should shove all the security work onto developers. That's not fair and it doesn't work. You can't just take existing security tools and practices and make the developers do them. They're not trained to do that, and the tools aren't designed for that. So we needed to do something different. To me, DevSecOps is really about transforming the work of security, exactly the way the development work was transformed by DevOps.

Jeff (04:44):

So, if you think about the three ways of DevOps, security work needs to be broken up into smaller chunks to create flow. Right now, security work doesn't flow at all. It's these giant deliverables that don't provide a lot of value and they're sort of way off track on what they should be delivering. So that's the first way is we need to establish security workflow.

Jeff (05:10):

The second way, security needs to create continuous security feedback. That's not an annual pen test. It's not delayed outside the pipeline. Instead, we need to make sure that developers and testers are getting instant feedback on security without a lot of false alarms, because false alarms require experts to triage and that slows down the process.

Jeff (05:34):

And so then the last thing, the security culture can't be about blame and hiding. And so we've got to root out these destructive forces in security and make sure that we're not causing friction in our process. We need a security culture that's encouraging, that's looking ahead to what threats are coming, as opposed to what the threats were five years ago when someone wrote a standard. So that's great in theory, but how are we going to accomplish these things, and what tactics and technologies can we use to try to achieve some of this? Well today, I'm going to talk about using security instrumentation to do this.

Jeff (06:17):

So I want to introduce this topic by talking about the 94Fifty. This is an instrumented basketball. It tells you dribble speed and shot rotation and the arc of your shot, how many you've made and missed, and a bunch more details. It also comes with a bunch of drills and metrics that show up on your phone. So I've been playing basketball for a long time. I started using this after one hour, the 94Fifty let me know that my shot was too flat. I didn't have enough arc on my shot. It was amazing.

Jeff (06:54):

I didn't need an expert coach. I didn't need a bunch of extra steps. I just did it myself. And all it took was a few sensors directly inside the basketball. That's really powerful. And I believe that instrumentation can democratize whole industries. If you think about coaching here, we can give this ball to a bunch of kids and we don't need a lot of coaches, at least the coaches can take a different role. They can be more strategic and they can let the ball do a lot of the work.

Jeff (07:24):

Think about medicine, where your phone or your watch knows you're sick before you do. Think about really any industry where experts are a bottleneck. And the key here is that instrumentation sensors can directly measure the problem from the inside. That makes them much more accurate and gives a lot of the easy feedback directly to the person using the instrumented thing, and it allows the experts to be more strategic, where they can be tool smiths and coaches and things like that.

Jeff (07:53):

Let's talk about software. Turns out instrumenting software is really easy and it's amazingly powerful. It's a great way to do security testing. So we're going to talk about how this works. So to use instrumentation, and this is language independent. Most platforms have some support for instrumentation already. It's a little different in each platform, but the basic idea is that you add an agent to your application. Now this isn't an operating system level agent. For Java, this is a JAR file using the Java instrumentation API. For .net it's a profiler, works roughly the same way, and we can do instrumentation in Node, and Ruby, and Python, and Go, and other languages as well.

Jeff (08:33):

Now, as the code loads, it starts out on disk, but it loads into memory. And what we do is our agent hooks that process so that we can add sensors directly into the application to measure exactly what we care about. And it's accurate because we're measuring the real running application, and our sensors can gather not just a little detail, but all the contexts that we need to really decide whether something's a vulnerability or an attack. We'll go into that a little bit.

Jeff (09:03):

So these sensors, we can add them surgically to exactly the right methods, to gather exactly the data that we need. And then we can record and analyze the telemetry from all those sensors to keep an eye on whatever we want to monitor inside that software. This is very powerful. The only limit here is your imagination. So I'm going to show you three things that you can do today with instrumentation. This is the first, we're going to talk about using security, doing security testing with instrumentation.

Jeff (09:35):

I hate to show code on the screen, because I know it's sometimes hard to digest, but this is really the best way to show how simple and powerful this technique is. So I'm using Java here. I'm using a library called Byte Buddy that provides instrumentation support. So it makes it easier to instrument your applications. What we're going to do is we're going to instrument this application to show me anywhere that I'm not using parameterized queries. Those might be susceptible to SQL injection. So I want to know about them.

Jeff (10:06):

In my organization, it's policy that we can only use parameterized queries. We can't use the old school concatenated queries, because they might be injectable. So once I compile this, it'll generate an agent and then you just add it to your application and use your application normally. You don't need to know anything about security and this agent will watch and detect any place you've got non-parameterized queries. So what this code does is it'll report that the unsafe SQL agent is installed. Then it's going to search for any uses of any ... It's going to search for the implementation of the Java SQL statement class. This might be in multiple different places. You might have different database drivers or different frameworks that are using different database implementations, but anything that implements this interface will be located.

Jeff (11:06):

We're going to find the execute method and the executeQuery method, and we're going to instrument it with this security advice. This security advice is defined right here. It's really simple. All it says is we're going to shove in a line right before this method exits that says, hey, you're using a non-parameterized SQL query, and it's going to show us the full query. So this is really simple.

Jeff (11:31):

You could code this into the driver yourself if you wanted to, but it would be a pain. You'd have to recompile all the database drivers you have and then make sure that everyone's using those recompiled drivers. This is much easier. You can add this to any application. When it starts up, it'll add this capability automatically. So it's simple and it's safe, and it's really remarkably effective without any code changes for the whole app.

Jeff (11:56):

Let's take a quick look at this. I've compiled this agent and I've called this unsafe SQL JAR, and all I'm going to do for Java to specify an instrumentation agent, all you need to do is set this Java tool options, environment variable. So I'm going to do that, and then I'm just going to run this application the normal way. So here I'm starting Tomcat and running this application and you can see, it says unsafe SQL agent installed, let's go back there, unsafe SQL agent installed. Our application starts up and actually it's already dumping a few SQL queries. They can happen during start-up, but we'll do another one. We'll do a DevOps here in this application.

Jeff (12:38):

When we hit submit, you'll see we now just get a dump that says, "Hey, you used a non-parameterized SQL query. Here's the actual query." You can see there's DevOps that I just typed. And you can see the exact line of code where this happened in this query unsafe method in this class, on this line. So really powerful way to zoom in on a potential security problem. You don't have to print this out to the screen. You could log this to a file. You could send it to a bug tracker. You can do whatever you want with this.

Jeff (13:06):

So it was powerful technique for security testing, and there's really no limit to what you can test with this. We've used this for some very advanced data flow related testing. You can find SSRF, and XXE, and a whole bunch of different problems this way. So I just want you to imagine, so this is just for the slides I captured this. And I want you to imagine now that your normal test cases, when you're doing like your end-to-end testing, your normal test cases can fail for security reasons because we just throw an ... Instead of printing the stack trace, we could actually throw a step, throw an exception and stop this from running. That'd be a nice way to get security testing right into your normal test suite, without having to change anything. You'd know exactly which test cases fail for security reasons.

Jeff (13:54):

Let's do a slightly more complicated example. So in this example, we've got some more sensors and I'm going to show this to you in Eclipse so you can see a more detailed example. This has a number of different sensors in it. We're going to capture some details from the service method. We're going to look at access control calls. We're going to look at encryption calls, and some other sensors related to the ServletRequest and the ServletContext. We'll capture queries. We'll even look at prepared queries as well to gather some information from those.

Jeff (14:32):

And then what we're going to do is we're just going to monitor this application as it runs. So I'm going to go back, I'll stop this. I'm going to switch to the access control agent. And now I'm just going to run that application normally again, and we'll take a look at what this does. What we're doing is we're just going to be monitoring a bunch of things within the application, and you can see, I just have it printing out some data to standard out here on a periodic basis as we use the application normally. So I'm just going to browse to a few of these pages, XXE, XSS, maybe this command page, and I'll show you what we're gathering on the monitoring side.

Jeff (15:10):

So here you can see we've got test coverage. We're checking which routes we've hit in this application. We've also started to generate a access control matrix. We'll get to that in a second, and you can see we've gathered some HDP parameters from the application as well. A lot of this information will be really helpful if you're doing pen testing, or if you just want to monitor what's going on in your application. It's a powerful way to do it.

Jeff (15:36):

What I want to find out is about access control in this application. So when I click on this Access A, you'll see that we've started to build an access control matrix. Here, this shows that inside Access A, we're checking to see if the user has Role A associated with their account. Now access control is traditionally incredibly difficult to test correctly. You either need to spend weeks analyzing the source code to find out if all the checks are in the right place, or you need to pen test and you need to assume each role and then test the entire application with that.

Jeff (16:12):

So we can do it much faster if we just cheat, and we have the application tell us exactly where all the access control checks are. So here, I'm going to click on these other methods, these other pages here. And very quickly we can see the access control matrix generated automatically through instrumentation. So we can see Access A checks Role A, Access B checks Role B, Access C Role C, Access D doesn't have any access control check. That's a little concerning. Every page should probably have some kind of access control check. Access E here has three different access control checks, which is possible, but a little unusual.

Jeff (16:48):

So I want you to imagine using this capability and generating this, then you can just look at it and see if the access checks are right. Once you have it right, you may decide to freeze it. And if you freeze your access control matrix, then you could even get alerted if anything there changes. So if a new page gets added without the right checks, you could get alert to see. We can also measure test coverage this same way. I've been using just the guest account so far here.

Jeff (17:16):

But imagine if I switched to User A and then I test A ... I'll do that for all these users. I'll just test A few each user. So we're just keeping track of what pages have been tested with what roles, or with what user accounts actually. Sorry. And then, so you can see here, there's a big gap. We haven't tested with a number of different users here. So again, it's another way of looking at how good your testing is.

Jeff (17:45):

Along the way we gathered information about encryption. So you can see here Access A, B, C, and E have encryption algorithms associated with them, but Access D doesn't. That's potentially concerning if you know Access D is processing something sensitive, you might be interested in knowing that it's doing the right encryption. You can also see there's a bunch of different encryption algorithms here. That's generally not a great idea. You probably want to standardize on one. That probably should be AES rather than password based encryption or DES or something weak like that.

Jeff (18:21):

So look, we've gathered a lot of information here about this application automatically just by using instrumentation. So let's go back to the slides here. I want to show you one more demonstration. I think this is really ... Here, this is just a summary of what we just did. I want to show you one more. I want to show you how you can prevent vulnerabilities from being exploited with instrumentation. So in this case, we're instrumenting the service method in Java. That's sort of the entry point for all web functions, and we're just going to enter scope, which means we're just keeping track that we're inside a servlet.

Jeff (18:58):

Our rule is that we want to prevent any operating system processes from being started while we're inside the service method. It's okay if they're built in the framework or something, but if we're inside the service method, we don't want to start any native code. So that's a possible rule that you might want to implement. And here, we instrument the same way. We're just adding the sandbox advice to the ProcessBuilder. And so here, if we enter that ProcessBuilder method, we've added this code into that method with instrumentation, and it just checks to see if we're in scope, then we're going to block that process from being started. We're doing that by throwing the security exception.

Jeff (19:42):

So this is easy to demonstrate. I'll switch to the sandbox agent. You can see here, I just set this environment variable, and then we'll just run that same application the same way, and with that start up. And now in this application, there's a command injection pages is vulnerable to command injection. We don't have to exploit it here. We can just submit. And you'll see that this was prevented because of this security exception that says, "Hey, we're trying to start a process from within the ServletRequest process."

Jeff (20:20):

So, this is a powerful defense against command injection, which is one of the most critical vulnerabilities. This would prevent the stretch exploit from a few years ago, which was basically an expression language that caused a native process to start. So this is quite a powerful sandboxing of a critical capability within the application itself.

Jeff (20:42):

A little bit to sum up here, I want to talk about the difference between doing security sort of the traditional way outside in, and this modern way that I'm proposing, that's talking about doing security from the inside out. Traditional way, you feed your code and your libraries, and some tools, and the output comes out. And then you need a lot of people involved to triage the false positives, to correlate the findings, to create tickets. It's slow and burdensome, often it takes days or weeks to get feedback into the pipeline. And then you have to repeat that at various steps through the pipeline. So it creates a lot of bottleneck, and ultimately not a lot of code comes out the other end or it's slowed because of this burdensome process.

Jeff (21:29):

And then all the feedback comes back in the form of PDF reports that are full of false positives and generates a big backlog. So this isn't a very efficient process for writing secure code. We can do a lot better with instrumentation. So now, instead of just analyzing the pieces separately, imagine we analyze the entire application, put together with some instrumentation, we'll add the instrumentation. And now when we feed this application into the pipeline, we can get instant feedback on vulnerabilities as part of our normal testing. And we don't need experts involved in this process.

Jeff (22:07):

And then the rest of the pipeline can repeat those same tests. So as you exercise the application in your test process, whether it's canned testing or human testing, all automated, and then we get all the way to prod and in production, we can watch for attacks and notify the people that need to know about attacks instantly all with a fairly, with instrumentation to do this. That'll enable us to generate much better flow through the application as we're generating security flow as well.

Jeff (22:41):

Ultimately, that's the secret to DevSecOps is getting this process moving with the three ways that I talked about with getting security flow, tight security feedback loops, and this will allow a culture of instrumentation, of innovation and learning in our DevOps process. So I don't think application security is hopeless, even though it hasn't been making progress in the past. I'm not saying that technology alone can solve our security issues, good security, the mix of culture and people and process and technology.

Jeff (23:18):

But with instrumentation, we can create a platform where development and security can finally work together. If you want to try some powerful instrumentation, you can use our community edition. It's free for one app, full strength instrumentation to find vulnerabilities, protect you against attacks and analyze open source. So with that, I'd love to take a few questions.

Lauri (23:41):

So Jeff, here is one. Let's see how much we have time. We have three minutes. If we are fast, we probably can run through one or two. Is there a way to use a static code analysis to find security vulnerabilities instead of doing instrumentation?

Jeff (23:56):

Yeah. So static code analysis can find vulnerabilities, but the problem is it's not very accurate. So it finds tons of false positives and that requires experts, which ultimately that means it's going to slow down your process, and it's kind of the opposite of what we're trying to do here. So instrumentation is a newer better option than static, which has been around for over 15 years and it's not getting better.

Lauri (24:21):

The second question, what are your recommendations to left shift security culture in big organizations where feature development is the primary priority? In parentheses, lack of time for feature development.

Jeff (24:33):

Yeah, that's exactly the point of this talk is to use instrumentation to enable development teams to do their own security testing, find their own vulnerabilities and fix them and check in clean code. That's not something that's easy to do with other kinds of tools because they're inaccurate. And if you're inaccurate, you need experts and if you need experts, it slows down the process. If it slows down the process, then you're impeding feature development. So, it really comes down to getting better telemetry from your application.

Lauri (25:04):

What's your view on automated security for tools like instrumentation and other ASD tools versus manual security testing by security experts?

Jeff (25:12):

Yeah. There's always going to be a role for manual security testing. I spent many, many years doing that. I've found tens of thousands of vulnerabilities that way, but it doesn't scale very well and you need experts. So it doesn't scale in terms of time. It doesn't scale in terms of people. We really need automated security testing and it has to be accurate in order to scale. Instrumentation is a way of getting there. It's considerably more powerful than sort of legacy techniques like static analysis and dynamic analysis.

Jeff (25:46):

So I think this can shift the level between what we can automate and what we can't. I encourage you to go past just seeking the known vulnerabilities, like the things in the OWASP Top 10. I encourage you to get to where you're testing your own security defenses. So I showed you how to test an access control matrix. Well, you can use that technique today to do things that have historically required a security expert to test. And so I think this technique makes it easy enough that you can create your own security tests for things that historically haven't been automatable.

Lauri (26:22):

Well, whenever somebody says, go and try it out yourself, I don't know about you, but I feel the urge to jump right into it. And you should really take the opportunity if you feel that way for yourself. Next time, we're going to hear an interesting talk from Google. Our next speaker is Abdelfettah, Strategic Cloud Engineer, who will talk about Knative and Cloud Run. His topic is called How Creating Abstract Layers Help Remote Teams To Deliver Results.

Lauri (26:54):

If you can't wait until then, you can always go online to watch a video recording on his speech. You can find the link at the show notes. Anyway, I'll talk to you next time. Until then, remember to get the process moving with security flow, with the security feedback loops in place and nurture the culture of innovation and learning.