In this informative lunchtime session at Prodacity, Arian Salgado, a solution architect at Red Hat, delves into the intricacies of DevSecOps with a focus on secure software supply chains on OpenShift. Amid the backdrop of recent cybersecurity challenges, Arian unpacks Executive Order 14028 and its implications for secure software development practices, emphasizing the need for Software Bill of Materials (SBOMs) and continuous monitoring. Through a detailed demonstration, viewers gain insights into the operational dynamics of a secure DevSecOps pipeline, showcasing how Red Hat's OpenShift facilitates compliance with stringent security standards.

So good afternoon everyone. I am Arian Salgado. I am a solution architect with Red Hat, and I will be your lunchtime entertainment. So for a little bit while y'all are munching away... Hey, thank you. Thank you. While y'all are munching away, I mean this is supposed to be a lunch and learn. So instead of me sticking with the higher end topics that you guys have been hearing today, instead I'm going to go down a little deep technical, not too technical, I'm still going to keep it very just over the top, but just go over some of those concepts that you guys keep on hearing about in case some of y'all might not actually know what it looks like or what it's supposed to actually be. So we'll go over some of that.

So essentially I'm going to cover DevSecOps with secure supply chain on OpenShift right now. This is not necessarily an OpenShift presentation. This is really more DevSecOps with secure software supply chain. Which I will speak about, it's actually pretty important, especially with the way we do business today.

So today I'm going to cover Executive Order 14028 and the company memos that it comes with. I'm going to go over what is a standard DevSecOps pipeline, and then what is DevSecOps with secure software supply chain. And then I'm going to go over a little demo. The demo was supposed to be live, but we have some issues. So we do have a video of the actual pipeline itself, and I'll go over that as we go through it.

All right. So Executive Order 14028. If you're not aware of it, it's a big deal. When that came out, that's really what added the fire to the fuel for actual software factories. You guys hear a lot about Kobayashi Maru, you guys know about Platform One. All these software factories that are coming out, a lot of them came out just specifically because of this, or really they were already in place and they really picked up after this came out. So as you can see, it charges multiple agencies including NIST with enhancing cybersecurity, developed guidelines which ultimately aimed at US federal agencies, but which are available for industry and others to use. So it does lay out to where you have these software factories that actually are open for everybody. They're not locked into whatever Service is using them and they're widely available. And then doing business with US federal agencies will require SSDF compliance, and that's what we're going to be talking about here today.

Now, there is accompanying memos. This is where it talks about, it emphasizes the need for software bill of materials, or an SBOM, and we're going to go over that as we go through the presentation. And it underscores a significant of continuous monitoring and iterating. So you guys heard a lot about the cATO going on. It's been spoken about here throughout the day. This plays into that, and we'll talk about that as we go through. And then agencies must only use software that is provided by software producers who can attest to complying with government specified minimum secure software development practices. So again, this is very important because we all do business with the DOD and doing that we need to be aware of this and be able to support this in its entirety.

All right, so what is a big deal with secure software supply chain? Well, it's a really big deal actually, as you guys just looking at the slide really quick. That's just one of the many cases that have happened here recently where there were attacks in the DOD networks, DOD infrastructure from packages that were included with software that they considered to be safe and easy to use. So once that came about, that's where this specific 14028 guidelines for SBOMs, for secure supply chain, all that stuff, all that stuff came about specifically because of these kind of attacks.

Now, speaking about SBOMs. You're going to hear that a lot. So what is an SBOM? It's a software bill of materials. Quite simply it is the list of ingredients of a software product. So essentially in the DOD, whatever software factory you're using, when you're bringing in an external product in, you have to make sure that everything that's actually in it is supposed to be in there. So for example, with Red Hat, if you consume a Red Hat product, whether that would be a UBI image or whatever the case may be. We sign it and provide the SBOM with it. And then that way you take it in and you're able to verify that what's supposed to be in there is actually in there and nobody modified that somewhere in between the chain between Red Hat and the DOD. So same thing.

That's where SBOMs become really important, and that is now a necessity. It used to be kind of an afterthought. It'd be great if you had it, if you don't, okay. It wasn't really part of DevSecOps in general, but it is now, and that's all because of this executive order that's come out.

And this is what an SBOM looks like. As you can see, most of it is human readable, but it's a lot of information, and that's just a quick little snippet. The simple app that I run for the demo, it's over 3000 lines for the SBOM. So it's a lot of information and you need to be able to not only read that information, but recognize what's supposed to be there and what's not. And so that goes a little further down the rabbit hole then I will cover today, but that's a whole other conversation for that. But at a minimum, you need to have an SBOM and you need to know what you're looking at and how to scan it and how to view it.

All right. So along with SBOMs, you have attestations, an authenticated statement, metadata about a software artifact, a collection of software artifacts. So an attestation is another document that tells you that yes, this SBOM that you receive with the signature that you have is good to go because I have all the metadata for it that matches. It's not a reviewable document. It's not something that you can just pull up a file and be able to look at it. It's something that has ingested by whatever system is actually reviewing your signatures for your SBOMs and your images and things of that sort. But this is now also a requirement. So not only do you need DevSecOps pipeline, but you also need SBOMs and you also need an attestation.

And then now simply, what is DevSecOps? This is a standard DevSecOps pipeline. You guys have seen this many, many times before. This in particular. I mean, this is what I kind of run on my environment. And so it's very simple without getting too deep into the details. You got your code that comes in, does a quick unit test. You got code quality from SonarQubes. That's kind of the static code analysis to make sure that whatever code is in there, there's no mistakes that are going to mess anything up. You got your artifact repository with Nexus. And then you got your security suite where you do your image scans, your image checks, and your deployment checks. That's where you check for CVEs and anything like that. Then after that, you do your deployment. That's part of your CICD and then your pen testing, and then your performance test. This is the standard basic DevSecOps pipeline as we know it today. Nothing major changing here recently. It's been like that for a little bit.

But now it is changing. Now this slide is a little crazy, but it's really not. It's really the exact same thing. But we've added now particulars for the secure software supply chain. And so what you're seeing now is, so you get your developer that makes a change in the code, comes in, does its code test. Again, same thing, the static code analysis and that sort of thing. Your rootless image build. And then after that, that's where you start seeing the differences. That's where you're seeing the signature, the store and the signing of the image. And you'll see that it does ACS, advanced cost of security, whatever security suite you're running, it's going to do your vulnerability scans, and it's also going to do a signing of it. So it signs the image right there.

Then after that creates and assesses the deployment. So again, another scan, but then also sign the deployment after that. So again, not only do you sign the image, but you also sign the deployment. The part that you don't see in there is the creation of the SBOM, but I will show that as we go through the demo. So it'll go through, it does its internal test, and then it actually does goes out there and promotes to prod.

So again, it's the same pipeline, but now you're adding signatures, you're adding attestations. So now your image and your deployment are both signed and you got an SBOM that is also signed, and you got your final attestation that says that all those signatures and all the things that you actually brought in and ran through that pipeline are actually good.

Another part that you don't see in there is, so down at the bottom left, you see the attest pipeline task. That's another part of all this. So yes, you sign your image, yes, you signed a deployment. Yes, you got an SBOM that's also signed and you got your attest station. Great. Well, how do I know that whatever. Did all that actually did it the right way? Well, now you also have to attest for your pipeline. So the pipeline attestation essentially is a blob that has the information of what the pipeline is and what it did. How did it check for those signatures? How did it sign? How did it store those signatures? So it does everything. So not only do you have that information particulars to that particular app and its build, but also the pipeline that you used in order to run all those tests and be able to actually put that out.

So it's essentially who watches the watchmen. So you got to make sure that not only is your stuff safe, but is the stuff that's watching it safe. So that's what it comes down to.

All right. So again, demo, it's going to be a video, but it's the exact same thing that we usually go through. It's going to be an OpenShift. I'm going to have a simple application. I'm going to make a quick change. And then I'm going to walk you through the steps as it goes through the changes and just show you what it's supposed to look like. So it's a little peek behind the curtain of what developers and administrators actually see when you're actually putting these different things into effect.

All right. So we got a quick spring pet clinic application. You got your cute little puppy and cute little cat. Everybody loves one or the other or both. So you see the welcome message. So me as a developer, I want to go in and I want to modify that welcome message. So I will go into my code and I'll make the quick change, and then I'll go down and I'll commit it. Never commit to the master branch. I know if you're a developer here, that's big no-no. But just let you know. So committed it. And automatically it kicks off a pipeline build inside of OpenShift.

So this is what your pipeline actually looks like. Now, I am showing you OpenShift, but this can be any flavor of Kubernetes that you may be running. It doesn't really matter. And everything that I'm going to be talking about, it's all open source. So whatever platform you're using, whatever tools you're using, you just got to make sure that you're hitting all the things that you need to do.

So first you get your source clone. Now it's going to go through the standard static code analysis unit test, all those things that we talked about earlier where I mentioned it was like SonarQube. So it's going to go through that first. Then after that, it's going to release the app, and I'm going to show you what it does after that. So release the app, signs the image. Then after it signs the image, you'll see right here... Oh, I'm sorry, it creates the SBOM. Then it goes through.

Now here, this is where it starts getting into it. You got your image signing, which we spoke about before, your image scan, check and deployment check. Those are done by ACS, which is our security suite. Then after that, we're going to sign the SBOM. It's going to update the deployment. This is the CD portion of CICD, and then it's going to go ahead. Once everything checks off, then it'll go ahead and actually launch a deployment. So this is just our code modification that we did going through making sure that everything's right, and then it'll launch it.

Now what that looks like behind the scenes, what is that actually made up of? So our pipeline is made up of, first of all, a co-sign pod. So that is the part of the tool that actually does the signing for us. It does Shaw 256. It works off of a private key infrastructure. Then we got our webhook, which is assigned to that actual code, our repository. So once we make a change, the webhook launches that pipeline automatically. Our GOG server, which is just a regular repo server, it's very lightweight and it's just used for this demo. SonarQube for a static code analysis. Then we got Nexus for artifact repository. And then the reports, and I'll show you the reports here in a little bit.

So here's the advanced cluster security. I'm going to rush through this because it's a video. So advanced cluster security, so this is our security suite. Again, whatever you have, you just got to make sure you mint all the widgets. The thing that I wanted to point out here is with our security suite, it does a lot. It does scanning. It goes really in depth. But I'm going to show you in particular why this is important. Because it's great that you have signatures. It's great that you create an SBOMs and attestations. So what does that mean as far as your actual platform?

Well, here as I scroll down, you see right there. Trusted signature image policy. So that makes it so that if a signature on an image does not match whatever public key I have stored in my security suite, it will not allow it to launch at all. And so that's the kind of thing that you want to have. Again, yes, you provide the signatures, you provide all those checks, but now what? You got to make sure that it doesn't launch in your system if it doesn't match for whatever reason.

So we got all that going. So it's releasing the app, and then you'll see right there after release the app, it's got the SBOM. So the SBOM for here, for this particular build, I'm using Anchore Syft, and I'm actually going to show you guys what that looks like as it's building it. So you'll see it come up here in just a second.

And I do have this demo actually live. I have it fully up if y'all want to see it afterwards, and we can slow it down and actually go into the details.

So here's the Syft scan. Now, it's going to take a few seconds because like I said, it's 3000 plus lines of code for the actual SBOM, but you're going to see it actually populate as it goes through. There it is. And it's an SPDX format. There's two other formats out there right now, but for this one we use SPDX. So there it is. And like I said, it's mostly human readable, but still you want to have some kind of tool or application in there that brings it in and actually compares it with whatever your customer site requires.

So once that's done, you got your SBOM now. Now we're going to build the image. Now take a look at that. So that is our pipeline code right there. That's our pipeline name right now. So with that, remember I mentioned about reports. So we just go to reports. Now you've seen the information from that SBOM, and really just simple tools. You can go in there. I'll find my pipeline here. Yep. There's my SBOM, it's saved. Put it in a safe store. So that let's say whatever customer happens to be using something similar to this, they have somewhere where they can pull it up really quick and be able to really easily read it and also easily digest it for whatever system they're going to be using.

This is the other part of those reports. So we got dependencies. We can go really in depth into what is actually being built here, what are their dependencies, what are their versions and anything of that sort that you might need. So again, it's this security stuff is getting really granular when you actually get down into it.

All right. Let's go back to our pipeline. See where we're at now. Okay, great. So we got our SBOM, we signed our image. Now it's signing the SBOM. Right boom. Now here's the thing, now that we signed the image and now that we signed the SBOM, what else do we need? The attestation. So you will see there at the bottom that we have the attestation, and it's also got the Shaw 256 signature. It's also got the signature for the image, and it's also got the signature for the SBOM. So it got that all created and also created the attestation for it after that, and I'll show you where that's at here in just a second. So signed image, signed SBOM, update the deployment. So now we're going into the CD portion of the actual application launch.

Okay, actually, that's pretty much it.

Oh yeah. So our registry. We signed it, we signed the image, we signed the SBOM, we got the attestation. So how do you know where everything's at and that it's actually good? So here's our registry. You can see there's our latest tag for that particular image that we just created, and you'll see those little shields that let you know that it is signed via co-sign. So now you have that, and right above that was the attestation.

And then that's it. It's going to finish up. And then I'll go to the website. I'll refresh it a couple times and then you'll see welcome from Red Hat come up. So I know it's really quick. I know it's a lot to really go over really quickly in such a short amount of time. But essentially this is just to give you a quick snapshot of what your administrators and your developers are seeing when you're dealing with DevSecOps with secure software supply chain. What does that actually look like? What does an SBOM look like? What is it supposed to be? What does signatures look like? What does an attestation? So that's the kind of thing that we're covering here, and that's where it's at.

This is me freaking out because it didn't come up immediately because I'm very impatient. So I'll refresh it a few times and then it'll come up and you'll see, welcome from Red Hat.

But yes, this is what it is. Any questions on any of this? I know I covered a lot very, very quickly, but any questions on software supply chain, DevSecOps, anything like that? There it is. Welcome from Red Hat. So that's essentially what happens behind the curtain. Whenever your developers go in there and they make a change, this is a very, very basic pipeline that actually takes place. Of course, depending on your environment and your requirements you're going to add more to that. But this is at its basic core, this is what you're really looking at. So when you hear about secure software supply chain, when you hear about any of these other concepts, that's exactly what they're talking about. That's exactly what it looks like.

All right, thank you very much. Enjoy your lunch.

‍