At Prodacity, Lloyd Evans, Director of Governance Risk and Compliance at Aquia, shares invaluable insights from his experience as the Principal Information Systems Security Officer for a platform as a service within the Centers for Medicaid and Medicare. In this talk, Lloyd compresses two years of groundbreaking work into an informative session, highlighting strategies for achieving efficient security control inheritance.

Lloyd Evans (00:09):

All right. Hey everyone. I am Lloyd Evans. I'm the Director of Governance Risk and Compliance at a company called Aquia. For the last two years, I've been the Principal Information Systems Security Officer for a Greenfield platform as a service within the Centers for Medicaid and Medicare. And before that, like I'm sure 70% of this audience, I worked at Kessel Run for a little bit. So, I am here to compress two years of work into about 20, 25 minutes, and then hopefully have some time for some questions afterwards. This talk gets really tactical, so if you have any questions on specific things, let me know and yeah, I can go through it.

(00:45):

All right, so let's kick this off. As I go through this, so this is a lot of stuff and I'm going to refer to my notes because I have a specific thing here. So, there's a lot that we're going to cover. My goal here is for you guys to take really just one thing from this presentation that you find useful, that you can use tactically in wherever you're working at, and then hopefully just experience a different perspective. Working in CMS was relatively new to me. My background, I'm an Air Force Veteran and came up through the DOD and worked at Kessel Run, so this was a lot that was new. So, if you can take one thing away, then I have succeeded.

(01:20):

Tying things back to the mission, which has been a central focal point here at Prodacity. So, the CMS mission is huge. I have a bunch of words up on the screen to describe tangibly what that looks like for me, my story down to reality. My fiance is a mental health therapist that works with kids and walking her through the one time she asked for me to help was to set up her portal for insurance, so that she could process claims. And so, I had a little shoulder angel that was telling me, "Hey, refer to all the hard Air Force stuff that you worked on." And I had empathy for her as a user, but it really opens my mind to I think coming up to the Air Force, we can get really comfortable with the status quo of things and even small incremental improvements can feel like massive leaps forward, and they are.

(02:08):

But at the end of the day, it really comes down to the actual user, the person who's using it, who's trying to run a business, who doesn't really have all of the time that a working service member might have to load up Office 365 for about 20 minutes while they're getting coffee. So, that's what is really important to me, and CMS is huge. So, there's a lot of really, really good work. There's about 200 disparate applications and the really forward-leaning CISO there, Rob Wood had this idea to pull together a platform as a service to reduce a lot of the compliance burden from the ATO process for these applications.

(02:44):

He has a tattoo of Batman on one of his arms, and so he named everything DC and Batman-themed, which is super cool. And so, we partnered with them there at CMS to build a Greenfield platform as a service. Here's a bunch of language describing that. What we're going to focus on, if you can piece out one thing here, is maximizing security control inheritance. So, without further ado.

(03:10):

So, here's how we did that. So, one of the things I want to call out is our outcome for this. So coming in, we wanted to first understand the ATO process as it is for CMS, so that we could alleviate as much of it as possible for our users. So, we put our platform through the entire ATO process and we were able to achieve 74% control inheritance for the acceptable risk safeguards. That is a CMS custom overlay of NIST 800-53, that brings in all of the different privacy requirements that they have as an organization under HHS, and base them into 352 controls for a moderate system that breaks down into 782 individual control elements. And so, tying our platform stack as we were building it to the actual controls, and I'll get a bit more about how we do that through talking tool-centric versus control-centric, we're able to build an inheritance package that encompasses everything that they inherit from AWS as our IS provider, everything from our IDM provider, tooling that we use as a part of our platform, the pipeline infrastructure - all of that bakes into this control's inheritance goodness that they have.

(04:20):

And it was really cool listening to Rob's talk earlier for the VA where they hit 70% because I was like, "oh, that's crazy. I heard nothing of what you guys were doing and we arrived roughly at the same spot." And it's interesting too where it's like a lot of that remaining 26% at least for us, is a lot of paperwork to be honest. So, we're kind of also trying to figure out how we can maximize our inheritance further to kind of dig into some of those aspects right now. So, how we did this, and this is one of the key things if you want to take away, one thing is that I've seen a lot of people in the GRC space focus on control-centric approaches where you kind of come into this situation, you look at your 352 controls and you're like, "oh my gosh, where do we start?"

(05:02):

Let's go through access control. And then by the time you get to the second control family, your whole security team's burned out, and you're just kind of just going through as fast as you can. I've seen this happen in other organizations, seen it happen before. And so, what we did was we actually started with our platform stack and the tooling as our first way of going about this. So, as we were building this platform starting from our Greenfield environment, we were using something called Architecture Decision Records. And so, for your IDM tooling, it was like I took this one and I didn't take any risks. So, we chose Okta for our identity management tool. And so, once we had each of our tooling stacks approved for various parts of the platform, then we would go through as a security team and then map that specific tool and capability to the 352 controls that we would think mapped to it.

(05:51):

And so, that's where you can see this small little diagram here. This takes a little bit of time where you have to just guess of, "hey, I think it maps to most of the AC family, probably some of the IA family, and maybe a few others for Okta," for example. And then from the security side, we then formed together as a team, did a second pass, validated some of those assumptions. And then for the third pass, we really got closely coordinated with our engineering teams to work with them as far as like, "hey, we're writing a statement for SSP that we're going to hopefully leverage for automation in the future. We need to make sure the system is accurate or this statement is accurate."

(06:31):

And so, that was a lot of working with our engineering teams, getting buy-in from the time investment because they're also focused on actually developing a platform and making sure that those statements were in line for SSP because we're going to get to a lot of the benefits of doing things that way in a bit. And then we'd do a final wrap-up, and then that is how we built, I'd say probably 70% or 80% of our SSP was just going through our whole tooling stack, mapping those to security controls kind of, I guess, I don't know what the word for it would be, grinding. Grinding down to a really good statement and then filling in the cracks from there. And so, that's how we built our initial SSP for the platform.

(07:11):

So, what was the outcome of that? We had a flawless SSP through our assessment and one of the fastest ATOs in CMS history. We had zero findings during our ATO assessment. And you can see in quotes, that was a direct quote from our lead assessor. We had one low POA&M for the entire ATO in the timeframe that we did this. I was actually telling some people yesterday this and they asked me, "How long did it take you?" And I was like, "I don't know." It was just, we were told by the organization we were the fastest.

(07:38):

So, I looked through my calendar, pulled through some dates and it was three months, 23 days from the official start kickoff of our ATO process to assigned ATO. And then we had a little bit of lead time developing the platform before that. So, it was about eight months, about nine months from our first control statement written to our first app in production. And then we've since fielded an additional four apps as a part of our authorization boundary and then one big application actually from another organization that we've completed onboarding into production. And that's something that I will dive into as well.

(08:09):

So, here's some tools that we used to kind of build this. One of the big things that I'll point out here is just self-awareness. So, there's kind of like a dichotomy with compliance at the very least where on one end of the dichotomy you have compliance of "why do we have to do this? These things are awful. There's a specific control in your SSP that says that you need to prevent split tunneling or something like that, but we don't even use VPN, so why do we have to do this? We have that math in our inheritance model." And then on the other end of the compliance, you have people who take compliance as the end all be all arbiter of the only thing that we should be doing. And as long as we're doing that, it doesn't matter if we have no users because we're doing compliance really well.

(08:54):

And so, just self-awareness of those two, right? Because interfacing with engineering teams, we're interfacing with our customers. And so, we try to maintain a good blend of saying like, "Hey, yes, we're asking a lot of time from our engineering team to put together this SSP, but here's why we're going to do it and it's painful, but all of our customers have to go through this process and our major job here is to alleviate the burden of this process from our customers. So if we don't really understand it, then really what use are we going to be?" And then also we'll get into a little bit of using compliance as a tool to market to customers because that's the language that they understand, so.

(09:34):

I'm going to skip to our roadmap a little bit because I think the lessons learned are more important, so I'm going to end strong on those. After we kind of had this four, eight month, whatever you want to call it, compliance and MVP security journey, we actually broke out our team from there into purple teaming, compliance automation, App Sec, and security engineering. We have structured our security team around these four things and we've done so because we're able to produce this very effective SSP, we're also able to use that to have the basis for how we're automating certain things and the evidence collection and be able to grow ongoing authorization within CMS and work with their ongoing authorization team as well, to be able to build that capability within CMS and support it.

(10:19):

There's a few things that we've done here as far as doing security alerts and detections as code. We've done a standards' library internally of like, "hey, we have this SSP, all of these great controls, what does that realistically mean as far as security standards for onboarding?" And that's actually another key aspect of how we're able to lift a lot of the compliance burden is writing automated checks for those standards, and then we can just check those as part of the process. And if all of those come through green and we can kind of dig through a little bit, then we can onboard that application much quicker. So, for our specific environment, for our platform, we can onboard applications as a part of our authorization boundary. And that is called I guess a Security Impact Analysis Process within CMS. And we've been able to reduce the timeframe from that from two to four months, down to two to four weeks.

(11:12):

So, one of the things as far as those breakouts, I'll really only touch on one here, which is the application security piece because this is kind of tying things back to control inheritance. There's a specific control under the risk and something family RA family, RA-5 that deals with risk analysis for FISMA systems. So, one of the things within the Federal government, I'm sure some of you who are familiar with this space might recognize this, is that right now severity in the form of CVSS scoring. So, you're kind of zero to 10, low, moderate, high critical vulnerabilities is the standard use. And that's where you hear things like no higher critical vulnerabilities within your systems. We've expanded upon that and baked into our pipeline not only CVSS but also including EPSS, which is the other half, very simplistically speaking of the risk scoring. So, I think within the Federal government using CVSS, there's a lot of equation of severity to risk. And if you kind of break down the components of risk, severity is only one small piece.

(12:18):

There could be I am terrified of open water and for whatever reason, I don't know, I grew up in the desert, so terrified of the severity of a shark attack. I swim in lakes though, so the probability of that shark attack is extremely low. Some would say zero. My internal body says it's above zero. But I would still face, in this loose analogy, the same regulations swimming in a lake that I would in the open ocean. And so, one of the things that we've brought into play is EPSS, which is a new data-driven model for vulnerability analysis that brings in that predictability scoring of saying like, "hey, this specific vulnerability, yes, well it may be highly severe, also virtually impossible or there's a specific number tied to that." And we've done a ton of interesting analysis vulnerabilities across our platform and you find out that 90 to 95% of vulnerabilities just aren't exploitable or very, very low probability chance of exploitability.

(13:21):

And so, this has really helped us have security control gates in our pipelines that prevent high-risk vulnerabilities, but at the same time don't halt developer velocity to the point that we're like, "hey, we have this great pipeline, it's all these security tools, but we can't push anything to production or it's a huge pain, and so we're not going to use platform." So, that is one of the things that we've been able to do. I'll kind of skip over a little bit of analysis because I really want to focus on the mindset shift piece of this. So, I think if there's three tools that we've been able to blend together to make this work, it's leadership, I'd say an understanding of that dichotomy of compliance, and then tying that compliance to the actual engineering that's going on. And the mind shift piece or the mindset piece is where leadership comes into play. So, change is hard, really hard. And when you have a body of security personnel who are very used to doing something in a certain way, and then you come in with this fresh new platform, it's really cool.

(14:29):

We have a bunch of Batman-branded stuff and it looks really cool and people are really interested, and then you come on to getting applications onboarded, that's when people say, "Oh, well you're saying not to fully use CVSS, does that mean you're hiding vulnerability?" There's a lot of apprehension to new ways of doing things. And I think that that's one of the immense powers actually that we've been able to leverage out of doing compliance in this particular way is that we've actually, going back to Marina's point or in one of our earlier topics, of using the bureaucracy against itself. Because we spent so much time building the platform, tying every little thing that we were doing to the compliance and then our SSP, which goes down into the controls that they inherit, we are able to almost sell a new way of doing things through the way that they're familiar with.

(15:21):

And so, when they're saying, "hey, this thing is unfamiliar," we can actually say, "well, we've actually assessed this through our SSP, the one that we had flawless results for. You don't have to worry about this, but we also have all of the compliance documents that you need to feel comfortable with this way of doing things." And that's been one of the biggest leverages that we've been able to use as a sales and marketing mechanism almost, is that when we're going to our customers, we're not saying, "hey, trust us for all of this new way of tools and doing stuff." We're saying, "oh, hey, we do things a certain particular way. We're going to onboard you. If you want all of the compliance receipts, and in a language that you understand as a customer, here you go. They're free to go. You can read through all of our statements, here's how we do things." And then we work really, really close with our customers to be able to walk them through these specific things, this vulnerability management process being one of many things that we've done.

(16:23):

There we go. Cool. Branching into leadership. This slide really just explains that. I read Extreme Ownership by Jocko Willink about five years ago, and I have used all of those tools and tactics to, I don't know, I guess lead where I am. And that's been one of the major, I would say driving factors for success is like I see compliance as almost a... because it's so often a constraint in the organization, it's a perfect place to lead from, where you can say like, "hey, RA-5, the control language says a specific thing, but we can attach that to a new way of doing things and then write these SSP in such a way that reflects that," and then lead our customers through this journey. A lot of that involves teamwork, intentional communication is huge with all of the stakeholders, understanding where each piece of the puzzle is coming from, and relating to them where they're at. One of the one things that I want to point out of this is allowing yourself to be taught, so you can teach.

(17:35):

I have sat through tons of presentations of things that I've heard multiple times before if only to have the specific person teach me how they're doing things, so we can start to build that rapport and reciprocity. I see a lot of people mess this one up where they immediately want to come into a new situation and say like, "hey, I have all of these new ways of doing things, GitHubs, pipelines, let me teach you how to do new things." And from what I've seen is that kind of builds innately this resistance of, "whoa, my understanding of where I'm at. Maybe I've worked 10 years in this position. Who are you coming in fresh of six months to teach me a whole new way of doing things that also not only I don't really understand and I'm kind of afraid of," as we were talking about earlier, "but also removes the power and my understanding of where I've grown my career in the last 10 years." And I think if you kind of come forward with a position of learning first and really trying to understand the person, empathize where they're at, and then maybe take one thing of just saying, "hey, what are you doing with vulnerabilities? Have you heard of this other thing?" I don't know. I've given a couple presentations of EPSS kind of broadly of just like, "hey, here's this thing." And I've seen that that has a lot of return benefits. So yeah, I don't know if I explained that one well, but I think you guys get the point.

(19:02):

So, lessons learned, it might be crazy to say this, but compliance is a really good marketing vehicle to federal organizations, and it's kind of the way I've seen it now. I've also read enough about marketing and on the business side to know that the easiest way to market something is just have a really, really good product. And I think that that's kind of the balance here is that if you're using compliance in a marketing aspect, but you're not actually doing the compliance aspect, then you're no different from an Instagram drop shipping company that's just lying about their brand and trying to sell you some stuff that they've marked up. So, getting things right and using it as an effective vehicle to communicate what you're doing is really powerful from what we've found. So, another thing we've learned, there's a lot of processes that are gated by institutional knowledge, just all over the Federal government. Pooling understanding, and then just sharing it out like, "Hey, here's what we've learned."

(20:00):

Gathering all the institutional knowledge into documents that make sense for gathering knowledge and then just publishing that out. That's something that we've done as a platform that's really, really helped us both build trust with our customers and also just for us to understand and be able to communicate certain things as we understand it. And then final thing, change in the unknown or difficult concepts. And I think something I haven't mentioned here specifically, none of this has anything to do with AI. And then if you're familiar with OSCAL, I don't mention any of that because I think what we've learned is that communicating in the language of our customers is really what enabled us to succeed. And none of those other new fresh and unknown things were the language of our customers. Now, in the future, and we're kind of looking at this, will we bring on those things and then communicate them through similar language, probably. But that's not where we started. And so, we started where our customers were.

(20:58):

And then a bonus...titles, how we named things CMS Experience Report probably didn't give you guys the best heads up of what to expect for this particular presentation. I've come across a lot of landmines of how we name things in the Federal government, and you can even see ATO, does that mean authorization to operate or does that mean air tasking order? I think that we've run into a lot of, we've named certain things certain ways which have really freaked out customers sometimes.

(21:28):

And so, just being very conscious of that, of naming things in a certain way that also requires that understanding of the customer where they're at, and then what will resonate or not resonate with them because your prior understanding, or at least for me, my understanding of how the DOD operates, some of that transferred over to CMS, a lot of that didn't. And so, it was definitely going to be an error. And there were some mistakes that I personally made where assuming that all of it was going to transfer one-to-one was inaccurate. And I think that just goes back to listening and listening to your customers. So, that's it for me. I don't know how much time I have left, but if anyone has any questions, I would love to answer.

Audience (22:11):

Are those SSPs that you mentioned, are those produced manually, by your team or...?

Lloyd Evans (22:14):

Yes. So, we have limitations with our GRC system. So, there's a specific template actually that is the only way that you can bulk ingest controls into our GRC system. So, we run everything off of that Excel template. And so, one of the things that we're actually working on is how can we recreate that in such a way that actually ties into some of the automation pieces and evidence collection that we're kind of looking at, but still exports that specific template so that we can upload it to our system of record.

Audience (22:49):

Thank you.

Lloyd Evans (22:49):

Yeah, I get teased a lot for using Excel. It's like a running joke internally to the team, but I had the opportunity to chat with Marina for a bit and I was like, "alright that's what our customer...it's like the language of the people that we're working with." So, that goes back to the marketing piece. Anyone else?

Audience (23:17):

You said system of record, is that eMASS or what do you use?

Lloyd Evans (23:17):

We use a customized version of Archer that is called CFAX. So, actually it's got a lot of similar state and problems that you would experience with eMASS. Anyone else? Awesome. Well, thank you. I really appreciate all your time. I appreciate it.

‍