Matt Nelson (0:16)
I'm super excited because you know, this is everybody's favorite topic, right? Like the DoD source selection process. Every change agent wants to learn about that. But I am, you know, super passionate about making sure that we create a culture and our culture, is the collection of individuals and the individuals also have to include our industry partners inside of who we're working with. And you have to create a solid culture by selecting the right vendor from the beginning. And that DoD acquisition process sometimes is more like competition theater than actual selecting the right vendor. And I'll get into that in a little bit. But you know, this talk here is not going to provide any cures for figuring out how to like eliminate…
(1:17)
All right. All right so this is not going to, give you a cure on how to like pick the best vendors, but it's going to give you a little bit of tactics to prevent fewer bad things happening because of bad contracts, right? So that at the end of the day, that's what it's all about. Because if you think about it, we're going to, you want to avoid this scenario where this theater is happening inside of source selections. And let me get to like my definition of competition theater. It's like this outward appearance of competition, but in reality all the meaningful improvements of the source selection process aren't really happening.
(1:56)
And like, I'll take a step back and kind of just give you my, background and how like I have 15 years of DoD acquisition experience as an acquisition officer. And then during that time I probably led over 30 source selections and then I got out and joined industry, as the COO of Rise8. And during that time, like we probably submitted about 30 proposals of that time. So I have like both sides of the coin from looking at this problem and this challenge from the government and from industry. And so like now I'm actually back in the government really trying to champion like, hey, there's a better way for doing this. 'Cause we got to avoid the mistakes of the past so we can leave it better for the generation that's coming next.
(2:55)
But like why does competition theater actually even happen in the first place? And you know, one thing that happens and it happens to a lot of us 'cause we're all just human, we all kind of want to take the easy path for figuring out what the next source selection's going to be. And we just do like a copy and paste of what's always been done before. So I can kind of get into a little bit for context here of like what is a request for proposal? So a request for proposal is basically the DoD's mechanism for selecting who's going to be the vendor that they're going to work with for the next three to five years. And normally, this is how it works. Like inside baseball, inside of the program management offices, five years ago someone awarded a contract and oh crap, like in the next six months we got to, we got to recompete this thing 'cause it's ending. So what do we do? We go and look at the old RFP file that's in the server and what do we do? We just copy and paste, change a little bit but really, really not changing anything. And then we expect different results. Then we expect, oh man we're going to get like a more innovative approach, or we're going to change how we're doing things but we're not changing how we are doing things. So we got to like follow a rule, like the boy scout world. We've heard a little bit about this in you know, the talks this week of like we got to set up success for not like impressing your bosses and saying hey we're going to create this safe path for least resistance 'cause it's been done before and it's, we will meet our timelines boss, it's going to be great.
(4:34)
We want to impress the generation that's coming behind us and this like, we want to impress them with a really solid acquisition strategy that they can learn and use and improve on over time. So you're not just, hey let's just, let's just go with the status quo and keep it going because it's safe. And that's one of the root causes for why the DoD you know is not getting really good acquisition reform. But I would say the biggest cause of why the DoD is struggling with really bringing in innovators and really disrupting the industrial base that's, just fixed inside of, where we're at today and it's market research. So right now we just pencil whip market research and market research is these conversations that we're having with vendors today. Market research is like going on someone's website and looking to see if like, oh you're an IT company. Let me make sure you have a decent looking website if you're going to build like good quality software for me. Those types of things.
(5:41)
But instead like GSA has a, market research as a service that they have. So you can just, hey GSA build my market research report for me and what that is in practice is they send out this scantron in electronic format, vendors, firewall that they can do everything and then you get a report with nice bar graphs. You had no direct collaboration with the vendor that you're going to spend the next five years of your life with. But you check the box that you did your market research report. And then the other thing that really blows my mind is like, okay, maybe we'll just set up our own request for information. So this is like what everyone does inside the government for market research. It's like we say, "Hey, in about six months from now we're going to have a contract. So we're trying to figure out the industry so send us this 10 page paper and we'll review it and that's going to help us shape our like strategy moving forward." That's crazy to me because it goes against the FAR, 'cause the FAR, if you think about it, the FAR says we shouldn't hesitate to have constructive exchanges with industry.
(6:55)
So I define constructive constraints with industry as two-way conversations, right? Like you need to like have those conversations with industry partners all the time. It's not just, hey in six months we got to award this thing so I'm going to do an RFI check the box. Like even one of the biggest bureaucratic red tape policies DoD 5000.02, which is the framework of how you are supposed to set up a program of record inside the DoD states that you should push for technical innovation and a culture of performance. And if we have learned anything inside Prodacity in the last three days to set up a culture of performance, you can't just do it by, hey gimme a document of a plan of how you're going to do something for me in five years and just write it down and you know, I trust you that it's going to work perfect in execution. Like you know, I mean that's not how it should be.
(8:00)
How it should be is the competition starts at the market research phase. I mean market research is now, like I said, so one pro tip that we can do to really be effective here is like let's take these software development practices that we have learned to build really great products and apply them to building really great strategies for competition and for source selection. So you know, a story that that kind of, you know shows this double diamond is, you know I was the head of acquisitions for Kessel Run for two years. And one contract that we wanted to stand up was an engineering services contract. You know, and Kessel Run was all the hotness at the time. So we put out an RFI and I was like hey just write a 10 page paper about how you do good engineering services, which in naive, like probably could ask better questions. But we got 109 responses and my team you do the math that's over a thousand pages that we got to read. So my team read over a thousand pages of RFI responses. It took us about six weeks but we did cast a big wide net and we're following like you know a good DNF process and then we refined it down to the top 10, you know the top 10 of those, white papers got a call to get a one-on-one. And during those one-on-ones then active like real communication is actually starting to happen. Which is wild, looking back on it in retrospect, we could all that six weeks that we were reviewing those white papers and our eyeballs are bleeding. We could have had just one-on-ones at that time and it would've been less stressful for me and probably less stressful for industry, because you're actually just talking instead of just avoiding, having the human network interaction, right?
(9:59)
We are talking about how we got to set up these networks. Another network is a people network. We need to really be deliberate about collaboration. So back to that story, we now at 10 or now we've kind of like did, we've kind of defined who our like top industry talent is and we did these one-on-ones and now we're kind of are going to say, okay, based off these one-on-ones, how can we create a good strategy, a good competition? Well the FAR says if you do a small business set aside and use a FAR Part 8 contract, I know this is kind of wonky, but that will allow you to do a limited competition of just three vendors. So we're like hey let's follow the FAR. We just limited it down from 10 to three without having to redo any single tech evals or anything from legal or anything from policy. All of the big heavy lifts that always slow down acquisitions, you did it, now you're having a competition with just three vendors. So like those are the processes and if you really go up front and do that, your market research is the most important thing inside source selection.
(11:13)
So always, always, always from a vendor perspective be empathetic and reach out and have those conversations. But from a government perspective be willing to have those conversations 'cause it's allowed by the FAR, which is crazy. And so the third thing of like hey, how do we get into this world where competition is just a theater and it's like all just bolted on the back end we do full and open whatever is, let me tell you a story about the Clinger Cohen Act. It's actually in the book "Recoding for America" as well. So back in 1996, Republican Senator William Cohen was a part of the house reform committee and they were like holy crap, we suck at IT acquisitions. And this is in '96 and you know, we're still having this problem today. And they said why don't we build some principles into law that says like before you do an IT acquisition, you need to follow these things. And they got it approved all the way through the house, through the senate president signed it in the law. 1996 and we treat it in the acquisition community as queep. We treat it as like ah, this is just some law that we have to comply with and we always bolted on on the backend. And we bolted on on the backend even though it's law and we have to kind of like tell creative stories that were in compliance with this law. But if we took like an actual time to read the 12 principles and the guidance that are in there, we'd be successful, more successful in our IT acquisitions.
(12:50)
And like I'm all 12 are pretty good but I'm going to highlight three of 'em here. Which you know, one says, redesign processes that of a system to maximize the use of commercial off the shelf technology. You know that's maximizing COTS and defining your system to maximize COTS is part of the law. You can see like what's not in the law is like, hey make sure you maximize the and avoid vendor locking at all costs. And that those are the conversations that we're having inside the halls of the Pentagon today is like, oh we got vendor lock and we can't use COTS because like that's going to lock us in. But you know, the IT the Clinger Cohen Act, how it got established was, it wasn't just Senator Cohen's or Congressman Cohen's and here's the 12 principles. They brought all the best IT professionals, from DARPA from industry and came together to say, hey, if you do this, you're going to be set up for success. And so these principles are actually like even in 1996 are still resonate today. And then finally like modular contracting, I know if you were part of the other briefs, modular contracting is an actual principle that came from the Clinger Cohen Act and then it also got updated into the federal acquisition regulation that says, if you're doing a major IT system, you should maximize the use of modular contracting. So instead of saying like, I'm going to have a contract that's going to, all my engineers are going to be on it, my designers are going to be on it, my platform team's going to be on it, my tools are going to be on it. Like do domain driven design and break that up into what you think is the most effective way to structure your organization.
(14:35)
Because you know, Conway's law says that, your software reflects how you're organized as an organization but it's also true for your software reflects how you're organized from a contract vehicle. If you have this big monolithic contract, you're going to probably get a big monolithic piece of software that you know because it's so expensive they're going to stand up their own software lab and you're going to you know, do all of these crazy things to make it work. And it's going to be so big and so complex that only the vendor that built it kind of knows how to manage it and then you're stuck into this black box IT system that's a ball and chain and really like bringing you down. And that's like what exists today when it comes to legacy 'cause we didn't do modular contracting.
(15:24)
And a story about this is, I started as a DoD acquisition dude back in 2006 and I didn't see a modular contract until 2017 and I was in IT the entire time and this law has been in in place since 1996. Which blows my mind because the one time we tried to do it with Kessel Run, it was like hey we're going to actually do modular contracting. Let's go out and like follow these laws and these principles. I briefed it up to our senior leaders and they said, "I'm a little leery of this. I think what you're really doing is breaking up the requirements to stay under acquisition thresholds so you can speed up your process." And I was like, "Yeah, that's exactly what I'm doing." And they were like, "But I don't like it I don't think it's right." I was like, "Well have you read, you read the FAR they say to do it, have you read the Clinger Cohen Act - the law? It says to do it." And they're like, "Oh okay now I'm comfortable." Which you know, blows my mind that. So that just means the culture that we have inside DoD acquisitions is really long to change.
(16:42)
You know, Jez Humble talked about continuous delivery is like something that you want to shape your culture around and it actually drives impacts and actual changes to your culture because those behaviors drive change. If you're doing something only once because you have one big massive contract every five years, it's going to be really long to change your culture around DoD acquisition processes and competition. And then you know, it actually says outcome based management. If I had a quarter for every time someone said outputs today in this, you know, in this week in this conference I'd have like 500 bucks. And like inside outcome based management is actually something that you want to do. But instead what we do is like oh we have the systems requirements doc, it has these thresholds and it describes everything that we need and we spent two years building it and it's beautiful. That's like the antithesis of good software development practices and that was here in 1996. So we just got to like use these types of laws like leverage the bureaucracy to help you and this is one way to do that.
(17:51)
Alright, so I've talked a lot about like why this exists but I want to talk now about like what is really competition theater, you know? So I put together like a roadmap of the competition theater and the negative incentives that I've seen on my side on both industry and government and here are some of the warning signs when you're driving down the road of competition. The first one is protests. It scares everyone in the PMO protest because it's going to tank your acquisitions, it's going to tank your schedule and it's going to tank basically all progress. So, but what we do is like we create these strategies that absorb a ton of protest risk because we didn't do that market research and filter it out ahead of time. So instead of, maybe three vendors getting a proposal, you're getting 15, and just math, you have much more likely to have one of those vendors protesting because of that 'cause you didn't do it. So like you've got, we have to be leery and we have to be cognizant upfront of ways to like reduce protest risk and then shape the RFP. Like senior leaders are going to shape it where they're comfortable. Just like that, modular contracting story that I told. And legal and PK, PK is contracting in this context are also going to shape it the way they feel comfortable. If they've never awarded a contract to a sole-source 8(a) for you know, just going directly to them, they're going to be like No I don't want to do that. I think we need competition, we need all this. And so you lose that battle because they're not comfortable with that.
(19:33)
And then industry, like when I was on, wearing my industry hat a couple months ago, they will shape the RFP as well. The larges will be like, hey I, you know at Rise8 I reached out to a couple larges to bid on some like $5 million contract or whatever and they're like "Nope, we only do things that are 25 million or more so we're going to pass on this one. Unless you can like convince them to like increase the scope." And like so, so they pride themselves on like shaping the RFP to meet their thresholds from a large perspective or they pride themselves on like hey let's shape it to directly like only do our technology. But you just have to be leery of that. Sometimes that's okay because that technology's good, but the way you're become an informed buyer on that is through market research. And here's my least favorite of all of these warning signs is to make it up in execution.
(20:28)
So you know, we need to be informed buyers of what the actual market rate is. Especially like top talent builds top software. Especially if you're assuming that you need services and the labor categories that are inside of the proposal that you receive are lower than the market rate and you still award them that contract with the assumption that they're going to like magically pull it off with this junior team, you're in fantasy land. And on the flip side of that, on the industry side, that's a deliberate practice of saying, hey we're going to bid these low rates just so we win the contract but I promise you we're going to make it up in comp, in execution we're going to mod the contract for higher rates but at least we won the contract.
(21:18)
I would, that is like my least favorite conversation and Rise8 never did anything like that but like it was such a painful conversation to hear. Because what that does, I think the statistics are like over 80% of industry or 80% of government PMO people, distrust industry it's because of practices like that. So if we really want to have a culture of collaboration, we got to like just be informed buyers and award the top talent that's going to be successful. Alright, so now let's get into, we're on this journey, we're going to go and like we have some levers to pull from competition 'cause you got to do it. There's this law called CICA which is the, I don't even know what it stands for, but it's associated with contracting. So it's in or competition. So you have to do competition in some way, shape or form. The common go-to bread and butter for DoD is paper only competitions. And it, you know, as someone that's reviewed a $90 million contract that it was a 30 page paper and we got 12 proposals over time it started looking like Mr. Anderson from the Matrix. They all kind of just blurred together.
(22:35)
So if you do that and your eyeballs are bleeding and reading all this stuff, how are you truly going to make an informed decision of which one's the best? And then on top of it, like you're naturally inquisitive when you read all these papers 'cause you're stuck in some hole for six months reading them. If you have a question about those paper proposals, you have to submit an exchange notice that gets reviewed by legal and contracting and it takes about five days for you to even to get out and then you give them five days to respond. So just a simple question of like, "Hey, can you explain this in a little bit more detail for me?" Takes two weeks to get feedback on how is that like constructive conversations or you know, communication that the FAR really wants you to have. So like I would avoid just paper only competitions at all costs. So like, okay, if you don't want to do paper, what about oral presentations? Then you can just like ask 'em, right there in the room of like, hey, explain what you mean by what you just said. But there are also negative incentives with oral presentations because it ends up being a beauty pageant. 1000 times over, especially if you really want, if you're asking for slides, then the vendor's going to be building these, Adam Furtado beautiful like slides that everyone like really loves and are impressed with. And then they're going to bring in their top dog is their KP on the contract. That's like the lead singer that like briefs as good as Bryon and everyone's super excited that like this top dog is, going to be on this contract and it's going to be like this panacea of excellence. But in reality what happens is teams hire oral coaches to like, instead of, really falling in love with the problem that they're trying to solve and get empathy for the government partner that they might be working with for the next five years, they're spending all their energy, time and money on like, "Hey, how do I like stand in front of room look cool?" And like that is not like the incentive that I would want to make a source selection off of. But that's what happens when you just do oral only presentations.
(24:52)
So then we're like, okay, well if paper only is not that great, oral only has some negative incentives, why don't we just do coding challenges, right? Like coding challenges have to be good. Like we're in IT, like whatever they're doing, we can just, have 'em build something for us and evaluate that. But I can say like from on an industry side, and I think Josh Miller would like attest to this, he's probably still talking to a psychiatrist about all the scars of these crazy, tech challenges that put like a lot of stress on the vendor to like stop what they're doing and just nail this, you know, coding challenge. And it ends up turning into something like something like with pure chaos, pure chaos. And we were like naive to think that from a hundred miles away that we can enforce, coding challenges, rules and regulations, right? Like how are, you know, how are we going to make sure that only the people that said that they were going to be on the coding challenge team or actually touching the code. And even if they don't, break that rule, they bring in their super team and their super team not necessarily going to be the actual team that's, going to be on this contract. And then on top of all of that, it's just a lot of stress and a lot of churn for making a decision when you could probably do it in a better way.
(26:25)
So these are all the negative types of plays that happen that kind of promote competition theater. So I'm going to end my talk on just talking about, hey, what are some ways to like create anti-competition theater? And this first play I know is kind of acquisition wonky for all the contracting officers in the room. You might get it but or you will get it, is you, we should be using those vehicles and those regulations that help us limit competition through market research. You know, example of this is a, a government-wide acquisition contract vehicle that you can just use because the vendors are already pre-established and they exist and you can just cut a task order off of that versus having to stand up your own massive contract. But you still got to like do the work because, just because the vehicle exists doesn't mean you should use it. You know, you got to look at the rates on that vehicle. You got to look at who's on that vehicle to make sure that you're an informed buyer. But then like you have, we have to like, if we're going to get into culture change and acquisition reform, we have to be willing to accept volume. And I know that's hard. I know modular contracting is hard, there's a lot more plates that are spinning and everything like that, but it is much easier to manage five contracts than it is to manage one big massive one that you hate. And so like, think about that when you're building your strategies and then on top of it, volume is a good thing. It drives efficiencies. You know, expecting to change by doing one thing every five years is a fool's errand. The only way we're going to change the DoD acquisitions is if we repeat DoD acquisitions and change by doing type of approaches that we've learned in Prodacity and then defend simplicity, right?
(28:28)
If you know someone's like, well what if we add like, the compensation plan or what if we add like the retention rates of the, you know, the vendor? What if we add this, what if we add that? And like, no, I like, I talked about that with them during market research phase. So I kind of already know that, so I don't need to add that into it and just slow down the process. Next one is, we should be incentivizing collaboration. Market research is the startup competition, right? So this right here is a way that we can leverage like these conferences is a way that we can establish, collision points between government and industry. You should be promoting going out to these conferences to have these conversations. And a lot of the times I see these conversations as they turn into like engineering pissing contests sometimes, like you're talking about the solution that you're going to build and it's like going to be this cool Kubernetes thing and the other engineer on the defender side's like, well have you done this in, orchestration and all of this, and then they're like, that doesn't really help under, give empathy to the Warfighter or the end user of your thing.
(29:43)
Instead, like when you go to these, like, this is my advice to government people, when you go to these conferences and you walk the booths, actually talk to the people. Don't just walk around and get the swag and the freebies, actually talk to them, but don't talk to them about the solutions that you want to build, talk to them about the problems that you have and then if you talk to them about the problems that you have, they will gain empathy for what you're trying to do and then they'll reflect on that. And maybe when you actually have a one-on-one with them, it is a really meaningful, powerful conversation and not just an engineering contest. And then, you know, I talked about avoid RFIs, just paper only, just talk to people. And then, this is from like government, right? Like one way to establish collaboration and some organizations are really good at this, is like publish your contracting, roadmaps publicly. So it's not just like a black hole of like, oh this organization over here I would really love to work with but I have no idea when their next contract's going to roll out. Or I have no idea what they're going to be doing. I have no idea about like what their budget is and like even if it's worth it to me, like we should be like explaining that upfront and honest so they can like get excited about the problems that we're having. You know, like as a government guy, we have three 100 million or four $100 million IDIQ contracts inside best-in, they got awarded September of 2022. So in two years we're going to be redoing those. So like, you know, now is the time like competition is starting right now for those let's have those conversations and be open and honest about that as you know, to your industry partners. Then finally, like let's, talk about source selection approaches, right? So the one thing, I don't know if you're in this Tradewinds talk, or if you're part of the SBIR process, but competition for those is pulled all the way to the left. You know, once when you're in the SBIR program or once when you're on the Tradewinds marketplace, you've met the need for competition. So if you find a vendor that you really like and you want to collaborate with, you can do that because competition is already met and so you're, all you're doing is then having to defend to your contracting officer and you're legal that like you've done your homework with market research and then yes, this is a sound solution to go forward on.
(32:14)
And then again, so we talked about all these like crazy things with paper only, orals only, and coding challenges only, but we got to defend simplicity, right? Less is more, small is greater than big. You've done your homework up front. So you don't need, you don't need orals with these great slides or anything, just have orals like a conversation, no slides, just talk to me about like how you're going to solve my problem. And then like, wouldn't it be great instead of doing this crazy coding challenge that you know keeps, people up till 3:00 AM in the morning for four straight nights, why don't you just say like, hey, you bring a representative engineer from your company to pair with us on our systems, work with our people and do a pairing challenge. And that's what we did at Kessel Run. Yeah, you, it's as an evaluator, it's the world's most boring three hours that you'll ever see. But Dan did it and he's still here so-
[Audience Member] It's nice.
(33:15)
Yeah, there you go. But it's, I think it's a way better experience I assume than, staying up till 3:00 AM for four straight nights. And then finally like why do we need 30 pages to make a decision? Like why can't you just say, tell me how you're going to solve my problem in two pages and like, because the other 28 pages that I'm going to read for everybody are all just going to blur together in my head. So why don't like we keep it simple and just do two pages. And those types of tactics again, this is not a cure, this is tactics for future prevention of collaboration or competition theater. Those are the things that like I would love to see us as leading from the front from an innovation community. 'Cause it's not just innovation with software, your offensive line for your culture and for your way of working is your acquisition shop because they're doing all the blocking and tackling. So you guys can be scoring touchdowns and this is like just bread and butter fundamentals, X's and O's when it comes to DoD source selection. So that's my talk. I just realized, you know, at the end that I didn't let you introduce me. So sorry about that.
