How do you modernize mission-critical legacy systems without introducing high-risk failures? In this thought-provoking talk from Prodacity 2025, Nick Kuene explains why AI alone isn’t the answer—but a measured, deterministic approach can be.

Drawing on his experience with COVID-19 vaccine development, clinical trials, and adverse event reporting, Kuene highlights why organizations struggle with modernization and how they can de-risk AI-driven transformation with strict behavioral equivalence testing.

🔹 Key Topics Covered:

• Why mainframes & legacy systems still dominate critical industries
• The false promise of AI-driven modernization—and its hidden risks
• How behavioral twins ensure legacy systems and AI-generated code match 1:1
• The role of deterministic data flows in building confidence in modernization
• How to incrementally modernize without high-stakes, high-risk bets

🕒 Key Highlights & Timestamps:
[00:03] - Introduction: Why mission-critical legacy systems persist
[01:39] - The true cost of outdated systems: GDP impact, CIO retention, and failed projects
[02:41] - The modernization dilemma: Risk vs. reward
[04:26] - Why AI can’t be blindly trusted to rewrite legacy code
[05:44] - The principles of confidence in modernization projects
[07:45] - Why incremental modernization is key to success
[09:45] - The power of data-driven behavioral twins
[12:47] - How to prove AI-generated code is safe for production
[15:36] - The importance of deterministic, measurable outcomes
[16:55] - Case study: Incrementally replacing legacy batch processing
[18:30] - Final challenge: We deserve better than high-risk modernization bets

🔗 Stay Connected:
Learn more about Prodacity: https://www.rise8.us/prodacity
Follow us on X: https://x.com/Rise8_Inc
Connect with us on LinkedIn: https://www.linkedin.com/company/rise8/

👍 Like, Subscribe, and Share:
If you’re ready to modernize mission-critical systems without the risk, give this video a thumbs up, subscribe to our channel, and share it with your network. Trust AI—but verify everything.

‍

Nick Keune (00:03):

So my name is Nick Kuene and I spent most of my career focusing on mission critical solutions. So one of those was looking at the predictive maintenance for producing drugs. One of those was doing the clinical trials for Project Lightspeed, which if you all know about was how we developed the Covid vaccine. One of those was for adverse event reporting, and this is where for the AstraZeneca COVID-19 vaccine, we were able to identify issues that were happening with people who had taken that. And so I worked on all these systems and usually in the reference diagrams there were all the boxes that I had and there were wonderful boxes. I knew what every single one of them did, and then there'd be this line that would go off the grid. It's no longer on my reference diagram. That box usually went to some sort of legacy system. Sometimes it was a mainframe, sometimes it was a system that was several decades older than I am. That is the oldest problem that we have in it. These systems that have been accreted for decades and decades and decades. But don't worry because we have the newest, latest, greatest approach to solving the hardest problem, the problem that you don't even think that the humans would be able to solve, guess what? We have AI that's going to solve the problem, right?

(01:39):

Not quite. So I want to stress the severity of this challenge. It wasn't just my pride that was hurt when I realized that my mission critical systems actually depended on the real thing. That was mission critical. Mission critical systems in terms of mainframes are a huge drain on our GDP. They have a huge impact on CIO retention attention. They make it so that developers are focusing on things. They're not delivering value or delivering for their mission. And so when you're dealing with the oldest problem, when you're dealing with that thing that was off on the side that we didn't want to talk about, there's this really hard challenge. I don't want to call it a fool's dilemma, but there's this hard challenge between do we modernize or do we not modernize? Now in terms of modernization, I think everyone in this room can champion these ideas.

(02:41):

Well, we want to deliver some sort of innovation. We want to deliver our mission. We don't want to have systems where if there's something that goes wrong, we don't know how to fix it. We want to understand the systems that we're working on. An example here, I was working with a healthcare company recently and they told me that to add an error message that told people the reason why their insurance claim was rejected was going to be a quarter of a million dollars to add text. That just was in that error message that said, this is a reason why you got rejected. The result of doing that would be students being able to actually go through the process of having their claims filed correctly. We want to get out of that and we also probably have some costs that we want to get out of or get away from.

(03:30):

In addition to that, we have to think about the other side though. There's personal risk. A lot of CIOs have lost their titles, lost their departments lost their rules as a result of this. The ROI is hard. We heard earlier this whole idea of a three to five year buyback is not going to work out in a lot of the departments, A lot of the agencies, a lot of the organizations that we represent, and there's been high failure rates, but don't worry AI's here to push the button on that side. But on the modernize side, if AI is my solution, this is the hardest problem. I don't even have a human team that understands this problem. I don't have a human team that could give me the user stories that sufficiently describe everything that's going on inside of that system. My AI is non-deterministic. It's stochastic.

(04:26):

That sounds delightful, but it's terrifying. It might come up with different answers and different code on different days of the week and it hallucinates. Have you heard of AI hallucinating? Do you want it to hallucinate new business rules? I was talking backstage and we said, I don't trust AI to do my taxes. Do I want trusted to build my tax engine? Oh, it's very risky. I think. So ultimately AI is going to help, but it doesn't address that root cause. I keep dancing around. I keep talking about which is this idea that this is a very high risk effort, but if AI is still low confidence, non-deterministic, I can't confirm that it's actually done its job correctly. That's not the correct solution yet. So I need to think about the problem a little bit differently. I need to think about how I can build confidence in a solution that is going to be able to take that oldest problem and allow me to now start modernizing it now, start building new capabilities on it, take it off of the slide that was not shown in the reference diagram and bring it into my environment where I have my software factory or the work that I'm doing.

(05:44):

When we talk about the elements of confidence, I think one of the first things that we need to align on is this idea of some principles. These are going to guide anything that we do from a process or from a technology perspective. The first principle is this idea that the running system that you have is generating a huge amount of data that you can use for test driven development and that you can use to confirm that any new system is behaving in the exact same way. We don't need to reinvent the exact same system from the ground up. It is telling us how it behaves based upon inputs and based upon conditions today as it continues to run and continues to carry on the mission. So what we instead are doing is we're thinking about how can we build a behavioral twin? How can we build something in a well factored modern code base using modern infrastructure that has security best practices baked into it that everyone's going to love to build on top of, and it has the exact same behavior as the existing system.

(06:52):

And I will say this principle is one of the hardest ones to actually stick to because everyone says, you're modernizing my system. Sure 50 years of backlog that I have. These are all the capabilities that I wanted to do. Pause principles. We need to actually think about building a system that can do exactly what you did before because that's the way that we have tests that allow us to have the highest level of confidence that we actually are measuring the same behavior in the resulting system as we had in the original system. Lastly, I think I'm speaking to the proverbial choir here. Let's do this in increments. And this isn't just because it's easier to measure, it's easier to deliver when you're talking about getting to a high level of confidence. I can have confidence that a batch job is running the same in two environments.

(07:45):

I can have confidence in online process or a string of processes are running the same in those two environments. I don't know that I can say that two systems are exactly equivalent. If you draw that boundary too big, you get to this point where it's really, really difficult to prove that you actually are doing all the same behavior in those two systems. So once you have those principles, you say, great. Now we need to start executing on this. Now, executing on this process, you might think about it abstractly. Think about it from first principles. First of all, I'm going to need to have some way that I can understand this entire mess. I'm going to need some technology and some tooling that's going to help me understand this ecosystem of code and is going to help me do the splunking through the layers and layers of different teams and different patterns and different approaches, and I don't have documentation and I don't have tests and it was written and done differently and sometimes there are comments that I wish I had never read.

(08:52):

I'm going to have to comprehend this. I'm going to have to understand fundamentally where are the entrance and the exit points for any of the executions that are happening for these workloads. How can I understand what it looked like before the workload executed and after the workload executed? As long as I can focus in and get to that level of granularity, I now have a problem that I can solve. I can start capturing that data. And as you capture that data, you now are building up test suites. You are building up all of the tests that you can do. Now, these are very, very different than just unit tests or the tests that you might get from saying AI generates some code for me. We all know that you can pass your own tests. If you write your own tests, you can pass your own tests.

(09:45):

Fantastic. This is saying if I capture the data at the beginning and the end of execution and I take those same inputs and I provide this to a new candidate workload, do I produce the same outputs? Now to some of the earlier conversations, this is that idea of getting to a very precise measurement of equivalence between these two systems. This is about getting to a point where you now have confidence. You truly feel like this new system is doing the exact same thing as your original system and it takes you now into a place where you can launch that into production. You can cut that over into your new infrastructure and you're confident that it's creating the exact same mutations to your database. It's interacting with your file system in the same way. It's consuming message cues the same way. It is allowing you to say that the exact same behaviors now being satisfied by a totally different code base, and this is critically important for delivering that high level mission that we wanted, which was the idea of being able to innovate on top of it.

(10:57):

The third piece, and really one of the most critical elements for us to consider in terms of confidence is this idea of determinacy. Now, remember when I said the AI jokingly was just going to push on that side of the scale and tip the scales and make it so that we say, yeah, let's modernize every single thing, rewrite all the things. The problem was AI is not determinant. AI doesn't necessarily produce code. It might produce beautiful code, it might produce idiomatic code, but I don't know that it's actually produced code that has the same behavior. Well, if you can solve for determinacy by using data flows, now you have the highest level of proof that you actually have something that is fully determined. And so let's play a little bit of this hypothetical test. Let's say you had a junior developer who is writing code for you.

(11:56):

How do you know that that code can be pushed into production? How do you know that it's safe for that code to be pushed into production? Well, you might do a code review. You wait for a green pipeline run. You might do some analysis to just make sure that it's delivered all the functions that you had identified in the documentation or the specifications that you had provided. Well, what if we can do those exact same tests? It doesn't matter that it wasn't a human. That fingers on keyboard wrote those keystrokes. It was done by the lace and greatest ai. Well, I now have a level of confidence that I did not have previously. I now have solved the big problem, the big thing that was making it so the AI wasn't the right solution. It's no longer a low confidence answer if I'm saying that code has to produce the exact same data flows.

(12:47):

I've added determinacy to this scenario. So a broad question for all of you. How do we trust AI to rewrite our legacy code? Well, my clear answer for you is don't. Don't trust it. Don't trust it at all. Don't trust it even a little bit. Instead, trust but verify. Let it try to rewrite your code. Let it rewrite your code, but use the highest level standards in terms of data flows and equivalents to confirm that is actually producing the exact same behavior before you let anything run in production. This is like that old Kent Beck idea of I write my test in the morning with coffee and I write my code in the afternoon with a beer. As long as you have the right tests around you, as long as you have that determinacy that is driven by data, you're going to be able to do this with confidence.

(13:49):

And so this is what we're doing right now with Mechanical Orchard. So we have some organizations who are going through this process and it's wild the first time you go through it because you say, yeah, I get it. AI can write code for me. I think we've all seen demos, we've all seen scenarios, we've all seen videos. Maybe we've even written some game or some app by prompting an ai, fantastic, it can write code for me, but to write code that just satisfied the same standard that I made my human team deliver. That's that aha moment. When you go from a human team that needed to spend weeks writing code to saying ai, I'm going to make you prove to me just like those humans, just like those junior devs, I'm going to make you prove to me that you really wrote something that has the exact same behavior and that does it.

(14:46):

That's a really exciting and really interesting moment, and that's something that we're seeing in the projects that we're engaging on. So recapping these elements of confidence for us. First of all, we have that running system that is the specification. Add nothing, change nothing, delete everything you can. When you do that comprehension process, there's usually a lot of stuff that you can delete. Fantastic. Don't let any of your stakeholders come in and say, oh, we need to add all these backlog things. Great, fantastic, love you. We'll get back to that, don't worry. But first what we have to do is we have to do what's best for you, which is to take that box and actually put in our reference diagram, put it into an environment that we understand, put it into a code base that we can understand, make it so a change request to add a little bit of error.

(15:36):

Text is someone's afternoon versus a quarter of a million dollars worth of build and test and confirmation. Let's do that first, and let's remember that as we're going through this process, we can continue to capture more and more data. Now, there are some great conversations that I'd love to have with you all as a follow-up where you say, well, wait a second, we have these edge cases with our data. I don't know that you're going to be able to capture enough of it from the production system. Fantastic. I'd love to talk with you about that. I'd love to talk with any of the HR geeks out here about the 27th payroll that you have to do that only happens once every seven years. Yeah, fantastic. Let's talk about that. There are ways that you can solve that problem, but now we're talking about a very different type of problem that you can solve with engineering and you can solve with computer science and the technologies that we have. Lastly, of course, we need to do this in an incremental manner, and one of the wonderful benefits of saying the behavior has to look exactly the same in the original system as the system that we are building is that you've made it so that cutover can be done incrementally. So a batch process might now be running in your trusted cloud environment.

(16:55):

It might be running inside that new environment, your target platform environment, but it's writing files that are being picked up by things that are still existing on the mainframe. You can still have that intermediary state. And the reason why you could do that is because you have held all of your candidate workloads to that highest standard of acting exactly like all of the code that used to be on your mainframe. So I would challenge all of y'all. We deserve something better than the traditional high stakes, high risk bet that people are doing on modernization. We deserve something better than throwing more humans into the fray and expecting that they're going to do it faster. We deserve something better than all the approaches that have been done in the past. Now, we do have an approach that can work, but we need to do it in a very measured way by not trusting AI at all. If you would like to get in touch, my colleague will be presenting here tomorrow, Edward Hieatt, and we'll also be around to talk about those really fun topics such as if you want to get into data flows and what are the complexities for being able to actually do this. But I hope that this has given you a new reason to not trust ai, but maybe to use it to actually build some mission critical systems. Thank you.

‍