Join Sarah Mazer, Federal CTO of LaunchDarkly, for a unique and interactive lunch session exploring the transformative capabilities of feature management in software development. This engaging demonstration showcases how LaunchDarkly's pioneering service enables developers and program managers to control application features in real-time, without redeploying code. Discover the myriad ways in which organizations, including prominent government agencies like CMS (Centers for Medicare and Medicaid Services), leverage feature management for risk mitigation, technology migrations, targeted experiences, experimentation, and mobile releases.

Hi everybody. I am Sarah Mazer, I am the Federal CTO of LaunchDarkly. It's day three. It's been a great conference. It's lunch. So we just wanted to do something short, sweet, fun, interactive, and so that'll be our lunch today. Thank you for joining us. Now, to make this interactive, what you're going to do is scan this QR code with your cell phone, and I'll give everyone a minute or so to do this. What this is going to do is immediately charge your credit card a thousand dollars, send it to an offshore account in the Cayman Islands.

No, I'm just kidding. What'll happen is it'll take you to this really cool, awesome website. And so this'll be the interactive demo portion so you can see how things will change on your phone. So if you miss a QR code, there's cards on the table. My colleague Jim is walking around with some. We'll also pop that up again a bit later. But first I want to set the stage for the demo before we get into the interactive part. So we do feature management. We are the pioneers of feature management as a service. We've been around since 2014 and our first public sector customer was way back in 2016. So CMS Centers for Medicare and Medicaid Services was one of those early adopters of a government agency starting to use feature management as a service. So if you're not clear about what feature management is, you might've heard feature toggles, feature flagging, is a software development process that allows you to change the behavior of your application on the fly without having to redeploy any new code. So it's pretty cool. It's just like magic. No more deploys.

So many of you might be doing feature management in your own manner, but you still are working with config files and you still have to do a redeploy. With us you don't have to do that, you just change the rules on the fly. You can also do something called dark launch, which means that you yourself as a developer can see a feature working in production before the rest of your users do. And so you can essentially separate deployments from releases, release the feature to yourself, safely test in production, and then turn it on for others when the time arises, when it's time to go live for your user base.

So since we've been around for a while and we now have over 5,000 customers and we manage over 26 trillion feature evaluations a day, so we know the base pretty well. And we asked our customer base, what are the main ways that our customers are using feature management? And we heard five main categories. So we reoriented our company around these five main use cases that I'm going to talk about today.

The first is de-risking and release, and that's obviously the number one concern for any of your applications. And then technology migrations is right after that. So you're constantly wanting to get off of those older legacy monoliths into more modern architecture and microservices. Then targeted experiences. So it might be personalization, it might be making sure that the right people get the right features. That's something that we support. And experimentation. So as a developer, we hope to encourage the culture of experimentation and everything that you do, and I'll talk about how you can build experimentation into your DevOps processes. And then mobile releases. There's not a lot of mobile apps in the government, but I think this is really one of the coolest ways to use feature flagging, and I'll talk about that too.

So reducing risk. This is first and foremost one of the most important reasons to use feature management in your application. The first is like a kill switch. So you want to be able to react instantly to turn off a feature if something goes wrong. It could be a security breach, it could be some backend component going down, it could be a failed deployment. And instead of doing that lengthy rollback, having to quickly code up a solution or find a working prior version, spinning up containers and going through the deployment process, very timely effort, you can actually just turn off that one broken feature, do a patch forward and move on. Another thing is when you're safely testing in production, then you can see and be sure that things are going to work in a production environment before the rest of your users see it. So no more late night launch parties at midnight when something's supposed to go live, and we call it push and pray releases, where you need to really worry and hope that things go right in a production environment. You've seen it working in prod.

And then you want to enable teams to go faster together so people can work together. Everybody's on the same page, they know what has been deployed in staging versus production, and it really reduces risk. So I'll give you a couple examples of Healthcare.gov and Medicare.gov that are using LaunchDarkly. We have over 30 teams now within CMS alone using feature management, and some of the use cases that I've seen are pretty interesting. So one is that, and I won't name names, but there's an older legacy database system from a company that everybody loves to hate, and it goes down somewhat frequently actually. This database is really critical to CMS because it actually houses all of the Medicare recipient data. And so when you are in the Medicare program and you try to log in to look at your plans, make choices for the next year, well, sometimes you can't do that because the database actually goes down.

And so what CMS is doing is they're reacting to an incident by using feature management to turn off the login capability to the database for the public. And so you don't have the public trying to log in and seeing corrupt data, your messages, you're not swamping your CSRs that are fielding calls from the public. So they might be hunting around and going, wait, where was that login? But that's better than the alternative. And then you have a team of people trying really, really hard to get this database up again on the back end. And as soon as it's ready, then they flip the toggle back and they turn on access so that people can log in. So that is one really neat use case.

Also for both Healthcare.gov and Medicare.gov, they actually deploy their code in a production environment and they schedule the feature to go live to the rest of the public. So using our platform to do the scheduling. So they don't have late night launch parties anymore, it's all been scheduled, but they can also breathe a sigh of relief because they've logged in as a fake beneficiary for weeks, making sure that the new information is working without any incidents in production. So they've already dark launched it, they know it's working in production, it's saving the agency from the front page of a newspaper embarrassment. And when they roll out the new Medicare plans for the public, it actually takes seven minutes to go boom, boom, boom, boom, boom, and check all the features that they have just rolled out and double check. But it used to be multi-hour process, now it's just minutes and most of it is scheduled.

Okay. All right. Switching to handheld. All right, so migrations are another important factor for the government. And I know in this conference we've talked a lot about the Kubernetes ecosystem, software factories, all of that. And that is really important in government modernization. And the cool thing is feature management can live alongside of it and even add more capabilities to that ecosystem. So you no longer will have big bang migration. So going from a blue to green deployment, that is a big bang migration. And even within a blue or a green deployment, you can have more selective targeting of your users and the ability to test in production. So it helps make sure that whatever you are doing is even safer by testing and dark launching, making sure that you can see something working in a production environment. And then also if all you're doing is needing some way to mitigate downtime, you can do that with feature management.

You may not even have to go all the way to Kubernetes, which is really complicated for some government teams that may not need it. So feature management doesn't have to be Boolean flags on, off. You can have different variations. You could have variations where you're just using the original database. Then another variation of that flag that you might be writing to the legacy and reading from the legacy, but also writing to the new database and so forth. And you progressively make that switch over using variations of a feature flag. So you can do that and using feature management on the back end, like a lightweight service mesh to control traffic from an older component to a newer component. So there's a lot of reasons to use feature management, not only for user-facing applications, but on the back end doing technology migrations.

Then with the directive for CX in government with the executive order, that is a new initiative. We don't see a lot of government doing a lot of personalization right now their websites, but they're moving towards that. Especially with CMS, I've seen that initiative. And so just that same technology where you've dark launched something and you're targeting yourself, you're seeing something work in production, you can now target people based on any metadata that's available to your application. So it could be PII, it doesn't have to be, you can target a subset of anonymous users. You could target on IP addresses, you could target on device types, you could target on someone's favorite pizza topping, whatever information that you want to use in a rule set you can use with feature management to make that determination of whether someone or something gets that feature.

So I also like to bring up how this aligns with zero trust architecture, because if you think about whatever attributes that you're using to target somebody, well then you're talking ABAC. So those security-minded people out there, you're thinking about zero trust and lease privilege usually from the aspect of data management. But you can think about it also with feature management, so really is attribute-based access control of features. And these lower-level network security tools that make the determination of whether someone or something gets data, sometimes that type of information is known by the application but it's harder to represent in a lower-level networking tool. Like the concept of a beta user, the concept of a congressional district or courts, just anything that's more nebulous that an application would understand, but not a lower-level security tool. And so why not let the application decide whether someone or something gets a feature? So I'd like to say it very much aligns with any zero trust initiatives that you have.

And then experimentation. So a lot of times when you hear experimentation, you think AB testing on a user-facing website, and that is certainly the case. We have Healthcare.gov and Medicare.gov trying to work on the wording and entice people to fill out forms and add in some information that they can't require legally, but it helps them and make sure that they get the best coverage that they have available to them. And so they're testing the messaging to make sure that they entice people to enter this information just so they can get coverage. And then also tracking error rates on pages. So you can do AB testing of a public-facing website. But going back to migrations, you can AB test your infrastructure.

So when you use feature management, your metric doesn't have to be user behavior. The metric could be page load time, memory usage, traffic, and anything that you might want to look at. And so why not use experimentation to AB tests, say different AWS components and look at that and make sure that you've made the right decision before it's too late to change your architecture. So it's just something to think about with experimentation and feature management. So all you're doing is you're creating a feature and then you're tying that feature and its variations to some sort of metric. And then we do the back-end analytics, we do Bayesian statistics determine the probability to be best for that result.

Finally, mobile. I think that this is really cool. I wish the government had more mobile apps so I could talk about it more, but it's such a pain to get an app through the app store. It's a lengthy review process. You're beholden on the app store people to review the code. What happens if you push out something to the app store and there was a broken feature? Now you've got to go through that process again to get that quickly updated. In the meantime, you're getting a whole bunch of one-star reviews. So instead, if you have everything controlled through feature flags, you could push out a release through the app store. If something's broken, you just toggle it off. Also, you could dark launch features so that you can be on your schedule and not the app store's schedule. So you just dark launch a feature and whenever you're ready at some point in the future, you go and turn it on.

Similarly, if you have older versions of applications, it's really hard to force some users to upgrade. So what if you targeted based on the release and you start shutting off features on these older versions, you essentially force people to upgrade. And so not only that, but you could put nice messages ahead of time and say, well, you better upgrade or else we're going to shut this off. But there's a lot of power that you have now if you use feature management with mobile applications because it is such a pain to get through the app store.

Finally, I've done podcasts on just this topic alone, which is developer morale because if you're on call in the middle of the night and you don't have feature management, you may have to code on the spot when you're sleep-deprived and then test it on your system, test it in stagings, get approval, spin up some new pods, and then move it over to production. Hopefully you did everything okay. But if you had feature management, at least you could toggle off the broken feature and get some coffee, buy yourself some time, maybe even the next morning. And so just having that access to feature management is a real big morale booster and developers just love this type of technology. And then it facilitates communication. It brings the business side and the developers together, so they often fight each other and government. And if the business side can take some pressure off the developers, that's always a good thing. And they can also see the status of their features by going to one source of truth and being able to see is that deployed in production or not.

But the cool thing is we make it easy for program managers to control the release of features. So that takes the burden off the dev team. So the program managers know the executive orders or who gets what when, they can go and schedule things and they can go control the progressive delivery of features. Maybe they know who is in a beta cohort, and so they can change the rules. It's a simple user interface that anybody can use. We made it really simple and you can have templates that you save. So they just reuse the template over and over again. So it's super simple for technical and non-technical people to work together.

And then cost savings. So Healthcare.gov or CMS, they only have one pre-prod environment, it's called Impel. And so it's a huge cost savings. Once you realize that you can actually test safely in production, you don't need dev tests, staging, QA, and all of those environments. You may need one, but it's also a big cost savings. And we have a lot of studies on the ROI of feature management we could share with you. All right, so with that, I've set the stage. Now we get into some fun stuff. All right, just moving my windows around here.

On the left-hand side, you have the coolest demo app ever, like a page with a word on it, is that not the coolest demo you've ever seen? And then on the right-hand side, that is LaunchDarkly a list of your features with some information. But this is Prodacity, so we're not supposed to be salesy here. So what I'm going to do, and we'll see how fast the internet is. All right, I've switched it, so you should see it update on your phone. So the application and the client has reached out to LaunchDarkly under the hood and it is connected and it's listening for changes. And so everybody's phone should have updated and our SLA is 200 milliseconds or less. And so next, if you didn't get the QR code, there it is again. So if you want to play along and get the QR code, you can scan it. And again that there's cards on the table. All right, so we're going to get rid of this guy. Bam.

Did I save it? So bam, that's lunch. Remember Emerald Lagasse, what happened to that guy? He lost most of his restaurants, it's lunchtime. Let's see, I turned off the wrong one. I'll turn off QR code. All right. So what I did is just toggled features, and that's a simple kill switch example. So if you just deployed 14 features out there and one is broken, then you can quickly turn off features on or off as you see fit. And so if you look at a real production environment, and I'm going to... There's reflection on my screen, okay, so let me say I released a whole bunch of features like this, and in this case it was an image and it's broken. And so that's a bad thing. And so what might happen is... I'm trying to get to some fun animations.

All right. Trying to make it fun and interactive. All right, so everybody's got the fire animation. All right. So if I've deployed a whole bunch of features to the application and one's broke, then in the past you would have to go and under the hook, fix everything and do a new deployment. And with LaunchDarkly, what you would do is just turn off that feature. And so I'm going to turn off the fire animation here, and then I'm going to fix the one component that was broken, and we have toggle mascot here. And so that is using feature management in the way that we're just toggling things on and off, but features don't have to be just completely Boolean on off. So what I'm going to do is go and show the suit of cards. So everybody should actually get a different suit. It's random on my phone. I've got hearts, on the screen I've got spades. So who has hearts? All right. Who has spades? All right. And diamonds? All right, so everybody, okay, so you get it.

So this is part of the demo, so I don't have to have you type in your email address or anything, but it's just a way for me to show selective targeting for this application. And then I'm just going to go and look at this background color of this application. And so the way that feature management works is you're going to evaluate rules from top down. And so I could add a prerequisite and I could say only people that get this feature are the ones that are already seeing beta UI of true as an example. So with CMS, they literally have a flag OE that they've used as a parent flag, and then all the flags after it just go live when that OE is toggled. So they don't even have to go and toggle like a hundred flags.

And then I'm going to start and I could target individual people. So this is targeting myself to see it in production or my dev team, QA team or beta users. And then if I've got a team together that I want to continually reuse, I call that a segment. And then the segments are used in these rule settings down below. So what we've got here is if you have a diamond, then you get green. And if you have a heart and the device is of Android or iOS, then you're going to see red, and then otherwise you are going to get yellow. And so I'm going to go ahead and turn on this feature.

And so now on the computer I have yellow, on my phone I have red. And so that is the way that you can selectively target individuals or groups of users. So entitlements. A court system is a good use case where they might have different types of courts that they want to give different courts, different features, then they would use information instead of the cards, they would use login information or maybe IP addresses to do the targeting. And it really can be targeting on anything. So if I wanted to, I could add a rule that is say a segment. So somebody is in beta segment, then they would get that or you could do it by email address. You could do, so custom user email.

And we've got all of these different selectors so ends with matches, regex greater than, less than, for dates, et cetera. So a lot of times people would do ends with at Agency.gov and then they would get through so all their internal users would see a feature, but everybody externally would not. And so you just build out these rules. And again, this is something that a program manager could do. So it takes the burden off the dev team. And so everybody should be seeing a different background color based on looking at something on a mobile versus a computer or the card suit that you got. And it just shows that you can safely target people in a production environment. One other thing that I'd like to mention is that we treat client side and server side SDKs very, very differently. So that is a competitive advantage for us where we don't trust the client.

So you're only going to send information to us and we're going to send back a true or false to a client that you should get a feature or not. But you're never going to be able to look at a payload and look at all the options of the rules for somebody on a client side. So that is something that is very important for our government customers. On the server side, we do trust servers a little bit more and you can download the entire rule set to the server application. So the server can make the determination in near real time, but client side is very, very safe. And so I just wanted to mention that as well.

So that's a simple interactive example where you see that everybody has different versions of a background color and so we're able to target on any attribute that you need. You're able to schedule feature flags, you're able to use rules based on any metadata that is available to you. And there's a lot of different ways and reasons to use feature management from risk mitigation to technology migrations and so forth. So that's it today. Thank you so much for coming to the lunch and learn. I appreciate it.

‍