Chris Romeo 00:00
Brenna Leath is currently the head of product security for a data analytics company where she sets the application security strategy for research and development and leads a team of security architects. Brenna originally joined us to talk about the executive order 14028 and the implications for private-sector programs. But as we were chatting about security champions and product security leads in the preamble to the interview, we decided to change our focus and cover these topics instead; product security leads a different way of approaching security champions. We hope you enjoyed this conversation with Brenna Leath.
Tiara Sanders 00:41
You're about to listen to the AppSec podcast. When you're done with this, be sure to check out our other show, Hi/5.
Chris Romeo 00:49
Hey, folks, welcome to another episode of the Application Security Podcast. My name is Chris Romeo. I'm the CEO of security journey. I'm also joined by Robert, my good friend, and co-host of the Application Security Podcast. Hey, Robert.
Robert Hurlbut 01:03
Hey, Chris, Robert Hurlbut, and glad to be here. I'm a Threat Modeling architect, and I'm really looking forward to our talk today. Trying this again, a couple of times, we tried and looking forward to it.
Chris Romeo 01:14
Technology is a wonderful thing when everything works as planned. But you know, as good security professionals, we're always happy when the security policies take effect and prevent us from doing something. No, we don't; we’re just as annoyed as everyone else on earth when that happens. We were going to talk about the NIST executive order originally. Still, in the preamble to this conversation with Brenna Leath, we started talking about security champions, and off we went. So, Brenna, we always like to start with our guests’ security origin story. So, tell us how you got into this crazy world of security?
Brenna Leath 01:50
Sure, I was trying to give you a new and different topic instead of just talking about security champions all the time. But you know, when you love something, you love it. So, I respect that. My name is Brenna Leath. I am the head of product security at an analytic software company. I have a small team of security architects, and I also have, we call them matrix reports, product security leads, which is what you guys got very excited about, as soon as I mentioned that. So, my journey into software security.
I have a master's degree in Film Studies, and a bachelor's in English, which a lot of people would think doesn't equate to working in technology and are surprised when they find out what I actually do. So, I started right out of grad school at the North Carolina State University Library. I was running a digital media lab with a lot of student workers. So, that's audio editing and video editing. Then I started working in a somewhat technical writing facility as a developmental editor, which turned out to be mainly project management. So, I transitioned into project management and joined research and development as a localization project manager. Then, after doing that for a while, I noticed that our security project manager was getting overloaded with a lot of work and projects. I had sort of automated a lot of my responsibilities that had primarily been taking up my time previously, so I asked to start helping with that. I ended up running one of our product security compliance processes that had to be driven throughout r&d. We have a large r&d; it’s about 5000 people, 2500 ish developers. So, after I'd been doing that, the security team asked me to join full-time as a security compliance engineer. After I'd been doing that for a while, it sort of morphed into more of a renaissance security engineer position. Fast forward to now, I lead the team.
Chris Romeo 04:02
So, film studies and English are a fascinating combination for me. I love to hear stories about how people have come into security from other disciplines and studied other things. So, I'm curious, what can you draw from your experience studying film and studying English? And how do you apply those things to your job insecurity?
Brenna Leath 04:26
So, I watched a panel a little while ago with a bunch of application security professionals. It was Netflix, Spotify, and a couple of other people, and the topics were overrated and underrated in application security. One of the top underrated skills was communication skills, and I find that to be so true. I have very strong communication skills because I studied English and film studies, which is a lot of writing. But what it really is, is a lot of listening, a lot of watching, a lot of analyzing and synthesizing information, and then drawing either conclusions or extrapolations or observations and bringing what may seem to be very disparate things into a clear focus through analytic thoughts. So, when you translate that to application security, I think, especially when you're looking at the secure software development cycle as a whole, there are many different stakeholders, there are many different processes, many different tools, weaving those together, in a way that seems to make sense. Or, as I'm constantly saying, getting to a root cause analysis instead of attacking the symptoms is a very analytic thought process. Being able to both communicate those observations and then persuade people to act on those observations and/or explain what they need to do ends up being a critical skill. Especially the more people that you end up having to interface with within a large organization like I'm in.
Brenna Leath 06:09
If we go into this topic of product security champions and leads and things like that; When we were chatting earlier, one of the things that really caught my attention is you describe for us how you have product security leads, and they have security champions that report and that work with them. So, I've studied champions programs, I've run a few of them, I've helped other people launch them. I've never had anybody attempt this. So, you mentioned in your intro that you had these product security leads matrixed to you. So let's start with that product security team. It's like a human pyramid of security champions; it’s amazing. So, when I took on the head of product security role, I realized that we had a lot of very big challenges ahead of us, and we did not have enough security people. There is a serious talent shortage in the industry, and I don’t see it getting any better anytime soon, unless we're home growing talent or looking in unlikely places. I might be slightly biased because I am one of those unlikely places/homegrown talent. I was a project manager plucked out of project management into security, and now I am a security leader in my organization. So, while that might seem odd to people, I think what it really is, is a testament to if you have intelligent people who are curious and quick learners who are passionate, who are innovative, give them something they can care about. They will show you what they can do. So, when I got into the head of product security position, my executive team asked me what I needed to be successful in the role. The first thing I said was, I need people, but I don't need to hire eight people at once. What I need is every division that we have to either internally nominate or open up an internal position to devote one person full time to product security and call them product security leads. The way I explain the role, I have a really cool PowerPoint slide that shows the liaison role of product security leads being the connection between the central product security office and our 150 to 200 Security champions that we have. Because we do have a security champion in each product team. But we have about 150 products throughout different divisions. So, it's very difficult to have a strong link between what the product security office needs and what the security champions know and understand. Because each of those products might have a slightly different workflow. They're using slightly different tools. They have some different flavors of agile or DevOps. So, what seems to be a universal software security lifecycle isn't really. There are a lot of different flavors of it being enacted in different areas, and people start to tune out if they think one thing isn't relevant to them; they stop listening. Or if you're having open forums all the time, like we were, and you're getting 30 to 40 people just listening, but you have about 150 people who need to hear what you have to say, there's no way to rope that in. So, my solution was, let's have people who are matrixed or Satellite as another term might be to a federated model of a central product security office. Where we strengthen our links from setting the software security strategy for the organization, setting the policies, setting the standards, setting the processes, choosing and approving the tools, maybe trying to design security features that get adopted and have a feedback loop between what the product security office is coming up with and what's actually getting down into teams and vice versa. If teams are coming up with great solutions, if teams are coming up with reusable code, if teams are coming up with observations about what's actually happening at that level that we don't have visibility into, whether that be technological or process, we needed some way to have a regular link. A tear of people whom all understand the same initiatives, understanding the same structure of what the product security office needs to achieve, even in areas that might not necessarily seem like they need a full-time security person. One of the very interesting areas was documentation development asked, "Do we really need a product security lead?" And I was like, "Yes!" A lot of people discount, again, communication skills, but also security documentation, and how just having someone who is focused on understanding how security works within the company, what the public face of that should look like, and what the customer workflow might be of people who are trying to use that information had ended up becoming us useful in so many ways that we didn't imagine when we first came up with the role. That's not to say that there aren't challenges with it, because there definitely are. Because these people don't report to me, even though it was sort of my and my team's effort to set up onboarding sessions for them to teach them about one software security at my organization, two software security as a discipline, and then trying to build that knowledge and foster both the understanding of the why and the understanding of the how to do various things. Whether that be threat modeling, secure design, the different tools that we use throughout the lifecycle, the different skills that they needed to be fostering, and the things that only human eyeballs can pay attention to in application security. It's been, I think, a really rewarding experience. Well, definitely for me, but from the feedback, I'm getting from them, they feel like they came from different disciplines, whether that be test or quality and engineering, development. We had a database engineer; I think we have someone who used to be a director, and obviously, a technical writer, technical editor. These people are embracing the security discipline and growing as professionals. You asked at the beginning of the episode; it’s interesting to find out how people get into this discipline. Because everybody brings a unique perspective to it and everyone has different skills. It is also great to see how these eight people who have the same role have to do it slightly differently because they're all in different divisions. So, they need to understand the big picture of what my group is putting out, and they need to understand all of the core security practices that we need to do. But then they need to adapt. They need to take what they learned from us, and they need to help their security champions understand how it works for them. Sometimes that means customization, tailoring, like, what we say is a standard if we say the what they have to figure out the how, if that makes sense. So, we say you have to do this, but maybe this looks different in 10 different product teams, depending on the product they're working on? What's the language they're writing on? What's the architecture they're using, it's very different, and it's impossible for a small central team to be able to give everyone that kind of unique and deep-seated guidance when all of them have a lot of product knowledge specific to their product. I heard something the other day that was interesting about product security architects’ attitude, which was knowing how to work with engineers who know a lot more about their code than you do. I was like, well, obviously, but to hear it put that bluntly when it was very funny. But that's why it's been cool to have a product security lead in each area. They can get that sort of homegrown knowledge and expertise within their area and have a close connection with their security champions that I wouldn't have. So, if I say, oh, my gosh, we need all security champions to update their package DML file and add this piece of information. How am I ever going to? Oh, okay, I'll just ask the product security leads to help make sure that message gets through. We were trying to rely on product and project management, and that just wasn't quite getting there. Like they needed their own, I don't want to call it ringleader. But, gang? They needed their own gang. Yeah, it's kind of a circus. But team, that's a good one.
Chris Romeo 16:15
Yeah. So, you used the word federated regarding application security and the collection of people. Can you give me a definition to help me understand how you're using that word federated?
Brenna Leath 16:36
Yeah, it's actually an interesting term. I picked it up from the BCM, and I really ended up liking it. Because if you liken it to the federal government, a federated centralized, setting, standards, policies, strategy, direction, and then out from there, you are reaching into the different areas. So, there is kind of, government's a dirty word, but to have like a federal, sort of centralized security office. Then you have one tier down, which I guess you could call state government, which would be product security leads, and then down to, like, county government, which I guess would be the security champions if you think about it like that link. But the way that Besian put it was phases of the software security initiative, SSI, maturity. So, the first phase is security as a cost center, the second phase being security as compliance, the third phase being security as tools, trying to automate and get to speed and then the fourth phase, which not a lot of people are there, but security as enablers of sales of the actual software maturity and a driver of appeal itself. So, I don't think a lot of companies are there; I don't think we're there. But I think it was a very big step for us to have the executive support and buy-in to say, yes, you can have these full-time people to do with what you will. They don't report to me; they’re not in my class center. They all report to different divisional heads who are higher ranking than me, but they matrix report to me. So, that means they look to me for guidance; they look to me for help about what they should be working on, or help to prioritize or how they should be communicating some of these security initiatives to translate them to what that actually means for their divisions and what their team should be focusing on. So, it's almost like a ripple effect and an amplifier for what we need everyone to know. A lot of people will say security is everyone's job, but that doesn't actually mean anything unless you can translate that into, Okay, but what do they actually have to do? Yeah, there has to be some action that they can take. So, I'm curious now when I think about this product security lead, and you said you've got eight of them. So, you've got some experience with a broader view of what they are and what they can do. If you could only give us three, what are the three kinds of tendencies or characteristics of successful product security lead? Because I'm imagining that a lot of people listening to this are just fascinated as I am because maybe they haven't thought about this as a way of approaching champions. They may be sitting here thinking, this is a really cool idea. How am I going to find those people? What am I looking for in those people? And I'm just curious if you've seen some qualities of those people where you're like, Oh, yes, a successful product security lead is someone who is a, b, and c. So, the first thing I did when I was describing what I wanted this role and what I wanted this program to look like was to start by describing what it is not. So, I made a very distinct role that was not a project manager, developer, security champion, or tester. And because I knew that that was going to be the first issue with this was they were going to try to turn that person into one of these roles, or think, Oh, well, now, project managers don't have to look after their backlog, because the security leads going to do it, throw it over the fence to them, or now, we don't have to worry about automating our SCA scans and grooming those results, our product security lead is going to do all that for all the teams. So, don't worry about that; go back to your regular development job, and don't be a champion anymore. I was like, No, that's not what these people are for. And I have, over the time that we've had them, which really hasn't been that long, I think they got in place. Shoot, maybe it's been about eight months now. But throughout that time, I've had to do some further role definitions, and it sometimes changes because, again, each one of them is different; what their division does is different. So, they might be spending more time on could be different from someone else. So, that's the first thing to define what that role should be and what it is not. For me, a product security lead is operating at a strategic level. Lead is a keyword; they need to be leaders, they need to be good communicators, they need not mind just diving in somewhere where there might not be a process defined, or maybe there might be a problem that crops up. They should be looking to us for guidance on how to solve it and what the best way for it is. But they also can't be afraid to be recommending things back to us. So, what's a good example? I would say I actually just recently came back from maternity leave; I had a baby. While I was gone, log4J happened. I had gotten the product security leads in place and gotten them largely onboarded. They understood the role, the initiative, they understood what to do, and they killed it. They did an amazing job during log4J, they impressed everyone, and I was a little scared when I came back because I came back. I knew that it was happening while I was gone, but I couldn’t log in. So, I came back about three weeks after it happened, and I was blown away at how great a job they did. Triaging, doing incident response, following up, being very on top of the ball. They are listening to the feeds; they’re finding out about things before they really even get in the news or definitely before customers do. So, they're looking into things; they’re being proactive, they understand the designs of their products. So, I'd say don't be afraid to learn; you need a curious person who doesn't mind. Security is a constant brain activity; you need to be learning, you need to be looking ahead, you need to be proactive, you need to be organized. Maybe only one of them has a PMP project management background, but it is important to know how to prioritize, how to figure out what it is you should be working on versus maybe what people are telling you to do. Then think a little bit more critically, like okay, I could spend a lot of time on that but am I attacking the symptoms? Or is this actually going to further my true purpose? Which is, you know, looking at automation or looking at all of the false positives coming from a tool. Which one is going to help you more? You could be grooming false positives forever and ever and ever, or you could be working on tuning the queries. which was good that you like a better return, you know? So, having people who can do that sort of critical thought, and having people who are not afraid to reach out into areas where they may not know a lot, or maybe they're not comfortable. Threat Modeling is scary. It doesn't need to be, but many people are scared of it.
Chris Romeo 25:41
It has a threat in the title, so you should be scared; this is a threat. We don't believe in that; we don't believe in scaring people with it. So, I've got another question, Brenna, about the product security leads. What I'm curious is, how much guidance do you give the product security leads about who the champions are that they go after? Or are you kind of focusing on capturing whom the entire security champion body is and declaring who that is and then handing those people to the leads? Or are they out there recruiting their people and running their own kind of mini-programs?
Brenna Leath 26:18
The answer is all of the above. So, we started with a list of security champions that we weren't sure how accurate it was. Some people had probably left, some people had kind of been assigned amongst their teams, we had a static page that wasn't getting updated. So, even though we try to keep it as data-driven as we can and keep that sort of information close to the source of the code, that's not necessarily reliable either if people aren't updating it because people always have to correct it. So, that was step one; when I asked the product security leads to come on board was okay, do your inventory.
Tell us who the champions are in your area? Are you short-handed? Do you need to recruit more? Do you feel like the ones you have a solid understanding and a grasp? Or is one person doing all the security work for a group of like 50 people? Which ended up that happened in a few places, and it has become difficult. You don't want a single point of failure anywhere, but definitely not somebody doing the security work on your team because if they get a job somewhere else, all that stuff just drops. Then you have to figure out how to get somebody else up and running. So, multiple people need to be doing security processes, whether that's reviewing findings, whether that's helping track remediations, or whether that's doing some secure design work. Somebody said something interesting in one of the groups that the goal was to, quote-unquote, kill security champions altogether. The lead who told me that was shocked, and I was like, actually, it's not really that shocking when you think about it. Because in the next era of security, security is everyone's job; we’ll all be security champions. It won't be such a, in DevSecOps it's not such a defined role, you all sort of know what is going on, you're all interacting with the security of the product, you're all trained to do those basic Threat Modeling questions sets of, what are we making? How can it go wrong? Did we do a good enough job? Like, you all know that there doesn’t necessarily have to be a special set of security people. Which I think is a good goal to aspire to. Still, there will always have to be people, whether the central team or product security leads, who turned out they turn their entire organizations into security champions. Wow. But right now, it is kind of like a little mini club per division. They all, I can't even have one on one with all of them; I’d be doing nothing but having one on one. So, the way I've tried to work it out is okay; I set up all these onboarding sessions for them. It was about an hour a week, every week. Then I also have what I call open forum lunch when it's just come and pepper me with questions. So, I just came to show up, and they asked me whatever they wanted. Now that's kind of morphing and evolving into all right, everyone. If you’re going to demo something you want to share with the group, let's do that. Then instead of having our onboarding sessions every week, we’ve transitioned to onboarding slash presentation, learn bi-weekly, but then the other bi-weekly is secure design board, where everyone brings their secure design reviews or their threat models we talk about about about that. Then I'll just do ad hoc things, sometimes just for team building. A couple of weeks ago, I did a fun Friday, where I got the OWASP Cornucopia cards, and there's an online beta version of the game. So, we all played the online beta version of the game, and somebody brought a recent data flow diagram that they were working on. So, we sort of learned about the game and poked some holes in that. So, that was interesting. So, I said, try this out with your security champions when you're trying to look for stuff to do. Maybe have lunches or coffee meetings with them where you talk shop about what's interesting, what's working, what's not working, and just have that regular touchpoint with them. Because I can't meet with all 150 Security champions, and even if I did open up a meeting, we do have open forums where it's just the product security office, and we say, all right, ask us anything. So a lot of times, it ends up being questions about tools, or maybe vulnerabilities or different methods or something that people want to share, and sometimes it's just me talking trash.
Chris Romeo 31:25
There are a lot of really cool things you've done in this program. You know, I'm somebody, like I said, Who, and Robert is as well, we kind of study these things, and put them under the microscope sometimes. But you know, you're doing a lot of really cool things just to build a security culture at the end of the day.
Brenna Leath 31:43
That's the goal to build the security culture. We did a lot of that last year, too. We had an r&d wide Bug Bash that leadership said, all right, everyone participates, and you're going to learn about ethical hacking today. We had everybody try to learn how to use the Burp Suite. And that was crazy because it was like, all whatever, 2500 people who were supposed to be participating and did them all in teams and stuff, but they ended up doing a great job.
Chris Romeo 32:14
Yeah, wow. Well, Brenna, thank you so much for sharing your expertise here and your experiences with us on this topic of product security leads and product security champions. We’ll have you come back to talk about what we actually planned to talk about.
Brenna Leath 32:28
I was going to say I prepared a whole topic that we didn't even get into today. So, you'll have to have me back.
Chris Romeo 32:34
Oh yeah, we'll do a second episode talk about that.
Brenna Leath 32:36
Well, I'm here you know where to find me.
Chris Romeo 32:40
Thanks for listening to the application security podcast. You'll find the show on Twitter, @appsecpodcast, and on the web at www.securityjourneycom/resources/podcast. You can also find Chris on Twitter @edgeroute and Robert @Roberthurlbut. Remember, with application security; there are many paths but only one destination.