Application Security Podcast

Omer Gil and Daniel Krivelevich -- Top 10 CI/CD Security Risks

April 26, 2022

Show Notes

Daniel Krivelevich is a cybersecurity expert and problem solver, with 15+ years of enterprise security experience with a proven track record working with 100+ enterprises across multiple industries, with a strong orientation to Application & Cloud Security. Daniel co-Founded Cider Security as the company’s CTO. Cider is a startup focused on securing CI/CD pipelines, flows, and systems.

Omer is a seasoned application and cloud security expert with over 13 years of experience across multiple security disciplines. An experienced researcher and public speaker, Omer discovered the Web Cache Deception attack vector in 2017. Omer leads research at Cider Security.

We hope you enjoy this conversation with...Omer and Daniel.



attackers, security, ci, defenders, production, cd, organization, environments, risks, domain, cider, application, systems, omer, people, attacks, developers, code, pipeline, daniel


Chris Romeo, Omer Gil, Tiara Sanders, Robert Hurlbut, Daniel Krivelevich

Chris Romeo  00:01

Daniel Krivelevich is a cybersecurity expert and problem solver with 15 plus years of enterprise security experience and a proven track record working with 100 plus enterprises across multiple industries. He has a strong orientation to application and cloud security. Daniel co-founded Cider Security as the company's CTO. Cider is a startup focused on securing CI/CD pipelines, flows, and systems. Omer Gil is a seasoned application and cloud security expert with over 13 years of experience across multiple security disciplines. He's an experienced researcher and public speaker. He also discovered the web cache deception attack vector in 2017. Omer leads research at Cider Security. Daniel and Omer joined us today to talk about the top 10 CI/CD security risks initiative. They put together this list of this new top 10; It has all the things that you need to understand, to know what's the risk in your CI/CD pipelines, things like insufficient flow control mechanisms and dependency chain abuse. They take us through why they made this list, what was the process for building it, and then what are the things that you're going to be able to get out of it, as a developer, as a security practitioner, as a DevOps person. We hope you enjoy this conversation with Daniel and Omer.

Tiara Sanders  01:23

You're about to listen to AppSec podcast. When you're done with this, be sure to check out our other show, Hi/5.  

Chris Romeo  01:32

Hey, folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo, CEO of Security Journey and co-host of the podcast. I'm also joined today by my good friend, Robert Hurlbut. Hey, Robert. Hey, Chris.

Robert Hurlbut  01:45

Robert, principal application security architect at Aquia and glad to be here to talk about this topic. This is something I know that we've been thinking about lately in the news, with supply chain attacks and all kinds of interesting things going on. A lot of fun.

Chris Romeo  02:04

It seems to be one of those topics that is getting a lot of attention right now. It just happened that our two guests had and via their company had posted this new top 10 CI/CD security risk thing. As I saw it went by I was like, Wow, that's so cool. I want to learn more about that. I want to understand what this project is about. We're joined today by Daniel Krivelevich and Omer Gil. Guys, we want to jump right in with your security origin stories, because that's what our audience likes to have an understanding where the people are coming from. You guys can decide who wants to go first. But tell us how you got into this crazy world of application security.  

Daniel Krivelevich  02:43

I'd like to start. I'm the CTO and one of the founders at Cider. My origin started way back in the army in Israel; army services were mandatory. I served in 8200, the equivalent of the NSA in Israel. I got my first exposure to the cyber domain in 8200 and spent about six years there. Then I jumped back and forth between defensive and more offensive positions between the defender's perspective and the attacker's perspective. I did a few years doing penetration testing, which is where I'm at OMO       (Open Market Operations). After that, I lead application security with Live Person, a big American SAST company. I spent four years with Signia, an Israeli company doing incident response. Over there, I lead the application and cloud security disciplines. My job was primarily to work with organizations on identifying their crown jewels, their critical applications, and understanding what they look like from a development and deployment perspective - then understanding how an adversary would be able to abuse those processes and apply controls to optimize the posture of these processes and prevent the next incident. Over there, I got a deep understanding of the challenges that organizations are facing on the CI/CD front, which is one of the primary reasons we set out to build Cider Security and focus on CI/CD security.

Omer Gil  04:32

Hi, I am Omer, the same as Daniel; my background starts in the army. We served in different units, but we got to know each other later when we did a security consulting penetration testing. Tennyson Yang, worked there for like four years, was a team leader there. Afterward, I moved on to Magic Leap to the cloud security team, been there for nearly four years, and managed the cloud security team. That was my last role there. Cloud security and application security were my main focus in Magic Leap. Then Daniel founded Cider Security, which he already represented. This is where I joined. Like in December 20, more than a year ago. I'm leading research at Cider Security, and my team tries to find the next big attack vectors and threats in the domain of CI/CD, which has a high focus today in the security world.

Robert Hurlbut  05:52

Daniel and Omer, the main topic today, what is the top 10 CI/CD security risk initiative? Another interesting list that you created out there, but what is it?

Daniel Krivelevich  06:09

We drew a lot of inspiration with this top 10 CI/CD risks initiative from other initiatives that we enjoyed as defenders in the past, like the OWASP top 10 and the serverless top 10, which was published by Pure Sec. Going into Cider, we understood that there's this whole new domain that's becoming more and more popular both with defenders and with attackers. I mean, we saw in 2021, or even earlier in 2020, with SolarWinds, with dependency confusion, with the code cough hack, with so many other high magnitude hacks, that attackers are gradually shifting focus to CI/CD. Defenders are asking more and more questions about how they can protect a CI/CD processes, systems, and environments. I think traditionally, when defenders when AppSec practitioners were asked what it is that they spend their day-to-day doing, the primary focus was the code, was implementing scanners that find flaws in the code, and doing secure design or secure architecture of use for the applications being built. But then CI/CD, that whole process of getting code from the developer's workstation or the SCM to production was for a long time; I wouldn't say neglected, but outside the scope of what application security practitioners were thinking about. It was also outside the scope of what was considered to be the organization's attack surface, which for many years made sense before CI/CD kicked in, before so many automations replaced manual processes before there was such a thing as infrastructures, code, and automated deployments. CI/CD has reshaped the organization's attack surface, has created endless new opportunities for attackers, and lists new ways to run malicious code in the CI to get to production. We witnessed, even before Cider, but after we started our journey with Cider, we witnessed that there are so many unknowns for defenders about what the attack surface looks like, what this new flavor of risks looks like, and what are the new questions defenders need to be asking themselves? And what are the new areas that defenders need to shed the spotlight on? That led us to the understanding that our goal, as a company that's focused on CI/CD security, is not only to build a technology that helps defenders create more secure CI/CD environments but also to create the knowledge and help defenders help the InfoSec community stir the discussion and sparked a discussion about the measures, the controls, the thought patterns that we need to adapt in order to create more secure CI/CD environments. For us, it was obvious that this type of artifact is something that we have to create to bring our knowledge forward and stir that discussion. The format of the top 10 is one that we chose simply because it already exists. It already makes sense to people. It's something that is precedented, just not in the domain of CI/CD. That's pretty much the reason why we chose to do this

Robert Hurlbut  09:47

We're going to be talking a little bit about how you arrive at the 10. But you just mentioned about, for example, the OWASP Top 10, that's changed in the order over the years. Is there a particular order of severity for this list or put it together and these are the things we saw?

Omer Gil  10:07

The top 10 risks, as we eventually said the order of them, there are set by the impact. We believe that the first one posed the biggest impact on most organizations because the consequences of exploiting these vulnerabilities and flaws pose a great impact on this kind of environment or organization. The probability that it's going to happen is higher than other risks on the list. We think that the first one is the most impactful. We also know that our other risks besides the top 10, but we know that this world of CI/CD security is so premature at the moment in most organizations that you need to start from somewhere; this is why we get at the top 10 list, something that everyone can focus on, and start protecting abandoned environments from this spot.

Chris Romeo  11:19

If I'm a developer, we have some people in our audience that are new to application security, their developers, maybe they just got out of college, or maybe even if I'm a seasoned application security person, what's the value proposition for me in checking out this document? What am I going to take away, what am I going to get from it? What's the value to me to examine this document and learn and absorb all that information?

Daniel Krivelevich  11:46

It's a good question. I think it's interesting that you pointed out developers as the audience because that was our intent as well; intuitively, the target audience is security practitioners because they are the ones that need to focus on security risks. I think with shift left, and with the whole shifting mindset of the relationship between security and engineering teams, a lot of the responsibilities and the ownership of securing engineering processes is now with developers with DevOps. I'm glad that you pointed it out because our target audience is both security and engineers. As far as the insights that engineers could have around this document are, first of all, understanding that CI/CD, the systems that they're working with, and the code that they're shipping to production has significant importance or weight in an organization's attack surface. Engineers and DevOps practitioners have a big responsibility in making sure that they're equipped with the right knowledge, the right tools, the right capabilities, the right relationships with security, to make sure that they have the appropriate security guardrails and that they're not making mistakes, and that they're not taking measures that might have the potential to increase the attack surface. Now, as far as the actual measures that they need to take, I think this document is a very good resource in helping them get acquainted with how the same building blocks that they work with on a day-to-day basis, the STM repo, or the CI pipeline, or the package and NPM, it's those building blocks that they know and use on a day-to-day basis, but didn't think of the security aspect of using them, they now understand what it is I need to think about when I define CI pipelines when I configure permissions on a repo when I configure my package manager, my NPM client to work with NPM. The more they get acquainted with the mindset, with the security building blocks, the better they're going to be eventually in instilling security as part of their day-to-day. It's not just about the actual technical bullets in the document. It's more about reading through the document and getting a better understanding of the attackers' perspective and sorting in their heads those building blocks that will allow them to think about security in a more effective manner.

Chris Romeo  14:48

You mentioned SolarWinds Codecov in the context of CI/CD systems. I'm curious with both of you coming from a pentesting, breaking background, why do you think that the attackers mindset has switched towards engineering systems? I'm from the early days of security in the 90s, when you could get into a Windows domain in like five minutes, like anybody could. They would just download a script and like, oh, look, I'm the domain admin, great. What's changed in the world to make attackers focusing on these engineering systems?

Daniel Krivelevich  15:29

This is one of our favorite questions.

Omer Gil  15:33

You just mentioned one of our favorite examples that Daniel likes to mention. We believe that today, similar to what you just said, CI/CD today is like Active Directory, as it was 10 years ago; as you said, around 10 years ago, an attacker that got access to a domain user could be any domain user, not to sell the domain admin, he could take over the domain, obtain access to a domain admin account, probably in a few minutes or hours. Not saying that it doesn't happen today; in writing assessments or actual breaches, it still happens. Today, this whole domain got a lot more mature than it was a few years ago, both from the vendor side also from the perspective of some application security vendors and security vendors in general that try to protect against these kinds of threats. Today, for attackers, it's much harder to gain access to control the domain or to gain access to other sensitive assets in the network. Because also, defenders have a lot more tools and knowledge to protect against these kinds of threats. When we see this trend happening, attackers need to find their path to sensitive assets into production, where the data is well, where eventually, the money is. They see that it's easier together today comparing to older methods like attacking or targeting the main environments. We think that today, attackers know that if they get access to an STM user account or access token, or an SSH key, or compromise an NPM package, they can easily get to production in a matter of seconds or minutes if they know what they're doing. What's even more important is that defenders don't know how to identify these kinds of attacks and don't have the tools or methodologies. It's not just that it's easier to carry out for attackers. It's also reasonable to think that the defenders won't even know that the attack occurs in real-time or even when they try to figure it out after it happens.

Daniel Krivelevich  18:32

I think what's not changed and what will never change is that attackers always choose the path of least resistance. Like you said, for many years, the path of least resistance was to gain a workstation in the domain, or user in the domain, find the group's XML file and become local admin on every single workstation in the domain including the domain controller within minutes. But that's not the path of least resistance anymore, and it hasn't been so for a while. Defenders have a wide array of products and methodologies to detect, prevent, and hunt down these types of scenarios. What we've witnessed is that even the most mature organizations with a sim and a sock and a threat hunting team and there are organizations that are extremely proactive in building their cybersecurity posture and finding the most sophisticated attack vectors. When it comes to simple and trivial tasks like shipping logs from the CI system to the Sim or looking for potentially malicious deployments to production, there's nothing there, and there are giant blind spots. What's also unique to the context of CI/CD is that the amount of opportunities for attackers over time is growing exponentially with infrastructures, code, GetOps, and all the automations. The ratio between how fast the domain of opportunities for attackers is growing versus how quickly defenders are adapting it's not linear. Think about something like dependency confusion, dependency confusion allowed white hats, but also probably hackers for a long time to run malicious code, run malware on millions of CI environments without even having any access to the environment with nothing, simply by uploading a malicious package to a public package repository, and then that's it, millions of CI systems are running malware. The ease of exploitability of some of these vectors versus how well defenders are equipped to know about, to identify, to prevent these types of vectors is what's shifting the focus of attackers to CI/CD because there are so many opportunities over there, and a relatively low level of maturity with most defenders.

Chris Romeo  21:29

When I consider the state of our industry right now, I would say, to your point, Sims and SOX and all those things; InfoSec has reached a certain level of maturity, organizationally, from a CISO down to the response and all of the pieces that go with threat hunting, the things you mentioned. I see the same thing currently in our industry where things that are AppSec and product-security focused are very immature across even the largest of enterprises because you'd think, oh, yeah, this gigantic enterprise, they probably have this figured out. No, I don't think anybody; I'm sure there are some outliers somewhere that have a good handle on this and are getting a better handle. But I think there is a lack of maturity right now on more the AppSec side, and there is a giant opportunity to provide improvements into that space.

Daniel Krivelevich  22:23

It sounds negative, or what we're saying sounds like the AppSec domain, or the defenders that are part of the AppSec domain are far behind, but I think this is the evolution of InfoSec. I mean, this is where people responsible for protecting cloud environments, or Active Directory environments, were 10-15 years ago; they were far behind the attackers, far behind the opportunities for attackers. Over time, they close the gap. This is what's going to happen with CI/CD security. This is part of our motivation to, as I said, not only build the technology but build the knowledge that will empower other groups to build strong technologies and controls. Over time, we will close that gap. But attackers are always a little bit in front of defenders.

Chris Romeo  23:18

Your least path of resistance comment makes me smile. Because I've probably said that 100 times, especially when you're talking to people that maybe aren't as familiar with security, it's like, Hey, listen, so attackers don't work, like in the movies or on TV. They don't find the most eloquent way to break into a system; they use brute force, and if they find an easy thing, they're like, Woohoo, victory, I'm in like, it's not like they are pouring over their terminal until 24 hours later, they find the eloquent hack, they could have taken the easy one, but they went with the more detailed thing. It's not the real world. It's the least path of resistance; when there's a door, they go through it. A maturing in the CI/CD space, which you all are helping to solve that problem. A maturing in that space is going to result in the least path of resistance is going to move somewhere else. Who knows where to land next, but I don't even want to try to predict that.

Robert Hurlbut  24:12

Let's dive into the top 10. What was the process for selecting these? You mentioned a little bit about in terms of severity and so forth, but how did you arrive at these particular 10?

Omer Gil  24:27

When we initiated the process, we had more than a few risks in mind that we knew should be a part of the top 10, or at least be candidates to be included in the final top 10 list. But we wanted to do a comprehensive job finding them and consulting with as many experts as we can, to eventually set up the best list that represents the actual biggest risks in this domain. We started with brainstorming, Daniel and myself, brainstorming all of the public hacks, all the hacks that went public in the recent few years, many of them happened in 2020 and 2021. But there were some CI/CD hacks that went public even before that. Though more than a few, I mean, I think it was dozens or even hundreds of public attacks. I'm sure there were a lot of attacks that never went public, but we read each one, analyze them, and started categorizing them to different risks that we believe led to these attacks to kill. Afterwards, we saw, after we analyzed all of these attacks, and also edit our own risk from our own experience, working with environments of customers from our experience, we had a big list of risks that we saw, which ones were more popular in the more high impact attacks, and which ones were less impactful. This is the place where we get the first draft of the top 10 list, started writing it down. But then we did comprehensive work, but we wanted to consult with security experts. This is where we started approaching people from around the world, from CISOs of huge companies with lots of experience from their roles, and also engineers; security engineers and DevOps engineers that we believe that we can definitely should consult with. Eventually, we had 15 people that wanted to join our live viewers list. Daniel will mention some of these viewers.


Daniel Krivelevich  27:17

We were fortunate enough to collaborate with some of the world's top experts on application security; we worked with the CISO of Atlassian, with the person that leads application security at Netflix, CISO of Lemonade, CISO of Relativity, there's a long list you can find in the top 10. But I think the general sentiment or notion behind all of this is that we wanted to make sure that if we're making this big contribution to the InfoSec community, we have a wide perspective and gained a lot of experience over the years. We wanted to make sure that it is as relevant as possible to as many different contexts as possible. This is why we were extra diligent in, first of all, identifying and reading and seeping through as many anatomies of public hacks that were exposed. You can see all the references and links in our document. We analyzed these hacks, one by one, understanding what went wrong, what capabilities the attackers had, and what capabilities the defenders had. Then had those discussions with the application security leaders, the CISOs, around the concerns that they were facing, around what they had experienced from their own personal experience. We also incorporated the dozens or hundreds of environments that we analyzed as part of our tenure at Cider. Again, we're fortunate to work with very collaborative organizations that we help secure their CI/CD environments, and we talk a lot about the architecture, the design of the environments, the type of challenges the defenders are facing, and the type of incidents that these companies had. When we put all this together, we had a very large data set that we analyzed for a long time, which ended up compiling those top 10 risks, and it took us a while to get to a point where we're in consensus because putting together 17 different opinions is not trivial, but we all realize that it's worth the effort because the final result is going to be reliant on a very wide data set and is going to be relevant to a very wide portfolio of organization types and levels of maturity. That was the process in a nutshell.

Chris Romeo  30:13

Robert and I are familiar with that as well from when we worked on a project called the threat modeling manifesto, where we got together a bunch of experts and debated, discussed, and had a lot of fun working together to build a collaborative product, we ended up when we got to the end, we were all proud of it, and we all were like, we will put our name on it. But certainly understand that it's an interesting and fun challenge to try to put that together with a group of people from different backgrounds and perspectives. At the end of the day, if you can pull it off, and you guys have, you end up with a list that nobody can go and look at that and say, Oh, that doesn't have, these people don't know anything about the real-world. That list of people you have, they're running the real world internet right now dealing with the threats that a lot of people in companies don't even know the scale and scope of what they deal with there. I want to dive into talking about a couple of these since we've set the stage about where the project came from and how it all came together. What I want to do is have an opportunity to talk about the first one on the list, insufficient flow control mechanisms. Let's go ahead and start there; give us a high-level view of what this particular item is. For me, I'm always curious about mitigations. That's always where I land; once I understand it, I'm like, Alright, now what do I got to do to fix it or get away from it. Let's start there with the first one on the list.

Omer Gil  31:41

The first risk on the list is insufficient flow control mechanisms. It's around the ability of an attacker that has got control or obtained control over one of the systems involved in assessing the environment, could be the source control management system or the CI. The ability to ship code or artifacts to production without any control or review stands in his way towards production, with the purpose of shipping malicious code and artifacts. The point of CI/CD is eventually to automate the process of shipping code and artifacts to production. When you write code that you want to do one to one production, when you push it to a repository, like a feature branch, it usually goes through a set of tests could be security tests, unit tests, integration tests, and reviews by other developers. In the build process of this code, there are more than a few stations on the way until the code eventually reaches its destination. It's all automated between different systems and pieces of code. Attackers can take advantage of it and ship their malicious code to production once they obtain access to the environment. We can talk about several opportunities for attackers. In this case, let's say that attackers obtain access to an access token of a repository for an application that hosts application code that eventually runs in a Kubernetes pod or as a Lambda function or whatever in the cloud, then you can add your code. If it's automatically deployed to production without any review or scanning on the way, then an attacker would be able to act the same as a developer and ship code directly to production. This is something that you would want to prevent. On the one hand, you want to provide the freedom to developers to be able to ship code faster production. This is the whole purpose of CI/CD, that's correct. Eventually, you don't want to prevent developers from working fast. But still, we all need to understand the impact of a case where an attacker obtains access to one of the hundreds or 1000s developers in the company. An attacker can obtain access to their username and password or access token or SSH key or compromise the GitHub app used in you're going to integrate in your organization. Once the attacker obtains just one authentication method to a repository, he can use it, if he's able to use it, to ship malicious code straight to production; that's probably something you want to prevent. We can talk about some mitigations for this.

Chris Romeo  35:19

Let's get into the mitigations now. I feel like I have a better understanding of insufficient flow control mechanisms. It's almost a generic category where a lot of different things, a lot of different components in the architecture could fit into it; it all comes down to the attacker being able to put some malicious code and drop it somewhere through the CI/CD pipeline, and in somewhere in production where they can get to it. What do we do? How do we help ourselves to not have this particular issue?

Daniel Krivelevich  35:57

Before the actual tactical measures, I think, and this goes back to your point about what type of things developers need to think about when they're reading the document, one of the special things about the mindset that we've adopted at Cider and general, the place where the industry is going as far as CI/CD security is that we are adopting the understanding that the source control system, the CI systems, the systems that were perceived for many years as outside the scope of what security teams need to be concerned about, they are now the path to production. They are now the best avenue for attackers to get to production. We need to consider, for example, GitHub or any other source control system; they are, for many years now, no longer a version control system that is is a system where developers collaborate together on code, with versions, and so on and so forth. No, it's a gateway to a series of automation that ends up in an artifact being deployed to production; this is the front door to production. Once you begin understanding that the source control system, or the system that does the deployment to production, is a system that contains the credentials that are ultimately used for deployment to production, credentials that allow accessing production, we need to treat them in the same way that we treat any other system that stores credential to production, we need to treat them like we treat the domain controller, or the way we treat the vault, the Cyber-Ark or whatever Pinpad solution we're using. Once we adopt that mindset, once we understand that our GitHub or whatever source control system we're using, or our CI systems, are systems that we need to be as diligent around security as we are with, as I said, the domain controller or any other crown jewel, then there are a lot of measures that we need to be proactive and understanding how to implement around how to secure a repo, how to prevent a reality where one single set of permissions, one push to a repo is sufficient to ship malware to production. Because then that's the same as one set of domain permissions, one low privileged domain user being one event away from being a domain admin, it's very similar. That mindset, once you adopt it than actually implementing the measures, once you're comfortable with taking those measures and taking those strides to protect the systems, the settings, the configurations, they are available. The shift in mindset is, I think, what organizations find the most challenging, but let's talk about the actual tactics.

Chris Romeo  38:52

Before we get to mitigations, I would want to share. What you just said, something just dawned on me that I can't believe I didn't have this thought before. All of those years of us saying, oh, that's just internal, this is a system that's just internal, like, it's never going to be on the internet, we can have a different standard of security for those things that we only keep on the inside of the network versus those things that are on the outside. It sounds like what you're saying now; we're in a CI/CD world where those things aren't on the inside anymore. There is no physical isolation anymore between those things. They're engaged in various internet components and whatnot. That's to your mindset point; that just dawned on me there; I was, like, I just had a moment I was like, wow, I hadn't thought of it like that. But that's the reality of the world we live in. Let's talk mitigations. Now, how do we solve this first challenge?

Omer Gil  39:49

There are many different measures that you can take to prevent a direct push of malicious code and artifacts to production. One common practice is production rooms, for example, in the source control management system side, something that we see that nearly any organization implements, at least in the most important repositories and branches, that are eventually used in production. Branch protection was preventing engineers from directly pushing code to sensitive branches that are then used in production. For example, you have a repository with the main branch; when you push go to the main branch, there is a pipeline being triggered, building, testing the code, and then shipping it to production. Usually, most engineers, all engineers, can directly push code to this sensitive branch, but they have to push go to a feature branch and then create a pull request, get approvals from doing the peer reviews, and only then they're able to introduce and merge the code to the main branch which will ship it to production. I think that this common practice, which is eventually a manual review, not necessarily though some other practices there, you can also add security scans in this phase of creating a pull request. But it's a phase of manual review; when people on your team review the code that you introduced, this gives a hard time for attackers introducing their code that will be pushed through automation to production - making sure that any branch that is eventually shipped to production is protected. You have to receive at least one review from your peers in order to merge it. That's a common practice that many organizations use. You need to make sure that you have these practices in place on all of your sensitive branches in all of your sensitive repositories in your organization. This could be a challenging task when you have 1000s of repositories or hundreds or 1000s of pipelines in a medium-sized organization. It could be hard to achieve. But that's a common practice today. Another example is that, let's say that you protected your sensitive branches, you have security scanning, and when the pipeline is triggered, let's say that it builds a container image and stores it in a container registry. Then this image is fetched and deployed to production by a deployment pipeline. You can say, okay, there are reviews and scans in place to make sure that this image was viewed and no vulnerabilities inside when this image was stored on the container registry. But what happens if some user that the attacker obtains access to has direct access to this container registry, and the attacker can directly store a malicious image del, that will be then fetched by the deployment pipeline and deployed to production. In this case, this whole pipeline in the protection del become worthless because an attacker can bypass the pipeline afflictions and put malicious and create a malicious artifact in the registry, which will be deployed by another pipeline later on. You need to think of all the moving parts in the pipeline from code to production. It's not just the first phase of triggering the pipeline, or the connection between the repository and the CI, there are many moving parts, and it looks different in any organization. It's challenging for an attacker to map all these different attacks and understand how it looks in the organization. But it's possible, and this risk, the insufficient focus on mechanisms, risk details, and various types of attacks, needs to be considered.

Chris Romeo  44:19

It's an argument for threat modeling the CI/CD pipeline and the top 10 that you've put together is a great input to people like Robert and I that are Threat Modeling fans to be able to absorb. Another pitch for people to go out and examine this document is this is now input for you as you're looking at a CI/CD pipeline to start looking for, what are the threats? How are we going to mitigate these? I'd love to spend the rest of my day and go through all 10 of these. I'm not joking. I would love to stay here and do an eight hour podcast where we go through all 10 of these. But I know we don't have time to do that. We wanted to go through one of these so that our audience gets a feel for what they're going to find when they go, there's nine more of these that we haven't even talked about yet. This is an advertisement for the top 10 CI/CD because you saw one, now you can go absorb the other nine. I really had two more questions. I know we're getting towards the end of our time here. I wanted to make sure you got a chance to talk about CI/CD goat. Daniel, at the beginning, when we were talking about the impact for developers and how that comes together, tell us the philosophy that you have about what CI/CD goat is going to do together with this top 10 list to really help developers understand this stuff.

Daniel Krivelevich  45:39

We realized as defenders, before Cider, that if you want to be a good defender, you have to adopt the attackers perspective, and you have to be familiar with the way attackers think. Otherwise, what you're doing is never going to be relevant to your risk landscape. Beginning to adopt the attackers perspective with CICD is learning the theory, understanding the building blocks that comprise the way attackers look at CI/CD as an attack surface. For that, I think the top 10 is a great artifact. If you don't know the way attackers think and what type of risks attackers are looking to exploit when they look at CI/CD, then probably reading the top 10 will be beneficial in beginning to think like an attacker, but obviously theory is never enough. You have to practice the tactics and and really try to think like an attacker when you're working these environments, and then fix these environments hands-on to understand what are the measures that organizations, like what what measures really make a difference. This is precisely why we created the CI/CD goat, the CI/CD goat is deliberately vulnerable CI/CD environment. It's a full blown environment that can be run locally on the machine of anyone who clones to get a project with a source control system, a build system, a build agent, and a full blown environment. That enables users to basically deal with 10 challenges of varying levels of difficulty. Each one represents different types of risks within our top 10 CI/CD risks. Once you've gained the theory, with the top 10 CI/CD risks document, then practicing those risks, trying to think like an attacker, trying to enact these scenarios, and then trying to fix them as the defender and understanding how the measures I take, as a defender, are now preventing those same attack vectors that I just enacted as an attacker is what we believe something that is going to make a significant leap for defenders, not just security practitioners, but also DevOps engineers, anyone who's interested in securing engineering environments. That's going to make a very significant leap in the knowledge and understanding and experience these people have with securing CI/CD environments. Do you want to add anything?

Omer Gil  48:43

The goat is meant for everyone, not just for hackers or people that like capture the flag challenges, different levels of difficulty, between easy to hard. There are hints that you can use. It's super fun to solve. You can easily in two minutes, you can have the entire set up on your machine. It's all Docker-based. It's really easy to start working with it, and I encourage you all to try it.

Chris Romeo  49:28

Love it. The call to action I'm going to throw back to our audience right now is, you've heard about the top 10 CI/CD risks that Daniel and Omer have put together via Cider. Go out, check it out, read it, and absorb it. You heard a number of reasons why developers, security practitioners, DevOps, everybody that's involved in this process needs to absorb this information and gain this awareness because we know as we talked about, least path of resistance right now is the CI/CD pipeline. Daniel and Omer, thank you for sharing this with the industry and sharing your knowledge with us today. We look forward to following along with all the other cool things you'll do in the next year.

Daniel Krivelevich  50:12

Thanks. It was very fun talking with you guys.  

Omer Gil  50:14

Yeah, thank you.

Chris Romeo  50:16

Thanks for listening to the application security podcast. You'll find the show on Twitter @AppSecPodcast and on the web at You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, with application security; there are many paths but only one destination.  

Need more information about Security Journey? Get in touch.

Ready to start your journey?

Book a Demo