By Brendan O’Leary, GitLab Staff Developer Evangelist and CNCF Governing Board Member
For almost all of human history, we have organized our communities and created specialties based on what is most readily available to us. And as long as communities have had a surplus in resources or supplies, there have been countless wars fought solely based on the ability of one side or the other to control or master the resources of their adversaries. Today, we might call these “supply chain attacks.”
In The Art of War, Sun Tzu writes, “The line between disorder and order lies in logistics.” Those words are as valid today as they ever were. We often take our modern inventions and comforts for granted, but it’s critical to remember that basically all modern conveniences are heavily reliant on ever-increasing and complex supply chains. And in a globalized world, those supply chains are also global. In the technological age – the age where “software has eaten the world” – software is just as reliant on supply chains as any other part of our human infrastructure (and software can be eaten by its own supply chain).
That’s what happened in the Solarwinds hack. That hack itself wasn’t a single event. It was a sophisticated, coordinated, focused attack that took months to execute. And it wasn’t conducted against the main targets – Solarwinds, in this case, was part of a supply chain for the real targets: high-profile, Fortune 500 companies and U.S. government agencies.
Instead of attacking head-on or through some sort of inflammatory attack aimed directly at their adversary, the attackers chose to hide in plain sight, blend in with the other software deployed, and then slowly exploit that point of entry for their real goals. This attack method, just as any supply chain attack throughout history shows us, is not a new one.
Attackers gained access Solarwind’s build system. Solarwind’s customers relied on the software they produced, installed it with expansive permissions across their fleets, and accepted updates directly from Solarwinds. But for months, those updates were controlled by the attackers – giving them administrative access to machines across various government agencies and untold enterprises. Given the manual way that continuous integration and continuous delivery (CI/CD) was configured, it was relatively easy for the attackers to influence the build process once they had accessed the build infrastructure. At first, they just tested this capability, releasing what is called SUNSPOT in September of 2019 to see if they would be able to inject code and have it get through undetected.
A few months later, they decided to release the real payload: SUNBURST, which took Orion and “trojanized” it, allowing the attacker to monitor and attach to any system that the infected Orion update was loaded onto. From there, the attackers would deploy a third type of malware called “Teardrop” that was installed via the backdoor that SUNBURST opened up for them.
No one is immune to attack
This practice of using dedicated VMs for the build and certification process is not unique to Solarwinds. As the CEO of Solarwinds aptly pointed out, they aren’t necessarily the only vulnerable party here. Many other enterprises and software companies use similar build processes and would be susceptible to similar attacks against their CI/CD infrastructure. In fact, most enterprises’ software have a lot of dependencies that exist in our code, so many that we don’t even think of them all as important.
First, it’s important to understand what I mean by “dependency.” One of the most obvious understandings of a dependency is an open source dependency. These are the libraries that, in theory, help us move faster to the end goal of solving a problem with the software we are writing. This type of dependency can be found everywhere.
But a software dependency doesn’t just stop there – there are also the third party libraries from other vendors one might use for their API. Those libraries may have dependencies of their own. And this is just part of the code we ship. What about how it’s shipped?
The DevOps process and how we handle it are also part of what we depend on to get code into production. Are the systems we build on the same ones that have rights to manipulate the production environment? What if those systems are compromised? Would it be better to separate out the build and test functionality from the deployment methodology or packaging of our code for our customers? And after all of that, there’s the production environment itself which comes with other parties and libraries of its own.
The sum total of all these dependencies that are meant to help us build code faster, ship to production sooner, and make our customers happy can be best described as a “software supply chain.” If we aren’t sure our entire supply chain is secure, we run the risk of becoming the next victim of an attack à la Solarwinds. So how can we identify and mitigate any potential risks to avoid this?
What is measured can be improved
The first step is to protect your supply chain and prevent risks is to identify all of the various dependencies, process dependencies, systems, security schemas, and environments involved. This exercise alone may uncover some previously unknown issues as well.
Once you have a decent understanding of your supply chain, you’ll need to set up proactive management of those dependencies moving forward. This is not an activity that happens once, tightens up your boundaries and securities, and then slowly degrades over time to the same state it was in when you began. Automate as much of the proactive management process as possible. Quickly understanding where any known issues that arise may impact your particular supply chain is one of the most critical aspects of security.
The typical vulnerability goes undetected for four years. It takes another 2-3 months for teams to actually patch their code once the vulnerability is identified. This entire length of time has provided attackers a surface to attack your supply chain, so whatever you can do to shorten the length of time from an issue being identified to it being resolved is one of the most productive things your team can do to secure your supply chain.
Supply chain defense-in-depth
To mitigate the four years the vulnerability was left undetected, I recommend supply chain defense-in-depth. Defense-in-depth is a concept in cybersecurity where one layers multiple defenses so that if one method of security controls is compromised, or one vulnerability is exploited, you can limit the impact or stop the attacker with other means.
For most organizations, this type of security is more a journey than a direct route. For less mature organizations, even the idea of supply chain security, much less a defense-in-depth strategy may seem far-fetched. For those organizations, just establishing security controls, ensuring teams adhere to policy, and securing the perimeter might be top of mind. But I believe that every organization, mature or not, can take proactive steps on all fronts. After all, there are really only two basic types of vulnerabilities: those that are known and those that are unknown and waiting to be discovered.
If defense-in-depth planning feels intimidating, here are some things you can do today to make your organization less susceptible to supply chain attacks:
- Understand your supply chain. If your team doesn’t already have a clear diagram showing all the various dependencies in your chain, draw your own, then publish it. It doesn’t have to be perfect, just make sure you disperse it to your team. Once it’s written down, even if incomplete, it will be much easier to iterate on.
- Know all of your dependencies, direct and transient, that are involved in your environment (not just “production” proper, but all of the infrastructure involved in getting code to production, like CI/CD, staging environments, artifact stores, and more).
- Make security a key part of your development process. This is critical to ensuring that you have auditable, repeatable security where you can add protection as you learn of additional chains that require increased vigilance.
- Remember this will be an ongoing process. Establish a plan for how your teams will continue to improve your supply chain security posture. Eventually, you’ll need to ensure that as many layers of defense-in-depth are in place as possible. But along the way, make sure to understand all of the various dependencies so that you can always be working on the weakest link in your chain.
While no one can ensure that all of the dependencies we use are risk-free or perfect, we can work to ensure we understand where our chain is weakest, and put reinforcements and depth of defenses there to help our organizations succeed.
To hear more of Brendan’s insights and advice for supply chain management, catch his talk at KubeCon + CloudNativeCon on October 12, 2021. Register here.
##
To hear more about cloud native topics, join the Cloud Native Computing Foundation and cloud native community at KubeCon+CloudNativeCon North America 2021 – October 11-15, 2021
ABOUT THE AUTHOR
Brendan O’Leary, GitLab Staff Developer Evangelist and CNCF Governing Board Member
Brendan O’Leary is a Senior Developer Evangelist for GitLab who connects with developers, contributes to open source projects, and shares his work with about cutting-edge technologies on conference panels, meetups, in contributed articles and on blogs.
