Tech Debt Burndown Series 1 E3: Supply Chain Vulnerability & Tech Debt

Show Notes

Where do you start looking for tech debt?

Recording date: April 17, 2021

Download at Apple Podcasts, Google Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

This is a common question, and Chris and Nick discovered that they do it differently.

Chris describes how he views tech debt from the perspective of his continuous integration and continuous delivery pipeline: “How quickly can I release, and what’s getting in my way?” He discussed that recently, Charity Majors has been discussing the importance of a quick feedback cycle for developers:

people really do leave their jobs over build times. i am not making this up.

the impact your deploy pipeline has on developer productivity and happiness is HARDCORE. https://t.co/z49xUxXWsk
— Charity Majors (@mipsytipsy) April 15, 2021

Basically, this comes down to, ‘you’ve got 15 minutes to build’ in order to be good at this, and this brings us to the Theory of Constraints thing: what things are blocking me from getting to that 15-minute build time, and how do I get rid of them?

Nick says he comes at this from a more traditional infosec approach, basically, how do I get in, how do I move around? Chris points out that this reveals two prevalent schools of thought in the industry when it comes to building software: “How do I make this work?” and, “How do I break this?”

How do we make it work leads to “How do I make this resilient”

Both of these approaches seem to meet up at resilience, which brings us to the broader conversion about software dependencies and dependencies management, which itself is based on the fact that today,

“software is mostly assembled as opposed to being constructed fresh from a clean sheet of paper.” –Chris

As we go about that process, we need to get open source, and buy tools, and many are pretty janky - and the janky tool is the one that gets you.

Chris thinks there is better tooling coming, and Nick is far more skeptical of that.

This gets into the fact that libraries get poorly maintained or abandoned and can in fact be taken over by malevolent actors, and now a trusted source is pushing malware.

Everyone understands that but to what extent do you have proper insight to what you’re assembling?

Supply chain breaches are top of the pops. But what are supply chain breaches?

What’s going on?

Supply chain attacks happen when people get into these dependencies, which are then incorporated, or they get into the build chain to insert software that shouldn’t be there.

They can also be broader than just development operations, attacking necessary business tools that the company uses as a whole.

Probably the earliest great example of this that we remember is the RSA SecureID breach of 2011. Chris remembers that the term Advanced Persistent Threat was particularly bogus, because it isn’t the threat, it’s the attacker that is sophisticated and advanced (Nick throws in a plug for an article he and Scott Crawford wrote in January, 2010 called, ‘It’s The Adversaries That Are Advanced and Persistent')

But there’s been a change in dynamics in the past few years - where it felt like the entire SecureID breach targeted Lockheed Martin, now it feels like “stage the attack first and then choose your targets" rather than choose your target and then stage your attack.

If we are a contemporary software company or even a business, and we’ve done the right thing by outsourcing the stuff that isn’t our core competency, we find that there’s lots of stuff that isn’t everyone’s core competency - email servers, ticketing systems, remote helpdesk.

So even though we all use a much more diverse set of universal tools, each tool in its category is just as much a single point-of-failure as was SecureID - we all use the same instant messenger, the same ticketing system, the same what-have-you, so targeting can be conducted on an industry vertical-level by attacking a tool that horizontally attacks the whole industry.

How do we protect ourselves against these third party risks?

Spreadsheets.

Chris talks about the Hyperscalers like cloud and the starting point of the Shared Responsibility Model in which security and compliance are the shared responsibility between the hyperscaler and the customer.

We’re now at a point where we rely on the Hyperscalers like AWS for security.

Nick points out, though, that AWS has a different idea about what “secure” means from its customers - them being secure is just not the same as us being secure, and Chris rightly points out that this is the other side of the shares security model. But the point of handover is sometimes unclear to the customer.

Many things that should be done are best practices are not enabled by Amazon by default, and this is a huge issue, which Chris points out is completely right and behind the idea that we read the dunderheaded thing of the week every week in Corey Quinn’s Last Week In AWS newsletter.

But if you want the audit trail, you need to flip the bit that starts the taxi meter to fill your other S3 bucket with logs.

Adversaries Have OKR’s, Too

In looking at attacks, it’s important to realize that attackers will always seek the path of least resistance as they target you - and often these days the path of least resistance is the tool. Chris mentions another Economics of Information Security Paper, Where Do All The Attacks Go? in which the question is asked, ‘If security is so awful, why isn’t everyone attacked every day?’ and the answer of course is that even attacker groups have bosses to please and budgets to live up to. Targeting is hard.

Threat Model Your Supply Chain Tools

So the idea to use supply chain attacks being in the news is to use them to justify the time it takes to consider something: given a supply chain attack against our CI/CD tool, what permissions would it have, and how can you follow those permissions throughout your CI/CD pipeline.

All too often we bring in these external tools to cover up what we’re not good at, which would be fine if they were better than they actually are. Threat modeling through a theoretical supply chain breach can help you identify new forms of tech debt.