Tech Debt Burndown Podcast Series 1 E5: The Metrics

Posted on Sunday, May 2, 2021
How do you measure technical debt? This week, Chris and Nick discuss the metrics around technical debt, and why there are so few of them.

Show Notes

How Do You Measure Tech Debt? What Are The Metrics That Work?

Recording date: March, 2021

Download at Apple Podcasts, Google Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

‘I think it’s really less about wasted time, and more about getting into flow state: is your tech debt preventing flow from happening?’ - Chris

How do you measure technical debt? What are the metrics you use?

Chris hasn’t found the magic formula. Over the past few years Chris has been acutely aware of the quote from Accelerate about not trying to inflict metrics on people before the culture change happens, they’ll game the metrics. Nick agrees, and also says he has the basic problem that coming up with the right metrics is hard.

They agree that time is a really bad one: trying to quantify tech debt around the time you spend on it actualy doesn’t work (though it’s not worthless).

Some organizations build a bit of burndown into every sprint, but Chris isn’t seeing that widespread - but Nick says that after about a year, his shop is finding ways to do just that.

It took a year of seeing the results of concerted bursts of burndown, and a company-wide Objective and Key Result (OKR) on Trustworthiness. One challenge he has is that it’s hard to tie in the minds of senior executives information security (which is what Nick does) to availability and scalability as opposed to firewalls. But his colleagues are making great progress.

Nick thinks the most useful thing he’s done in the past year is to embed engineers, developers, SREs and security people into standups as often as possible to try and find the overlapping interests. And that has led to some success around thing together tech debt they need to code around and the performance of engineering squads.

Chris spent a bunch of time recently looking at Chaos engineering and how to introduce it into what they do, but he’s finding that if you’ve not already got to the stage where you can do good quality progressive engineering you’re just not ready to do Chaos engineering. Similarly, Nick thinks that applies to Eng teams that find themselves working harder but not having a commensurate increase in productivity.

What Chris sees in the bottom 50% of companies is related to a Charity Majors comment that they have the engine underwater and they have to work ever harder to see that they need to do something to drain the water that is flooding you.

When technical people have to talk to non-technical people about this, people who are smart but don’t see the tradeoffs in feature development, the non-tech people don’t understand the tradeoffs. Not that they don’t want to understand, but that they don’t see it. so the quest for metrics is really about finding Indicators Of Tech Debt.

Chris points out that time is bad as a measure because for engineers, it’s really less about time and more about getting into flow state: is your tech debt preventing flow from happening?

He mentions this is related to cognitive load, and that he’s been following the work of Team Topologies, whose principles Matthew Skelton and Manuel Pais have been spending years with organizations starting the DevOps topologies, finding patterns and anti-patterns and how work flows betwen those teams. They feel teams need to be organized to minimize cognitive load.

It seems to be about putting boundaries around the members of teams so that they can actually apply their brain power to the problem at hand in the context of their work, and Chris thinks we can start to reason is that tech debt is adding a constant cognitive load that everyone has to deal with.

This is a self-inflicted wound: rather than just doing a task, they have to remember oral histories of “when you’re trying to make something in this environment, here are the 24 undocumented things you have to look out for or they will bite you.”

Nick and Chris raise the idea that tech debt is not necessarily just “old stuff” because mainframes are just often not tech debt, because they still work.

Tech debt is more tied to the fragility of environments.


Chris Swan

Chris Swan

Chris is a frequent speaker on topics such as serverless, DevOps, cloud, containers, security, networking and the Internet of Things. He’s also a cloud editor for InfoQ and a contributor to open source projects such as Docker, CoreOS and DXC’s Online DevOps Dojo.

Nick Selby

Nick Selby

Nick provides information security, disaster- and cyber incident-readiness assessments at Fuzz Technology, a subsidiary of EPSD, Inc. From 2021 to 2023, Nick served as VP of the Software Assurance Practice at Trail of Bits (where he was the voice and executive producer of its podcast), and from 2019 to 2021 as Chief Security Officer at Paxos Trust Company.

From 2018 to 2020, Nick served as Director of Cyber Intelligence and Investigations at the NYPD Intelligence Bureau, where he helped the department understand how it investigates online, and how Cyber Enabled crime affects New Yorkers.

In 2005 he founded the information security practice at industry analyst firm 451 Research, (now S&P Global Market Intelligence) where he served until 2009 as 451’s Vice President, Research Operations.