Recording date: May 19, 2021
“Where is the time going? Are your people spending 80 percent of their time on crap that doesn’t move the business forward? Crap that is toil, that is miserable, that should be automated away? Or are they spending 80 percent of their time on stuff where they’re creatively solving problems, that moves the business forward?”
With her work at Honeycomb, Charity is now known as a leader - Chris said, “the queen” - in Observability. “I have learned that if you can’t see what’s going on, you’re fscked,” Charity tells us, “So much of what we are doing in tech is the hard way. It can be so much easier.”
Chris begins with Charity’s definition of “Ops”, which he says is the best he’s heard. We then ask Charity about technical debt, which Charity feels are the investments you make to get to the next step in your business.
One thing Nick finds interesting is that Charity’s idea is that there can be good debt in life (like going to college and learning life lessons) and bad debt in life (like using a high-interest credit card to buy a big flat screen TV) and in the same way there are tech debts that are good and bad. It’s about doing tech debt deliberately.
“We should all be so lucky,” Charity says, “as to survive so long as to have to deal with our choices.”
It’s about making the correct investments.
So much of the complexity of the application is now bound up in the operations. That means we need to have full ability to see within each aspect of the operations to see what is happening. Charity discussed her work at Facebook using a tool called Scuba, and how its capabilities made a huge impression on her.
After Facebook, Charity realized that she couldn’t engineer without this kind of visibility, and established Honeycomb. Charity said the term came from mechanical engineering, and politely says they are rather chuffed that so many people have begun to use the term for essentially all visibility.
In the beginning Honeycomb thought it would be a niche thing, but it seems that most people in all kinds of ops are finding this useful, and the company is doing very well. Nick mentioned that industry marketers have co-opted the term in the same manner that Big Ag has co-opted the term “Organic” - “Observability” now is applied to any kind of visibility/telemetry tool.
Chris asks how customers are using observability to understand their tech debt, and like a good CEO, Charity replies that it isn’t that It helps with tech debt, it’s that it helps with everything - but especially around understanding how your code feels to users, and shines a light on tech debt by helping the humans understand how they are spending their time - on toil, or on creatively solving probelms to move the business forward.
Nick asks about metrics of tech debt, and how you separate toil and misery form good work. In addition to echoing us on Dora metrics, Charity suggests that the best metric of performance as a generic signal is how often you get pulled in after business hours to solve problems.
But more broadly, if you can’t ship your software using CI/CD in 15 minutes - you have an idea and a few minutes later you are looking at it live - if you can’t do that very quickly, within the golden interval, that is the most important thing any technical leader can do, setting it up so that the pathologies don’t have the time to creep up and overwhelm your team.
And that is often not an engineering problem, it is one of political will.
Charity Majors is an operations and database engineer and sometimes engineering manager. “Right now I am the CEO and cofounder of Honeycomb, where we build observability for distributed systems. (“Monitoring” doesn’t have to be a dirty word; give us a try.)
“Until recently I was a production engineering manager at Facebook. I spent 3.5 years working on Parse (both pre and post-acquisition by FB). I also spent several years at Linden Lab, working on the infrastructure and databases that power Second Life. I am the co-author of Database Reliability Engineering by O’Reilly.
“I was a classical piano performance major in college, but dropped out because it turns out I prefer not being dirt poor. I have been building systems and engineering teams ever since.
“I love startups, chaos and hard scaling problems, and somehow I always end up in charge of the databases.”
Chris is a frequent speaker on topics such as serverless, DevOps, cloud, containers, security, networking and the Internet of Things. He’s also a cloud editor for InfoQ and a contributor to open source projects such as Docker, CoreOS and DXC’s Online DevOps Dojo.
Nick is Vice President of Assurance at Trail of Bits. He was formerly Chief Security Officer at a financial technology firm. Prior to that he was Director of Cyber Intelligence and Investigations at the New York Police Department. He is co-author of many books, including Cyber Attack Survival Manual; In Context: Understanding Police Killings of Unarmed Civilians and Blackhatonomics: An Inside Look at the Economics of Cybercrime.