Tech Debt Burndown Series 1 E4: Wendy Nather on What Works in Security

Show Notes

What Works in Security? The Cisco Security Outcomes Study

Recording date: December, 2020

Download at Apple Podcasts, Google Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Chris and Nick welcome their old friend Wendy Nather, Head of Advisory CISOs at Duo Security (now Cisco), to the podcast to discuss her work on the Cisco Security Outcomes Study. Now, technical debt isn’t only about security, but they’re related - and one of this report’s most solid conclusions addresses it specifically.

When Wendy began her work at The 451 Group (now 451 Research), she started asking her CISO friends questions: ‘If you’ve just taken a CISO job at a company that had no security, what would you buy?’ The answers fascinated her. Now at Duo (Cisco), she set out to formalize research that was released as Cisco Secure’s Security Outcomes Study.

Security professionals are great at doing benchmarks, but it turns out that our peers may just be bad at security.

So for this report, Wendy and her team sought a correlation between things we struggle with in security practices, and the outcomes of the programs they run to address those.

The research was completely technology and vendor free. They queried security professionals (using YouGov to run the surveys) from around the world about their security practices across 25 areas, then brought in Wade Baker’s team from Cyentia to analyze the data.

The methodology is actually one of the best things about the report, because it is decidedly, simply put, not bullshit.

The study was double-blind, with 4800 respondents not knowing who was asking, and Cisco not knowing who was answering. The question was, “What appears to correlate between security practices and outcomes?”

As a great study will do, it raises more questions than it answers, but it also is a report that, by applying analytic rigor at the problem, it is just a great report.

Nick points out that there are several issues that security pros take as gospel are just not true. We won’t spoil it but tropes about people who own compliance, people who own apps, security awareness, identification of the top cyber risks … All of these have very interesting real-world correlations to results.

Meeting Compliance Regulations

As it turned out, for example, having someone own compliance turned out to not be correlated with better compliance results, but just buying new IT gear raised the likelihood of success in meeting compliance goals by more than eight percent. Other activities correlated with better compliance outcomes included well-integrated tech, timely incident response, having a sound security strategy, and setting deadlines for remediating vulnerabilities… There’s lots more.

Chris points out that this makes sense: those programs mentioned in having a positive correlative outcome are all about reaching across the organization and speaking with people.

The Two Practices That Most Strongly Correlate

Wendy is quite sensitive to the idea that the two practices identified as most likely to correlate to better outcomes, Proactive Tech Refresh and Well Integrated Technology, are of course seen by the more skeptical of us to have been directly driven by the fact that Cisco is a vendor of that stuff. Wendy insists (and Wade has insisted) that the double-blind nature of the survey administration and the analysis means that it actually was independently learned, not sponsored. We believe it. Wade tweeted about it recently:

2/2 My answer is always the same: A third party survey firm fielded the study, respondents didn’t know Cisco sponsored it, and @cyentiainst did the analysis to obtain that funding. I’m sure they like the outcome but they didn’t influence it. Get the study: https://t.co/ffYkfyhrf1

— Wade Baker (@wadebaker) April 24, 2021

Also, the survey didn’t ask, ‘Do you think buying new stuff makes stuff better?’ they asked separate questions. The correlations were stronger than they thought.

What Works?

But Chris happens to think that a lot of this is hogwash. Chris thinks that, as an example, tech refresh is already built in to the market (because virtualization and cloud), but Nick points out that Chris has lived five years in the future as long as Nick has known Chris, which is 16 years. He said that Chris, Wendy, and Nick have been very lucky, but that most people don’t have the ability to have contemporary kit and work in environments where spinning disks are still the norm.

Nick also pointed out that “IT Refresh” is unclear - do we mean servers and firewalls, or laptops and printers? And Wendy pointed out that the questions were in fact vague: we don’t know whether the question means any or all of those things. Which supports the idea that a tech vendor didn’t put its thumb on the scale for this question.

Sentiment

Chris speculates that people who feel good about their company will say more positive things here. And there is a real discussion about whether shinier MacBooks help with talent retention and compliance success.

Wendy says that some of this seems obvious, but some isn’t. How getting sufficient budget helps with executive buy in may well be a circular argument.

Or take Identify Top Cyber Risks. That feels outlandish to Chris, who would have thought it would seem that identifying the top risks you face would be top of the list, and it turns out to be worthless.

We think that the podcast is a good listen (well, we would) and that the report is absolutely worth a read.

We especially like that the report is available to read with no registration required.

Guests

Hosts