Tech Debt Burndown Series 1 E4: Wendy Nather on What Works in Security

Posted on Saturday, Apr 24, 2021
When Wendy Nather began work as an information security analyst, she started asking her CISO friends a question: ‘If you just took a CISO job at a company that had no security, what would you buy?’ The answers fascinated her. Now at Duo (Cisco), she set out to formalize research that was released as Cisco Secure’s Security Outcomes Study. Wendy discusses the process on this week’s podcast.

Show Notes

What Works in Security? The Cisco Security Outcomes Study

Recording date: December, 2020

Download at Apple Podcasts, Google Podcasts, Spotify, iHeartRadio, Spreaker or wherever you get your podcasts.

Chris and Nick welcome their old friend Wendy Nather, Head of Advisory CISOs at Duo Security (now Cisco), to the podcast to discuss her work on the Cisco Security Outcomes Study. Now, technical debt isn’t only about security, but they’re related - and one of this report’s most solid conclusions addresses it specifically.

When Wendy began her work at The 451 Group (now 451 Research), she started asking her CISO friends questions: ‘If you’ve just taken a CISO job at a company that had no security, what would you buy?’ The answers fascinated her. Now at Duo (Cisco), she set out to formalize research that was released as Cisco Secure’s Security Outcomes Study.

Security professionals are great at doing benchmarks, but it turns out that our peers may just be bad at security.

So for this report, Wendy and her team sought a correlation between things we struggle with in security practices, and the outcomes of the programs they run to address those.

The research was completely technology and vendor free. They queried security professionals (using YouGov to run the surveys) from around the world about their security practices across 25 areas, then brought in Wade Baker’s team from Cyentia to analyze the data.

The methodology is actually one of the best things about the report, because it is decidedly, simply put, not bullshit.

The study was double-blind, with 4800 respondents not knowing who was asking, and Cisco not knowing who was answering. The question was, “What appears to correlate between security practices and outcomes?”

As a great study will do, it raises more questions than it answers, but it also is a report that, by applying analytic rigor at the problem, it is just a great report.

Nick points out that there are several issues that security pros take as gospel are just not true. We won’t spoil it but tropes about people who own compliance, people who own apps, security awareness, identification of the top cyber risks … All of these have very interesting real-world correlations to results.

Meeting Compliance Regulations

As it turned out, for example, having someone own compliance turned out to not be correlated with better compliance results, but just buying new IT gear raised the likelihood of success in meeting compliance goals by more than eight percent. Other activities correlated with better compliance outcomes included well-integrated tech, timely incident response, having a sound security strategy, and setting deadlines for remediating vulnerabilities… There’s lots more.

Chris points out that this makes sense: those programs mentioned in having a positive correlative outcome are all about reaching across the organization and speaking with people.

The Two Practices That Most Strongly Correlate

Wendy is quite sensitive to the idea that the two practices identified as most likely to correlate to better outcomes, Proactive Tech Refresh and Well Integrated Technology, are of course seen by the more skeptical of us to have been directly driven by the fact that Cisco is a vendor of that stuff. Wendy insists (and Wade has insisted) that the double-blind nature of the survey administration and the analysis means that it actually was independently learned, not sponsored. We believe it. Wade tweeted about it recently:

Also, the survey didn’t ask, ‘Do you think buying new stuff makes stuff better?’ they asked separate questions. The correlations were stronger than they thought.

What Works?

But Chris happens to think that a lot of this is hogwash. Chris thinks that, as an example, tech refresh is already built in to the market (because virtualization and cloud), but Nick points out that Chris has lived five years in the future as long as Nick has known Chris, which is 16 years. He said that Chris, Wendy, and Nick have been very lucky, but that most people don’t have the ability to have contemporary kit and work in environments where spinning disks are still the norm.

Nick also pointed out that “IT Refresh” is unclear - do we mean servers and firewalls, or laptops and printers? And Wendy pointed out that the questions were in fact vague: we don’t know whether the question means any or all of those things. Which supports the idea that a tech vendor didn’t put its thumb on the scale for this question.


Chris speculates that people who feel good about their company will say more positive things here. And there is a real discussion about whether shinier MacBooks help with talent retention and compliance success.

Wendy says that some of this seems obvious, but some isn’t. How getting sufficient budget helps with executive buy in may well be a circular argument.

Or take Identify Top Cyber Risks. That feels outlandish to Chris, who would have thought it would seem that identifying the top risks you face would be top of the list, and it turns out to be worthless.

We think that the podcast is a good listen (well, we would) and that the report is absolutely worth a read.

We especially like that the report is available to read with no registration required.


Wendy Nather

Wendy Nather

Wendy Nather is Head of the Advisory CISO team at Duo Security (now Cisco). She was previously the Research Director at the Retail ISAC, as well as Research Director of the Information Security Practice at independent analyst firm 451 Research. Nather led IT security for the EMEA region of the investment banking division of Swiss Bank Corporation (now UBS), and served as CISO of the Texas Education Agency. She is co-author of The Cloud Security Rules and was listed as one of SC Magazine’s Women in IT Security “Power Players” in 2014, as well as an “Influencer” in the Reboot Leadership Awards in 2018. She serves on the advisory board for Sightline Security, an organization that helps non-profits improve their cybersecurity.


Chris Swan

Chris Swan

Chris is a frequent speaker on topics such as serverless, DevOps, cloud, containers, security, networking and the Internet of Things. He’s also a cloud editor for InfoQ and a contributor to open source projects such as Docker, CoreOS and DXC’s Online DevOps Dojo.

Nick Selby

Nick Selby

Nick provides information security, disaster- and cyber incident-readiness assessments at Fuzz Technology, a subsidiary of EPSD, Inc. From 2021 to 2023, Nick served as VP of the Software Assurance Practice at Trail of Bits (where he was the voice and executive producer of its podcast), and from 2019 to 2021 as Chief Security Officer at Paxos Trust Company.

From 2018 to 2020, Nick served as Director of Cyber Intelligence and Investigations at the NYPD Intelligence Bureau, where he helped the department understand how it investigates online, and how Cyber Enabled crime affects New Yorkers.

In 2005 he founded the information security practice at industry analyst firm 451 Research, (now S&P Global Market Intelligence) where he served until 2009 as 451’s Vice President, Research Operations.