Recording date: May 3 and 8, 2021
In a private Twitter chat, Nick and Kenn White, who leads applied encryption engineering in MongoDB, were discussing tech debt burndown, and Kenn mentioned that Mongo maintains an entire team to remove distractions and handle barriers to engineer happiness. So this episode, Kenn joins Chris and Nick to talk about that, because Chris had been mentioning enabling teams in the past couple of episodes.
Mongo recognized that when something takes one person two or three or four hours of frustration, that’s real money when you’re talking about three or four or 500 engineers. The Developer Productivity Team at Mongo set out to assure that everyone has on day one a fully functioning and configured Linux laptop, with all the tools, libraries, permissions, etc., that they will need on day one. This includes wikis and internal resources.
That “get out of the engineers' way” approach is what Chris has been describing, and this is a real life example. Kenn mentioned 10,000 to 20,000 hours of testing on every build, so it’s expensive if a build fails. The way Mongo has approached this supports that kind of environment. This extends down to “I can’t make Ruby 3.6 work on this particular Linux flavor” and the enabling team will tackle those kinds of problems.
But when we sought answers on metrics, they seem to elude Mongo as much as anyone else, saying that metrics are divorced from real productivity; Kenn boils doing what works down to what makes engineers happy. That’s not a bad metric. He says that the trick for his teams has been post mortems on every major project to find the tar pits and management pains in the butt, the lack of approvals (he puts a lot of emphasis on getting expense reports reimbursed quickly, which is another one we like), etc., and those lessons are iteratively applied.
The group began to follow a thread that had begun pre-show, about Kenn’s early jobs re-factoring cardiac monitoring software, but went off on a tangent that, while interesting (to us), was off topic. Which is where the quote about monkeys nesting in the satellite dishes, above, came out (it was a story about on-the-ground realities in an African networking project).
But Chris and Nick were very interested in medical device manufacturing issues: if you’re in an industry that is heavily regulated to the extent that you can’t make changes to anything once launched, and at the same time, the industry makes machines that can spend decades in the doctors' offices. How do you avoid tech debt if it is almost mandated by the law?
To answer this, we brought in Bill Pelletier, who has tons of experience in exactly this area.
Bill sets out a recent history of medical device manufacturing from custom hardware and waterfall processes through Software as a Medical Device and today’s builds. Once you peel back the fancy, shiny cover from a computerized tomography (CT) scanner, you are going to find commercial off the shelf, hardware, nothing special about it, and in many cases it is running off-the-shelf software operating systems, which is fine right up to the moment you put a network jack on it.
That gets us into the meat of the problem: testing systems for these devices are designed for these big monolithic, all-encompassing, “test everything” processes which just doesn’t make economic sense in the new paradigm. Enter, over the past five or six years, the Software as a Medical Device. No longer constrained to this older hardware model, we were one step closer to more readily available software updates, using agile methodologies - but the testing regimes are still a huge barrier. This has led to the rise in recent years for calls for better software bills of materials, and pushes to require manufacturers to identify every software component with a standard taxonomy so that users can know immediately what’s inside.
THe discussion turns to the realities of this struggle to balance security and software concerns with those of patient safety that, ironically, is often better maintained sticking with the known state - as in many critical environments, the bias is toward saying with the known good because the cost of being wrong is so high.
Kenneth White is a security engineer whose work focuses on networks and global systems. He is co-founder and Director of the Open Crypto Audit Project and led formal security reviews on TrueCrypt and OpenSSL. He currently leads applied encryption engineering in MongoDB’s global product group. He has directed R&D and security Ops in organizations ranging from startups to nonprofits to defense agencies to the Fortune 50. His work on applied signal analysis has been published in the Proceedings of the National Academy of Sciences. He created software powering the largest clinical trial and cardiac safety research networks in the world. His work on network security and forensics has been cited by the Wall Street Journal, Reuters, Wired, and the BBC.
Bill Pelletier is currently the Enterprise Security Architect for a large healthcare payer in New England, covering all aspects of security for their on-prem, cloud, and hybrid architectures. Previous roles in his career include heading up Product and Strategy at Scope Security, a startup in the Healthcare Delivery Organization Managed Security space – and as a Product Security Leader at GE Healthcare, working with CT, PET, Mammography, and X-Ray engineering teams in ensuring that new medical devices were being made more secure by design as well as in ongoing operations. Prior to dedicating his career to the healthcare space, Bill was the Enterprise Security Architect for the Personal lines business unit of Liberty Mutual Insurance.
Chris is a frequent speaker on topics such as serverless, DevOps, cloud, containers, security, networking and the Internet of Things. He’s also a cloud editor for InfoQ and a contributor to open source projects such as Docker, CoreOS and DXC’s Online DevOps Dojo.
Nick is Vice President of Assurance at Trail of Bits. He was formerly Chief Security Officer at a financial technology firm. Prior to that he was Director of Cyber Intelligence and Investigations at the New York Police Department. He is co-author of many books, including Cyber Attack Survival Manual; In Context: Understanding Police Killings of Unarmed Civilians and Blackhatonomics: An Inside Look at the Economics of Cybercrime.