What’s in your software?

The past week has seen a furore in the .NET software development community.

Whilst the exact nature of the issue is of narrow interest to some software developers, it raises a wider issue — how can we trust our (or our supplier’s) software given that we (or they) almost certainly didn’t create all of it themselves?

Background

In short — and glossing over the nuance — a software component used by millions of developers (and therefore tens or hundreds of thousands of products) introduced a “feature” designed to nag developers financially, which in turn created privacy implications. Even Microsoft’s own products had the potential to be affected.

Ignoring the extent to which this new feature is or isn’t an issue, and how it was introduced is or isn’t a problem, it raises interesting questions for both developers and users — do you fully understand what’s in your software product?

As a user, you may assume that the company who sold you the software you use fully understands what’s in it. After all, they wrote it, right?

That’s true to a point, but inevitably software contains numerous components — the stuff your supplier actually wrote, platform stuff from major companies such as Microsoft or Amazon, clever stuff your supplier may have bought in such as chat technology, UI-based visuals, etc and almost certainly at least a few (or maybe more) open source components.

If you’re familiar with open source, you may want to skip the next couple of paragraphs. If not, Open Source software is software which people have designed, built and shared. Motivation varies — some people do so to be altruistic; others do so to build a profile and making getting jobs or contracts easier; others do so to make a living (for example, selling support). A large proportion of the internet runs on Linux (an open source language) using MySQL (an open source database) and PHP (an open source programming language). Open source can be really useful as a software developer, because it gives you options to leverage other people’s work solving common problems, so you can focus on what makes your software unique.

Open source is often described as “free”, but it doesn’t have no cost. You may not pay anything for the software component itself, but you ARE responsible for the hidden costs of maintaining, updating and securing it (or paying someone else to do so). That’s the rub for many open source providers. Organisations such as Microsoft are happy to provide open source products to drive sales of paid-for products; but for individuals giving their time for free as volunteers to create something, popularity creates expectations which are unsustainable without an income.

So what’s the problem?

Well, inevitably, the software you use daily at home or work will contain components from multiple sources — a mix of collaborations between your supplier, their contracted suppliers, their suppliers’ suppliers and open source creators. And if you don’t fully understand that network of collaboration, it can lead you into trouble.

The Moq issue causing the furore this week was caused (to put it neutrally) by an open source developer introducing a feature which many users of his software objected to. Similarly, commercial dependencies can suffer issues too — Ticketmaster received a £1.25m fine in 2020 arising from a security hole in a third party component they bought-in. And there are countless other examples.

As soon as you lose track of the network of collaborations making up the software you write, sell or support, you’re placing blind faith in persons unknown to look after your customers’ security, privacy and trust.

Meeting the Challenges

As a software product owner, I’d want to know the following was in place:

• Do we map the relationships making up our software?
• Do we have clear rules on what components we trust, and what components we closely monitor?
• Do we have a plan in place to support and maintain third party components — both open source components, at the whim of someone else’s generosity, and commercial components where the provider may cease trading in the future?
• Do we have a process for approving new suppliers and components, and verifying the components we introduce technically?
• Does that process of verification extend to updates?

As a Technical Director, CIO or customer procuring software, I’d want to know my team or my supplier had this in place.

What does a reasonable solution look like in practice?

For non-safety critical industries (excluding things like medical devices, airline transport, railway signalling etc), it’s not feasible for your own team to read, evaluate and test every single line of third party code. So, a proportionate approach is necessary.

An approach that works for me looks something like this:

• We have a clear list or map of dependencies we’re using. That dependency map includes version, license, supplier / publisher and any sub dependencies.
• We maintain a list of software and vendors we can trust, either because there’s a commercial contract in place limiting our liability, or because the supplier / publisher is inherently trustworthy.
• Where possible, we audit our trustworthy vendors; where impossible to audit, we require assurances before using their components.
• For open source publishers, where we have no contract or assurances, we evaluate new components in depth. Does it meet our code quality standards? Are we confident it doesn’t introduce security or privacy vulnerabilities? Is the component maintainable by us if its publisher folds up their tent tomorrow? Is it lower cost/risk to use this component, or should we bite the bullet and write our own?
• For vendors we can’t regard as trustworthy, we must have access to the source code that makes up their product to evaluate as if open source.
• Our update programme must balance the need to keep up to date with the need to verify and assure. For open source components, we should review the code and ensure we understand the update’s rationale and that the update fulfils its purpose without introducing unintended consequences or malicious intent.
• Vulnerability scanning — we scan the software we produce for known vulnerabilities, using the best tools available. For my team, working in C#, we use automated scanning of NuGet packages every time we build our software.
• Knowledge is king — we tap in to the wider knowledge of millions of developers worldwide through social media, industry publications and security initiatives, and respond to issues reported by the wider community.
• Peer review and audit — we peer review each other’s work, and where appropriate formally audit what we’re doing.

No solution is ever perfect, but we feel this is the minimum standard for a software product people rely on in day to day usage.

If your product is safety critical, you will already be doing much more; if it’s trivial or its failure is inconsequential, you may continue to do less.

But, even the most trivial 10min mobile app has the ability to introduce security or privacy vulnerabilities, and thereby attract opprobrium (and fines). So if you’re one of the many developers and teams doing nothing, now’s the time to think about your third party software assurance, and put something in place.