A recipe for good SOUP: Part 2 — Ensuring safety & performance in SaMD

In Software as a Medical Device (SaMD) development, Software of Unknown Provenance (SOUP) presents a unique challenge when it comes to ensuring device safety and performance, particularly in the areas of testing, verification, and risk management. 

Continuing our series on SOUP, now that we’ve covered what constitutes SOUP and how it fits into the software development process, we can dive into some of the topics that form the basis of regulatory compliance. A key question remains: how can we ensure device safety, functionality, and clinical performance when evaluating software that does not itself conform to regulatory standards?

Considering that SOUP functionality can range anywhere from providing support during development (e.g. testing, logging) to essential features that enable the core intended purpose of medical software (e.g. user interface for clinicians to operate the device), it is essential that the functionality of these components is verified in some way.

So let's explore the limitations of testing, verification, and risk management in the context of SOUP, along with the approaches that can be taken to fulfil the responsibilities of manufacturers in evidencing device safety and regulatory compliance despite these limitations.

Testing & verification

Unit testing and component-level testing are generally not applicable to SOUP regulatory compliance. Testing external software at this level is simply not an option.

Depending on the source of your SOUP, there may be limitations around:

• Your access to source code
• Your ability to make changes/fixes to the component
• Your access to documentation and traceability material
• Regression-testing methodology

This means that integration testing and system-level testing are of heightened importance when it comes to verifying the functionality and performance of SOUP in your device.

As a result, the focus of testing for SOUP tends to shift away from code quality and function coverage, and instead towards interface compatibility, boundary behaviour, fail-safes, function dependencies, security, and regression testing.More details on integration testing requirements can be found in IEC 62304 Clause 5.6.

Risks

All software, including SOUP, potentially exposes your users to risk. On top of the typical risks associated with internally developed software, the fact that manufacturers often do not have control over SOUP development introduces additional risks specific to SOUP.

A key consideration to factor into risk analysis when it comes to SOUP, is not simply what happens when you do decide to incorporate an update or new version of a specific SOUP component, but also:

• What are the risks of not upgrading to the latest version?
• What are the risks of the SOUP component being deprecated or unavailable?

And don’t forget about data! Protecting patient data is crucial in SaMD, along with data retention and access-authorisation policies. If you plan to pass any sensitive data into SOUP, this is another key factor to consider in your risk management.

Risk controls

Where SOUP can also differ is in how hazards can be detected and mitigated through risk controls. A brief recap on the general categories of risk controls available for SaMD:

• Inherent safety by design
• Protective measures within the device or development processes
• Information for safety & user training

Since the ability of a manufacturer to modify SOUP is limited, the first and preferred method of risk control (inherent safety by design) is often not available — at least not within the SOUP component itself. Depending on the way SOUP is implemented, some level of inherent safety by design could be applied to internally developed software components that interface with SOUP.

This leaves the remaining risk control options as the main methods of risk management for SOUP. When it comes to protective measures for SOUP components, examples include:

• Frequent and thorough regression testing of SOUP integrations
• Logging and monitoring of SOUP-dependent functions
• Monitoring of SOUP updates & security vulnerabilities

On that last point, IEC 62304 Clause 7.1.3 explicitly calls for the evaluation of anomaly lists published by SOUP manufacturers. This is important not only in the context of software development prior to release, but also in the post-market environment. 

You will want to be aware, across all of your SOUP, of any updates, end-of-life notices, and emerging vulnerabilities — factoring these into both your risk-management strategy, post-market surveillance plan, and communications to users/regulators.

SOUP in summary

Navigating the complexities of SOUP in SaMD development presents unique challenges, particularly in testing, integration, and risk control. However, through a solid grounding in regulatory standards such as IEC 62304, and thorough application, manufacturers can harness the benefits of SOUP and modern software development while ensuring their device remains safe, effective, and compliant.

Stay tuned for the final entry in our series, where we will discuss SOUP through the lens of AI, and how LLMs might fit into the definition of SOUP.