A recipe for good SOUP: Part 4 – Defining requirements for SOUP

The use of Software of Unknown Provenance (SOUP) supports the rapid development of cutting-edge SaMD products but remains subject to one of the core tenets of regulatory compliance for medical software: clear and verifiable requirements.

The growing collection of third-party tools and open-source software increasingly enables the development of innovative products. It's important therefore for manufacturers to keep first principles in mind, and especially to focus on requirements.

While it is reasonable for manufacturers to not be responsible for the full, detailed documentation of the various SOUP components in their device (typically due to lack of access), they should still consider and clearly document:

Functional and performance requirements

IEC 62304 Clause 5.3.3 states, for Class B & C software items:

If a SOFTWARE ITEM is identified as SOUP, the MANUFACTURER shall specify functional and performance requirements for the SOUP item that are necessary for its intended use.

This means that manufacturers may need to go beyond the scope of internally developed software items when it comes to functional and performance requirement definitions. Examples of SOUP functional and performance requirements include:

• The (SOUP ML component) shall return a classification result and confidence score for each valid input
• The (SOUP database component) shall store patient records with unique identifiers and support create, read, update, and delete (CRUD) operations
• The (SOUP component) shall return a processing result within 500 ms for 95% of valid requests under nominal operating conditions

It follows that SOUP components critical to device functionality are subject to some of the same software-development process requirements as internally developed software, including software verification – ensuring that device functionality is holistically verified and validated, regardless of the source of software items that make up a device.

For further information on how to form well-crafted requirements, have a look at our other blog posts on the importance of crafting requirements and making them testable.

Hardware and software requirements

In addition, IEC 62304 Clause 5.3.4 states, for Class B & C software items:

If a SOFTWARE ITEM is identified as SOUP, the MANUFACTURER shall specify the SYSTEM hardware and software necessary to support the proper operation of the SOUP item.

The clause also lists various software requirements examples: processor type and speed, memory type and size, system software type, communication, and display. Again, this all supports the reliable operation of the device and its ability to perform its intended use.

This is particularly important if this differs from the minimum hardware and software requirements listed for the non-SOUP components in the device architecture, as a failure to account for these SOUP requirements could ultimately result in compromised device performance, or at worst, device failure.

Note that, in the event that SOUP minimum hardware and/or software requirements align with those of the overall device, manufacturers can consolidate these device requirements and have both non-SOUP and SOUP components refer to the same set.

SOUP compatibility

In addition to requirements for SOUP components themselves, manufacturers should also consider functional requirements defined for internally developed software related to SOUP compatibility.

IEC 62304 Clause 5.2.2(a) includes a note:

As appropriate to the MEDICAL DEVICE SOFTWARE, the MANUFACTURER shall include in the software requirements:
a) functional and capability requirements;
…
– need for compatibility with upgrades or multiple SOUP or other device versions.

This introduces another consideration when incorporating SOUP beyond the functionality and performance of the SOUP components themselves.

Consider an internally developed software item with multiple third-party library dependencies – an incredibly common scenario in modern software development: SOUP-A has critical functionality that is dependent on specific versions of SOUP-B and SOUP-C. In this scenario, it is important to document these dependencies, which will in turn support other software-development activities.

What about LLMs?

In the previous blog post in this series, we explored the implementation of LLMs as SOUP due to a variety of characteristics common to modern LLMs (particularly third-party ones).

The points raised in this blog post would therefore also apply to any LLM component classified as SOUP. While the hardware and software requirement aspect may vary depending on the implementation, the necessity of functional and performance requirements, and their subsequent verification and validation, form part of the challenge of implementing LLMs while meeting regulatory expectations.

This is why it is crucial to consider how LLM outputs will be validated and what will be needed to ensure robust functional verification testing when incorporating LLMs into any device. These elements surface as a direct result of the need to define the functional and performance requirements of the LLM component.

Why it matters

Clear and unambiguous requirements for SOUP components enable:

• Clear traceability
• Establishment of tests
• Ongoing verification and validation

These activities are particularly important when you factor in software updates – most of all in the area of cybersecurity, where urgent security updates need to be managed in the context of the device and all its dependencies.

Summary

Incorporating SOUP may introduce elements that sit outside a manufacturer’s direct control, but it doesn’t lessen the need for clear, intentional, and traceable requirements.

By defining what the device depends on each SOUP component to do, the performance it must deliver, the environment it requires, and how internally developed software must remain compatible, manufacturers can ensure that these third-party items are integrated in a deliberate and compliant manner.

In the end, thoughtful and thorough requirement definition enables manufacturers to support reliable device performance and reinforces one of the core principles of safe and effective SaMD development.