Test automation for SaMD within IEC 62304: How to be effective and stay compliant

IEC 62304 has several expectations regarding test management that must be considered when adopting test automation as part of your Software as a Medical Device (SaMD) development process.

Test automation has become a staple component of modern software development. It increases the speed and efficiency of testing, reduces the risk of defects, and improves the overall quality of the software.

However, when using test automation within SaMD development, some specific considerations need to be taken into account to ensure that your tests comply with the medical device software development standard IEC 62304.

They include construction of test strategies, creation of test documentation, traceability of test artefacts to other regulatory artefacts, reliability of automatically generated test results, and reproducibility of automated test executions.

Let's look at these more closely.

Test strategy

In the development of SaMD, a combination of testing activities is essential to ensure the software is correctly implemented, fulfils its intended use, and remains safe and effective.

A well-structured test strategy may incorporate automation for specific testing tasks, though not all. For instance, usability and validation activities are typically conducted by humans and are not amenable to automation. Software verification, which involves assessing software requirements and design, is generally more compatible with automated testing.

Depending on the software safety classification of the SaMD, IEC 62304 requires several levels of software verification, including unit testing, integration testing, and requirements verification testing. Automated testing is highly effective when verifying the inner logic of the medical device software through software testing activities, such as software unit testing and software integration testing. Manual testing of these activities is challenging as the test inputs and outputs are not easily exposed to a human tester, and automation is generally preferred.

Automated testing can also be effective for requirements verification testing. It is typically conducted using system-level, functional tests, which incorporate real devices and environments, and better represent the intended use of medical device software than software unit testing and software integration testing. However, system-level automated testing can be more costly to set up and maintain. Therefore, manufacturers should weigh the advantages of it against the effort required to develop and maintain the test code.

Regression testing is another software verification activity where test automation is effective. Regression testing is used to verify that the software continues to meet the software requirements and software design after changes have been made to it. An investment in automated test scripts for regression testing is generally worthwhile, as these tests can be executed with each SaMD release, providing long-term value.

Not all software verification testing should be automated. Some activities are complex to automate, and the return on investment may not be justifiable. Others, like user interface testing, are better suited to human verification. Overall, the manufacturer should carefully consider several factors when constructing their test strategy, including the safety classification of the SaMD, the complexity of their software system, the expected rate of change of the software, and the potential return on investment of the test automation.

Documentation & traceability

Modern software-testing methodologies, such as test-driven development (TDD) and behaviour-driven development (BDD), have encouraged the development and adoption of software testing tools that allow software developers to create test specifications within the confines of the software development codebase and repeatedly execute these tests within a continuous integration environment.

This is incredibly effective at providing regular synchronisation between the software code and the software testing as the development of the medical device evolves. However, the documentation and traceability of this testing must be captured and maintained per the expectations of IEC 62304.

IEC 62304 mandates traceability between items under test and their corresponding test activities, specifications, and results.

Items under test include software units, modules, and requirements. When test activities and specifications are implemented as automated tests within the software codebase, they should be clearly named or annotated to ensure traceability to the relevant items they verify. When verification tests are executed, their execution reports should be archived for audit purposes - complete with relevant versioning information.

Additionally, it is beneficial to use software development tools that can export test specifications and reports from the codebase and continuous integration environments. This integration allows this test management data to be incorporated into the medical device's technical documentation and made available to a Notified Body.

Validation of the test-automation systems

Where a test-automation system is used to conclude a test outcome automatically, the test-automation system must be validated for this purpose. This ensures it can produce consistent and accurate results and does not produce false positive or negative results, which may invalidate the verification of the SaMD.

Manufacturers are highly recommended to choose industry-standard test-automation tools with a proven track record of accuracy and reliability. Using unknown or internally developed test-automation tools can increase the scrutiny of a Notified Body, requiring the manufacturer to demonstrate the validation of their bespoke test-automation systems.

Version control & reproducibility

IEC 62304 expects that test executions, whether manual or automated, are reproducible. A Notified Body may request evidence demonstrating how automated tests can be reproduced to verify that the manufacturer has met this requirement. Simply recording the version of the software under test is not sufficient. It is also essential to document the version of the test scripts, test data, test frameworks, and any other software dependencies required for the test execution.

Additionally, details of the test equipment and environment must be recorded to ensure they can be reproduced. This may include hardware serial numbers, operating system versions, and environment configuration settings.

To support reproducibility, manufacturers should use a combination of software version control tools and documentation to capture all relevant information before executing verification tests.

Automation to enhance, not compromise

Test automation is a powerful tool that can significantly enhance the efficiency and effectiveness of SaMD development. However, it is essential to consider the above guidance to ensure your testing strategy, tooling, and documentation comply with IEC 62304.

While not all testing activities are suitable for automation, a well-planned combination of automated and manual testing can help manufacturers deliver high-quality medical device software that meets regulatory requirements while optimising development resources.

The key is to strike the right balance and implement automation to enhance, rather than compromise, your IEC 62304 compliance.