Release often, stay compliant: Automating documentation for medical device software

In the world of medical device software, keeping your technical documentation up to date is a major challenge. Below, we unpack how automation can help streamline the difficult parts of the process. 

As outlined in our previous blog post, automating your documentation requires investment in two things: 1) continuous integration (CI) pipelines that build, check and test every change, and 2) a reliable process for bringing the resulting evidence into your technical documentation. With both in place, each change reaches review with most of its evidence already assembled and linked to the relevant documentation. Here’s how to set this up.

1. Define your test protocols in code

Traditionally, test protocols are written in documents, executed manually and recorded separately. This approach is slow and prone to drift, where the documented protocol and the testing performed gradually diverge.

Instead, consider defining your test protocols in code. Each automated test can carry its protocol information: its purpose, preconditions, acceptance criteria, test data and the test steps themselves. Because the protocol lives in code, it can be easily version-controlled alongside the production code it verifies. Code review records can double as evidence that the protocol updates have been reviewed.

Organise your tests into suites aligned with the verification activities in IEC 62304 (unit, integration and system testing), so the purpose of each test is clear.

2. Link your tests to the unique identifiers in your technical documentation

Your technical documentation will already assign unique identifiers to your software requirements, risk control measures, and the software items and units in your architecture. If you tag each test with those identifiers, verification traceability becomes machine-readable and no longer needs to be maintained by hand.

You should also give each change a unique identifier in your change management records, and reference it both in the commits and pull requests that implement it, and in the change history of your technical documentation. Each change is then traced to the software requirements, risk control measures, software items and tests it affects, documenting both its implementation and its verification.

3. Automate your regression testing 

As mentioned in our previous post, proportionate regression testing is a key part of any testing regime. Unfortunately, it is common for manufacturers to minimise regression testing to save time. This can lead to software defects being missed and released to production. 

The answer is to automate regression testing strategically, focusing on key user workflows, historically defect-prone areas, and functionality tied to risk controls.

Automated tests require an upfront investment of engineering effort to write, validate, and maintain. But that investment pays off as software changes become more frequent: you can run a broader, more rigorous scope of regression testing with each release, without a proportional increase in effort.

4. Make your test environments reproducible

To trust a test result, you need to know the environment it was produced in. Using infrastructure as code (IaC), you can define your test environments (operating systems, dependencies, configuration, and any simulated hardware or integrations) in version-controlled files, and provision them identically on every run.

Container images and pinned dependency versions mean that a test executed today can be reproduced months later. And because environment definitions are under configuration management, any change to the environment is tracked like any other.

The practical result is that when a defect is reported from the field, you can rebuild the exact environment and re-run the tests to investigate it.

5. Build quality and security checks into the pipeline

To support your human review, your pipeline should automatically check every change against your software development and security standards before it can be merged. Typical checks include:

• Static analysis, to enforce coding standards and detect defects early

• Dependency management, keeping your software of unknown provenance (SOUP) components on supported, pinned versions

• Vulnerability scanning of your code, dependencies and container images

Define in advance which failures block a change from progressing and which allow it to proceed with a documented justification. For example, a critical vulnerability might block a merge outright, while a lower-severity static analysis finding is flagged for review. Encoding these rules in your pipeline configuration keeps decisions consistent and gives you a clear rationale for every exception.

For manufacturers, the advantage is that defects and vulnerabilities are caught at the point they’re introduced, when they’re easiest to fix. Each check also leaves a record you can use as evidence for your software development and secure development lifecycle activities.

6. Generate your evidence from the pipeline

With identifiers, tagged tests, and automated checks in place, your pipeline can generate the artefacts your technical documentation needs on every build. These include:

• A software bill of materials (SBOM), including the verified version of the medical device software and SOUP components

• Test execution reports, showing which tests ran, in which environment, and with what result

• Static analysis and vulnerability scan reports

• Verification traceability matrices, generated by joining requirement, software item, and test identifiers with test results

• Release notes and lists of known residual anomalies

Each artefact should be tied to the commit and build that produced it, and stored where it can’t be altered. 

Extending the principles to AI as a medical device

For AI as a medical device (AIaMD), the same principles apply, with your models and data treated as configuration items alongside your code. Version your datasets so you can show the lineage of every training, validation and test set. Define your training pipelines as code so retraining is repeatable, and record the code, data and environment behind each trained model.

When a model is retrained, your pipeline can automatically evaluate it against a locked test set using predefined acceptance criteria, including performance across relevant patient subgroups. Linked back to the change request, the evaluation report gives you a traceable record of why the model changed, and evidence that its performance has been maintained.

Conclusion

Building this infrastructure takes time and expertise. But the payoff is a repeatable process that allows you to release as often as you need to while staying safe and compliant.