An IEC 82304 fix for software validation frustration

What is the objective of software validation?

Its role is to prove, through the provision of objective evidence, that the use requirements of a product have been met. In essence, to validate that the product you have built meets the criteria of the product you set out to build and the targeted intended use has been met.

Start with a plan

Proper planning lays the foundation for the effective and compliant performance of software validation, and IEC 82304 Clause 6.1 outlines the core requirements. A good validation plan should cover the scope and limiting constraints of validation, required qualifications and independence level of validators, the validation methods (activities) utilised, and any enabling systems required.

Identify constraints

Though organisational scale and resources differ, all SaMD manufacturers work within constraints. Identifying and factoring these constraints into validation planning is key to establishing an achievable and compliant validation plan.

An example of a universal constraint is team size. No team has an infinite supply of qualified personnel available to conduct validation. Therefore, the scope of validation will always be limited by the size of the team. By recognising this constraint, manufacturers are empowered to plan a software validation process that ensures an appropriate scale of validation relative to their SaMD and team size.

Another common constraint is limited access to hardware platforms for validation. For example, if a device is intended for home use and targets Android devices generally, performing end-to-end validation testing on all available Android devices is not feasible. Instead, a manufacturer may opt to select a limited array of devices within a supported hardware specification range, based upon the CPU, GPU, or RAM requirements of the SaMD.

Establish scope

With the constraints identified, establishing scope requires manufacturers to determine how they can objectively validate their use requirements within these constraints. Objective validation must be achieved regardless of any constraints, this is still entirely possible with a scope that is limited by resources.

For example, depending on organisational constraints, it may not be practical or desirable to conduct a separate validation activity for each use requirement. However, it's vital that all use requirements are covered by the validation activities conducted in totality. Logical and traceable grouping of use requirements into appropriate validation activities is an effective approach to both limiting scope and ensuring full coverage.

Additionally, clear scoping makes it easier for manufacturers to leverage activities conducted as part of other development processes during validation. For instance, analysis of the results derived from summative evaluation or clinical evaluation studies.

Validator independence

Establishing appropriate validator independence is imperative for both ensuring and evidencing the integrity and adequacy of software validation. Equally, it's an area that SaMD manufacturers, especially those with smaller teams, often struggle with. At a high level, manufacturers must be able to prove that validation of software has been conducted by individuals with a sufficient degree of separation from the designer of that software.

But what constitutes a "designer" in this context? It might be tempting to consider "design" as a non-technical process, separate from the technical work enacted by software developers. However, in practice developers rarely work based upon a completely rigid design specification. Many technical decisions made by developers prior to and during implementation reasonably encompass design. Therefore, those directly involved in development and implementation are also usually designers of the resulting software.

IEC 82304 Clause 6.1.f and Annex A Clause 6 offer scope for achievable validator independence regardless of organisation size. It's important for manufacturers to leverage this, ensuring they don't overcommit to an independence level that they cannot realistically follow.

Let's examine two potential validation independence statements:

• "No individuals involved in any software design activities will be involved in any software validation activities"

Whilst effective in ensuring appropriate independence and achievable for large organisations, this approach would likely be too restrictive for a smaller organisation. A dedicated team of technically proficient validators that had no involvement with any design activities is required.

• "No individuals involved in the design of a software item will be involved in software validation activities associated with that item"

This approach leverages the "separate person" example from IEC 82304 Annex A Clause 6. Validation integrity is maintained by creating an appropriate degree of independence between the designer of the software item and the validator of the software item. Additionally, it's achievable for organisations of almost any size, utilising appropriately independent individuals from a single team.

Validation activities

The specific validation activities required to adequately validate any given SaMD depend heavily on the discrete use requirements defined. There are however core principles that are applicable universally.

Validation activities should aim to:

• Produce objective evidence
• Comprehensively and traceably prove associated use requirements have been met
• Utilise appropriate input data
• Enforce sensible acceptance criteria

Common validation activity types include:

• Testing:
  • End-to-end system functionality testing
  • Penetration testing
• Inspection & analysis:
  • Accompanying document analysis
  • Verification test report analysis

Software validation ≠ software verification

Software validation as defined by IEC 82304 is a distinct process from software verification and software system testing as outlined in IEC 62304. The former focusses on the validation of use requirements, whereas the latter focusses on the verification of software units and testing of software requirements. Nonetheless, the boundary between these processes can sometimes appear blurry to SaMD manufacturers.

During software verification, individual software units are tested, often through the use of automated unit tests and regression testing. Though validation can encompass high-level end-to-end system-testing activities, the goal is to validate the functionality against the high-level use requirements of the system. An area where the two overlap is the analysis of evidence produced by the software verification process, which itself can constitute an acceptable validation activity. For example, depending on scope and constraints, it can be a valid approach to perform analysis of a software system test record to validate a use requirement has been evidenced as met, without the validator directly re-testing. 

Closing thoughts

The approaches we've covered outline several ways in which SaMD manufacturers of any scale can leverage IEC 82304 to achieve effective and compliant software validation smoothly. Manufacturers must ensure all software product use requirements have been objectively validated by appropriately qualified and independent individuals, effective identification of constraints and establishment of scope facilitates this. Finally, effective validation of requirements is contingent on a strong foundation of requirements, our previous coverage on crafting proper requirements outlines some key approaches in this area.