MEDdesign

Automated Software Validation: Fast-Tracking MES Implementations in Regulated Environments

By Carla Neves
July 7, 2021


• Categories: 03-Manufacturing Execution, 04-Product Design & Development, 05-Quality/Regulatory, MEDdesign

In its “Case for Quality”, the FDA has been striving for at least the past decade to elevate the focus from baseline regulatory compliance to sustained predictive practices that advance both the quality and safety of medical devices and diagnostic instruments. Along with other healthcare regulators, its goal is to make it easier for manufacturers of medical devices and other regulated products to devote less time to documentation protocols and more time on reducing costs and improving quality. The burden of computer software validation (CSV), however, has been deterring the investment in the automation systems needed for a quality-centered production model that fosters a continuous improvement process. Automated software validation offers a solution to overcome this problem and reduce the time to implement and validate computer systems.

Instead of CSV, which has become an onerous task focused on passing audits rather than building in quality, the FDA has changed its emphasis to computer system assurance (CSA), a risk-based approach that allows manufacturers to concentrate testing on areas that have greatest impact on product quality and, ultimately, patient safety. This approach recognizes that commercial off-the-shelf (COTS) manufacturing software, such as manufacturing execution systems (MES), product lifecycle management (PLM), or quality control systems have positive impact on healthcare costs outcomes and add benefit to users, vendors and patients. The faster these systems can be implemented, the quicker they can add value to the production process.

CSA of software in a regulated environment establishes validation procedures on a risk-based approach. The FDA’s 21 CFR Part 820, however, has its own definition of process validation: “Establishing by objective evidence that a process consistently produces a result or product meeting its intended and predetermined specifications.” It requires that the end product conforms to performance, material and quality specifications for repeatability. The more complex 21 CFR Part 11 requires, “Validation of systems to ensure accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records.” An MES in a regulated environment must meet all of these oversights. Automated Software Validation offers a way to reduce the time and cost associated with this requirement.

The Need for Speed

The ultimate objective of validation is to ensure that each regulated medical device is safe for patient use and will function correctly, without error and the potential to harm. But it also has many benefits for manufacturers, including improved repeatability, fewer mistakes, less rework and redesign, faster time-to-market, improved competitiveness, and lower production costs. Although some are already reaping such rewards, most are probably not. A study by the IVT Network, for example, found the following organizational and process issues interfering with validation:

• Inconsistent practices, objectives and expectations
• Unnecessary duplication of roles, responsibilities and efforts
• Excessive repetition, rework, handoffs, and resource allocations
• Excessive reviews and approvals of protocols and other documentation.

These are the kinds of issues that the regulatory bodies and standards groups are looking to eliminate.

Traditionally, software validation can take four to eight months or more for a single system, or about one third of the total project timeline. For MES, the greatest challenges in performing and maintaining a CSV process relate to end-to-end traceability between requirement and end-testing and the execution of the tests, which can be error-prone and time consuming. However, by using a next generation validation solution to automate the validation process, the pain and time taken for the CSV phase is dramatically reduced. The approach improves data visibility and accountability, and, ultimately, reduces the time for system deployment from months to just a few days.

MES Software Validation: The Next Generation

The new generation of validation automation involves three main elements: Validation templates, product documentation and automated validation processes.

Validation Templates

Good Automated Manufacturing Process (GAMP) provides a basis for the creation of templates to guide validation of all aspects of the software, from user requirements specifications (URS), functional specifications (FRS) and design specifications (DDS), all the way through installation quality (IQ), operation (OQ) and performance qualification (PQ). The templates are also available for the associated traceability matrix, master plan and summary.

Many companies have successfully used such templates to complete their validation requirements, and, as the newest GAMP-5 guidelines call for an even more risk-based, iterative approach for validation, such templates will be increasingly valuable in supporting the emerging new generation of software validation.

Product Documentation

Besides the User, Installation, Operations and Developer Guides, System Requirements and Tutorial that are part of the MES, the package also include a set of requirements defined for the medical device industry, their respective test case specifications and test results for every product release.

Automated Validation

In this fully integrated, fully automated, cloud-based validation platform, scenario, the vendor hosts a portal for end users and validation partners to access a “NuGet” library of information needed for future validation efforts. This could include full test scripts for out-of-the-box implementations that would streamline validation. Test scripts, written in C#, for example, might allow the MES provider, end-user or partner to contribute to test script development. All would be defined in a master validation plan culminating in a validation summary report signaling completion and readiness for sign-off.

An integrated validation environment enables mapping of the end-to-end engineering workflow developed with cloud-based development tools such as the Azure DevOps platform. These are used to develop plug-in test case and requirements management, build functionality such as requirements management, test case specifications, traceability, source code repository and a test results repository, and fit naturally within the development lifecycle of an implementation project.

Automating the validation process across a common platform, shared by the software vendor and the end-user enables complete traceability of all transactions and the addition of expanded features, which can be plugged in to support product documentation, master validation templates and version-related documentation. Such a platform facilitates close cooperation and tight collaboration between the client’s implementation team and the vendor’s delivery team, allowing them to interact and assess progress on a common platform, which records and reports progress automatically. Additionally, it serves the purposes of both validation and training-related requirements.

Other modern features that could be incorporated in a full-integrated validation include:

• Links of all the functional requirements to backlog items
• ‘Smart’ document management system
• Printouts of reports
• Developing a full traceability report in minutes, not days
• eSignature-based approvals for Part 11 compliance validation collaboration, which would be enhanced by Microsoft Teams and SharePoint that enforce continuous collaboration throughout the project

Managing Validation Backlogs

The shifting of focus from documentation to risk-based validation has also enabled a next-generation MES development strategy that unleashes the combined potential of templates, build-and-test-servers and a fully integrated environment to reduce backlogs.

Where traditional validation solutions might use a waterfall approach that puts validation off until all user requirements are defined perfectly and the whole application is ready, the relaxation of documentation requirements enables the use of more agile software development strategies that enable development and validation of multiple requirements in parallel.

Agile software development includes Epics as an iterative deliverable. An Epic is a defined body of work that is segmented into specific features based on the needs/requests of customers or end users. Epics, which typically represent a business initiative and features, representing a shippable component of software, are broken down into deliverable building blocks to enable faster project development. Validation test scripts are associated to the Epics to reflect the requirements to be validated.

Where traditional validation approaches hold on releasing any software until all features have been built and validated, by breaking down into Epics, validated features can be released more quickly to users. The simulation test script is run in parallel with other development and testing to avoid bottlenecks. This approach further means results can be discussed and needs clarified before implementation, minimizing post-implementation rework.

Next-Generation Validation in Progress

Manufacturers that have implemented a fully integrated approach have reported that they are validating their software faster and with less effort, getting products to market faster and seeing ROI more quickly. For example, by automating the validation process, one manufacturer serving regulated high-tech industries shortened its validation time by 30%, while also producing training materials. It had automated test cases to cover end-to-end operator processes and procedures and was able to create training documentation by running test scripts. Printouts of all automated test scripts were generated. The process also used screen captures, to gather validation documentation throughout testing, and these were converted to short training videos. In addition to the 30 percent reduction in time to implementation, the manufacturer was able to run all this concurrently with other tasks, without waiting until the end of the process.

The FDA believes that automation and technology can dramatically streamline validation efforts. Cisco Vicenty, CDRH Program Manager at FDA, offered his unique insight on the topic:

“The FDA supports and encourages the use of automation, information technology, and data solutions throughout the product lifecycle in the design, manufacturing, service, and support of medical devices.

Automated systems provide manufacturers advantages for reducing or eliminating errors, increasing business value, optimizing resources, and reducing patient risk. These capabilities provide significant benefits in enhancing product quality and safety.”

Table 1 represents a consolidation of information from a number of sources including the FDA and shows what is feasible from this risk-based, quality-centered approach to validation. By automating validation processes, there is potential to reduce time taken for analytics and reporting by 75% and software vendor qualification by 80%. More agile development techniques will further reduce testing time.

Non-Product CSV Modification Impact
Activity	Current Approach	Modified Approach	Impact
Streamlined Analytics Reporting	Consumes 43 hours per report	Reduced to 10.5 hours per report	$90 per hour average cost $3870 per report vs. $945 Drives reduced use of analytics and operational research
Risk-based software vendor qualification	~2000 hours of labor	80% reduction in labor	$100 per hour average cost $200,000 –> $40,000
Ad-hoc/unscripted testing (AGILE)	14 hours per test script	2 hours	a $10 billion firm with 100 systems could save $1.62 million annually, which can be reinvested in innovation & quality improvement More robust software (safer product)
Table 1. Impact of modification on non-product CSV. Source: FDA, a collaborative FDA and Industry Perspective: Automation + Non-Product CSV, Cisco Vicenty, Program Manager, case for quality, FDA CDRH OC, Siemens PLM-Medtronic Excellence Event, May 15, 2018, Minneapolis, MN

Summary

Automating validation processes gives manufacturers the flexibility and capability to make changes, updates and re-validation, making it easier to upgrade or add new MES modules to their process. The approach also enables users to validate their systems while using them, which saves time and cost in validation efforts. Customers utilizing this new approach of validation using Azure DevOps, report that it is that it is taking less effort on their part to actually perform the validations, and the validations are completing much faster than ever before.

Simplifying the validation process through automation leads to faster ROI and time to market. Ultimately, it removes the barriers to MES, enabling it to fill its critical role in helping manufacturers minimize risk and maintain quality be getting greater control over production data.