Medical Device Software Safety, Quality, and Compliance: What Really Makes A Difference?

By Carl Wyrwa
October 19, 2015


• Categories: 04-Product Design & Development, 05-Quality/Regulatory

Medical device software quality is extremely important.  Our expectation is that software-controlled medical devices are of high quality and will operate without issue when performing diagnostic evaluations or therapeutic treatment on us or  loved ones. Within organizations selling software-controlled medical devices, senior management expects their technical staff to develop software consistent with these quality expectations. This requires that software organizations find and employ techniques that yield the highest levels of safety and quality possible.

What Really Makes A Difference

This article is the first in a series of pieces that will provide insight into some of the elements that have the most significant impact on achieving software safety, quality, and compliance. Although there are many important software engineering best practices to consider, some are essential to have in place. This article will discuss several of these key areas, and future articles will dive into some of these areas in detail.

A Strong Software Quality Philosophy: Zero Defects in System Test

The number of defects found during a software system test can be a strong predictor of how the software will perform when a customer uses it. Testing steps at the end are never 100% effective in removing defects; in fact, they are typically far from effective. If several defects are found during system test, then it is very likely that more defects will be found by the customer. It is very important to set crystal clear goals in order to ensure that an organization can establish practices to meet its safety and quality goals and determine if their practices are effective. Setting a goal of zero defects in system test is very unambiguous. It focuses an organization on early defect removal practices that eliminate as many defects as possible prior to the final testing steps, resulting in the delivery of much safer and higher-quality software. Granted, achieving zero defects in system test is a big stretch. However, this philosophy establishes a mindset that promotes the right activities early in the process. Teams adopting this philosophy and implementing the following practices may not necessarily achieve perfection, but they will certainly realize huge gains in the software quality area.

Carl Wyrwa will be speaking at the Medical Device Software Conference | March 21–22, 2017 | Washington, D.C. | Learn MoreRisk Management: Fully Integrated Into Entire Software Development Lifecycle

One of the most important key competencies each organization and individual must have is the ability to anticipate potential harm-related risks and eliminate or reduce these risks as much as possible. Risk management must be an ongoing set of activities that address risk identification and remediation throughout the entire software development lifecycle. There are key objectives that must be met during the software requirements, design, coding, integration, verification, and validation phases. These objectives can only be met if everyone involved in the development of the software participates in the risk management process every step of the way.

Software Requirements: Set the Foundation for Success

Software requirements form the basis for the entire software development process. When well written, they establish a good starting point for risk management and design activities. Organizations must determine the appropriate amount of detail desired in the requirements to ensure that individuals are only making requirements-based decisions when and where it is appropriate. In order to keep the requirements well organized and meet traceability requirements for medical device software, organizations should consider automated requirements management tools.

Software Design: Make Key Decisions before Coding

One of the most powerful activities contributing to software safety and quality is the design activity. Sometimes it is tempting to begin coding as soon as possible and make design decisions along the way. This can be problematic and time consuming especially if the component being developed is of medium to high complexity.

Making key design decisions prior to beginning the coding phase can help reduce the number of ad-hoc design decisions that have to be made during the coding phase. This usually results in the development of less code and the performance of less rework. Establishing a straight-forward design approach that uses basic design views to clarify what should be considered and decided prior to coding enables developers to efficiently create designs.

Software Peer Reviews

Peer reviews continue to demonstrate that they are one of the most effective defect-removal activities available. Peer reviews are essential to supporting the philosophy of zero defects in system test. Peer review effectiveness is highly dependent on how the peer reviews are conducted and which techniques are employed. The more formal the peer review process, the more effective it is. Low formality peer reviews can find some defects, but typically they do not have the necessary effectiveness needed for medical device software development. Formal inspections have proven to be the most effective method and should be tried by organizations to review their requirements, designs, code, and test procedures. Walkthroughs can also be very effective but do not yield the same effectiveness as inspections. A hybrid of inspections and walkthroughs can be a practical starting point for organizations, using inspections for the more complex and riskier components of the system. Peer reviews make good economic sense since the cost of finding and fixing defects through peer reviews is significantly less than the cost of finding and fixing defects during testing. To achieve success in peer reviews, be sure to have the right people perform the reviews, use tailored checklists, and spend the appropriate amount of time performing the reviews.

Software Personal Reviews

Software personal reviews follow the same methodology as peer reviews with the exception that they are performed by the author of the work product. Using the same techniques as peer reviews, individuals can become super-reviewers of their own work, sometimes finding more defects than are found in peer reviews. Personal reviews should be conducted just prior to peer reviews.

Static Analysis

Software peer reviews, personal reviews and testing are highly effective at finding certain types of defects, but they can be weak in finding other types of defects. Modern automated static analysis tools are an excellent supplement to manual methods of defect detection. Static analysis tools vary significantly in what they find. Some are more focused on style issues while others are more focused on defects that can cause the system to malfunction. When evaluating and selecting static analysis tools, organizations should pay special attention to the types of defects the tool is finding, and how many false positives the tool is generating. Selecting a tool that is focused on performance-related defects and one that produces as few false positives as possible typically results in the most effective implementation.

Software Root Cause Analysis

Injecting defects while developing requirements, designs, code and test procedure is a natural occurrence since software development is still mostly a human-intensive process. Understanding the types of defects being introduced during the process can significantly increase the effectiveness of peer reviews, personal reviews, and testing.  As defects are found and fixed, they should be recorded in a database. They should be evaluated to determine key characteristics such as severity, phase introduced, phase found, type of defect, what caused the defect to be introduced, how could it have been found earlier, and how could it have been prevented.

Software Process Metrics

Certain aspects of the software development process can be understood intuitively, but other aspects of the process can be counterintuitive. For example, adding in design activities, personal reviews and peer reviews usually produces an intuitive sense that the development work will take longer. For most of our day-to-day activities, adding an activity to our to-do list usually does result in more time being expended. When design activities, personal reviews and peer reviews are implemented properly and effectively, safety and quality will increase, and the overall development time will usually stay about the same or decrease as a result of reduced amounts of defects and associated rework. In order to understand these dynamics, basic metrics should be collected to confirm the effectiveness of the processes being implemented. These metrics include time spent on specific activities, size of work products, defects injected and found, and schedule deviation.

Software Coordinator Roles

In order to monitor the effectiveness of the software development process, someone must oversee key practices and regularly report to the team on key information regarding these practices. A good starting point is to assign an individual and perhaps a backup person to monitor software safety and quality. These individuals would collect the basic metrics during the lifecycle activities and perform an analysis on the data.

Schedule Estimation

One of the key challenges in all of development is estimating how much effort will be required to deliver certain functionality. Human nature typically results in optimistic estimates, relying on processes to occur right in order to make schedule commitments. Using a structured team-based approach to making estimations that leverages historical data leads to very accurate estimates, which are extremely important to safety and quality. If efforts are underestimated, there is a tendency to rush through activities, and oftentimes, these involve the safety and quality activities.

Include Quality Activities in the Plan

Finally, including the specific quality activities and the appropriate amount of time in the plan is essential. This will allow team members to feel comfortable spending time on the activities resulting in effective outcomes. If these activities are not included in the plan and individuals are asked to find time to conduct these activities, they are usually rushed or not done at all.

Process Evaluation

Take a moment to rate your organization or individual projects on their effectiveness of the implementation of these activities. Consider looking more into those areas that appear to bring some benefit to your software processes.