Developing High Quality Medical Device Software Leveraging The Team Software Process

By Carl Wyrwa
December 15, 2012


• Categories: 04-Product Design & Development, 05-Quality/Regulatory

This is the first in a series of articles that will provide insight into the Team Software Process (TSP), its applicability to the medical device industry, and how it helps achieve the high level of software quality required for medical devices. This article will provide an overview of the TSP and will highlight some of the key quality elements of TSP.

Quality is the primary focus of the TSP and quality is essential to achieving a safe and effective medical device. So what is the best way to develop software in order to achieve the highest levels of quality?
In a recent study of different software development methods and quality practices, the TSP and Personal Software Process (PSP) were determined to have the greatest success for medium to large applications and were rated second for small applications.¹ Organizations developing software for the first time and organizations seeking to improve their current software development practices can benefit significantly from the TSP/PSP. Those adopting the TSP/PSP can rest assured that they are developing software using methods that are recognized as the best available today. It is important that medical device software development organizations take a look at the TSP/PSP and benchmark their current practices against the TSP/PSP to determine if they can benefit from it.

The Team Software Process and Personal Software Process

The Team Software Process was developed by Watts S. Humphrey at the Software Engineering Institute of Carnegie Mellon University and was introduced in 1998. The Personal Software Process, which is an integral part of the TSP, was also developed by Humphrey in 1993. He was awarded the 2003 National Medal of Technology for his work in software engineering, receiving it from President George W. Bush in 2005.²

For teams of engineers developing a software-intensive product, the TSP provides a framework that combines their personal process discipline skills with proven process management techniques.  This enables them to produce high-quality software as a team.³ The PSP teaches individual software engineers an approach to developing software that yields high quality software and accurate schedule predictability.

What are the benefits of the TSP?

The primary benefits of the TSP are twofold, much higher quality and much more reliable schedule estimates. Quality is paramount in the medical device field, however, both quality and accurate schedules are critical to the success of a medical device business.

Quality levels can be significantly improved using the TSP.  Figure 2 shows that some TSP teams have achieved a post-release defect density level of 0.06 defects per 1000 lines of code – that’s only 60 remaining defects in one million lines of code.  Very mature processes, like those implemented by CMM Level 4 and 5 organizations, yield between 2.28 and 1.05 defects per 1000 lines of code respectively.  This demonstrates that TSP projects can achieve a much higher level of quality than other very mature practices – in some cases by a factor of ten or more.

Organizations must be able to accurately predict how many developers they will need and when they will finish. Figure 3 shows the effort and schedule estimation levels that TSP teams have achieved relative to the non-TSP teams evaluated.  TSP teams have achieved schedule deviation ranges of -8 percent to +20 percent, where non-TSP teams showed schedule deviation ranges of +27 percent to +112 percent with almost all projects finishing late, some taking more than twice the original estimated time to complete.

Key differences between the TSP and other methodologies

There are many unique elements of the TSP (Figure 4), some of the most important are:

1. Quality Planning. The TSP Quality planning process realizes that defect injection is inevitable. The team develops an estimate of the number of defects that will be injected during development and then builds a plan to remove those defects.  One of the key philosophical differences is that TSP teams attempt to remove the vast majority, if not all, of the defects prior to software system testing.
2. Planning and Estimating. The planning and estimating are done by the individual team members themselves guided by a TSP coach. This results in team members taking much more ownership of their plans and commitments.  Over time their estimates become more accurate as each team member improves their estimation skills through the use of actual data.
3. Personal Software Process. Individual developers are specifically trained in personal techniques for producing high-quality software with improved schedule predictability.
4. Self-Managed Teams. Team members participate in the overall management of the project.
5. Metrics Framework and Data. Four simple measures are captured while people are doing the work.  These measures allow individuals and teams to really understand where they are and how the process is working. The TSP includes a comprehensive set of quality and planning metrics.
6. Coaching. A trained TSP Coach works side-by-side with the team as they implement the TSP.  This ensures that the team is performing all of the TSP activities as planned and that each individual is aware, through their personal metrics, how well they are performing and where they can improve.

Continue to page 2 below.

How does the TSP address Quality?

“The primary focus of TSP team members is on producing high quality components and products”.³ The TSP leverages defect prevention methods and early defect removal methods. The strategy is to inject fewer defects during development and to find and fix the remaining defects earlier and much more effectively prior to software system test.

The TSP recognizes that software development is intellectual work done by people who, because they are human, consistently inject defects as they do their work. The TSP quality planning process takes this into consideration and develops an estimate of the number of defects that will be injected and builds a plan to remove those defects.

Many organizations assume that the software system test team will find several defects during their testing. This can lead to a very time-consuming and expensive testing phase where it can be very difficult to determine when the product will be ready.

One of the key philosophical differences is that the TSP attempts to remove the vast majority, if not all, of the defects prior to software system testing. When a group is challenged with a goal of “Near-Zero defects delivered to system test”, they look for a way to accomplish this. This results in significantly fewer defects found in system test and shorter system test durations. The TSP provides the skill training and essential framework enabling organizations to achieve such superior software quality performance.

Another essential element of the TSP planning is that the plan includes all of the early defect removal practices in it and allocates the appropriate amount of time to perform those activities effectively. Many organizations build schedules that include the development activities of requirements, design, and code but often times leave it up to the developers to “find time” to peer review these work products.

Microsoft IT has achieved significant quality improvements using the TSP quality practices. The number of defects in System Test was reduced from 473 found in Version 2.4 down to 10 found in Version 2.5 (Figure 6). Defects found in User Acceptance Test were reduced from 153 down to 3.

Doesn’t this approach take longer?

The simple answer is no. Correctly using the TSP approach gives you the quality and schedule predictability benefits while reducing the amount of expensive rework typically experienced at the end of the lifecycle. It is human nature to quickly jump to the conclusion that if you add more process steps to your current process, things will take longer. This actually does occur in many of our day-to-day activities. The opposite happens with the TSP.

With TSP, defects are removed earlier in the process at a much lower cost than removing them later in the process. Defects removed by early reviews and inspections take a fraction of the time to find and fix as compared to removing them in system test (Figure 6).

Intuit has achieved significant productivity improvements through the use of TSP (Figure 7). Doing it right in the first place takes less time.

Why aren’t more companies using the TSP?

It appears that the use of the TSP is very limited in the medical device arena. The TSP offers higher quality, better schedule predictability, and increased productivity – so why aren’t more people using it? The reasons are not so clear.

Informal shows-of-hands at conferences, where TSP presentations have been given, show very little awareness of the TSP by the medical device community. One of the purposes of this series of articles is to raise awareness of the TSP as a leading software development method available for the medical device community to leverage, and to encourage people to take a look.

Some organizations already have process or quality improvement initiatives in progress and are reluctant to move away from these efforts. TSP is very flexible and can easily be integrated into and supplement existing efforts.

It is human nature to assume that more structured processes with more steps might take longer. As discussed earlier, this is typically not true at all with TSP teams (Figure 7).

TSP/PSP and Agile

Some organizations have implemented Agile methods. TSP/PSP and Agile can be a strong combination. According to McHale, “When you investigate the details, you will see that these two methods at least, Agile and PSP/TSP, are not the options of an either-or choice, but compatible practices that have produced some remarkable results.” “What they may not know (yet) is that Agile co-exists quite comfortably – one might even say “thrives” – when applied with PSP/TSP”.⁶ With the recent release of the “Guidance on the use of AGILE practices in the development of medical device software,” this could be the perfect marriage.⁷

Is the TSP for you?

• How well do your teams produce reliable schedules?
• What are your current levels of defects found in software system test?
• How many defects do your customers report?
• How many customer-reported defects result in adverse events reportable to the Food and Drug Administration?
• What would your world be like if you:
  • Could reliably estimate the completion date of a project?
  • Could significantly reduce the number of defects found in software system test?
  • Could significantly reduce the number of defects found by your customers?
• What would this ultimately mean to patient safety?

References

1. Jones, C. (2010). Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies, McGraw-Hill.
2. National Medal of Technology Winner Watts Humphrey (2010). 1927-2010 (Software Engineering Institute, Carnegie Mellon University, University News Items, October 28, 2010).
3. Humphrey, W.S., Chick, T.A., Nichols, W., and Pomeroy-Huff, M. (July 2010) Team Software Process (TSP) Body of Knowledge. SEI Technical Report CMU/SEI-2010-TR-020.
4. Over, J. (2010). Team Software Process, (Software Engineering Institute, Carnegie Mellon University).
5. Over, J. (2010). Introduction to the Team Software Process, (Software Engineering Institute, Carnegie Mellon University).
6. McHale, J. (2010). Capers Jones’s New Book Ranks PSP/TSP and Agile as Top Method. (Carnegie Mellon University).
7. Guidance on the use of AGILE practices in the development of medical device software. (2012).  Association for the Advancement of Medical Instrumentation, Arlington: Virginia.

Additional Resources

Humphrey, W.S. and Over, J.W. (2011). Leadership, Teamwork, and Trust: Building a Competitive Software Capability. Addison-Wesley.

Special permission to reproduce portions of the “Team Software Process” ©2010 and the “Introduction to the Team Software Process” ©2010 by Carnegie Mellon University, has been granted by the Software Engineering Institute. The Personal Software Process(SM), PSP(SM), Team Software Process(SM), and TSP(SM) are service marks of Carnegie Mellon University.