At the 2023 Military Health System Research Symposium (MHSRS) in Orlando, Florida, researchers with the Medical Device “Plug-and-Play” Interoperability & Cybersecurity Program (MD PnP) at Massachusetts General Hospital (MGH), presented a poster outlining a systematic testing method for remote device operation that can be used by developers and system integrators to ensure the safety of such systems.

For “Systematic Testing of a Ventilator Remote Control System Towards Safe Use in Tele-Critical Care and Prolonged Care,” Nihon Kohden OrangeMed (NK) and DocBox, supported by the MD PnP program, developed a prototype system for network-based far remote-control of the NKV-550 critical care ventilator, with the goals of identifying and implementing foundational remote-control capabilities, and exploring essential performance, interoperability, and cybersecurity requirements, needed by Tele Critical Care (TCC) systems in public health emergencies, disasters, and for prolonged casualty care (PCC).

Equipment

The NK-DocBox ventilator remote control system used in the study includes:

• The commercially available NKV-550 ventilator enhanced with a network communication protocol, as well as necessary software and hardware updates, to enable remote control.
• The DocBox app, a remote-control application on the DocBox Apiary interoperability platform—a bedside interoperability platform conforming to the AAMI 2700-1 Integrated Clinical Environment (ICE) standard—to control the NKV-550 from a location close to the patient and ventilator (“NEAR-PATIENT REMOTE CONTROL”).
• The DocBox web portal, a web-based application that communicates with the Apiary platform at the patient’s bedside and in turn controls the NKV-550 ventilator connected to it from anywhere across the Internet (“FAR REMOTE CONTROL”).

Testing Strategy

The MD PnP team established a simulated clinical use environment in the MD PnP lab to verify the overall NK-DocBox system at the component, integration, and system levels. Testing included:

• Communication protocol testing to evaluate functionality and robustness of the network protocol in NKV-550 to enable safe remote control.
• System integration testing to assess correctness and resilience of the NKDocBox system in normal and simulated austere use environments, and quantify minimal network requirements for the system to function.
• Usability testing to evaluate the human factors design of the DocBox app and web portal with the targeted user population, which was Registered Respiratory Therapists (RRTs).

Communication Protocol

A remote control app was developed on the OpenICE interoperability platform to verify the communication procedure, data terminology/adequacy/frequency, and control functions of the NKV- 550 network protocol. The remote-control app was modified to transmit to the NKV-550 ventilator both normal and ill-formed control commands at varied frequencies to assess remote-control robustness against possible transmission errors.

The testing confirmed that the NKV-550 network protocol transmits sufficient device, patient and alarm data and reliable remote-control capability to enable a remote caregiver to safely provide virtual TCC. In addition, the ventilator can work as expected when remote controls are received as frequently as once per second; and can ignore remote controls that attempt to set invalid ventilation modes or out-of-range ventilation parameters.

System Integration/Resilience Testing

Two separate LANs were set up in the MD PnP cyber range to deploy the NKV-550 ventilator + DocBox Apiary ICE platform and the DocBox web portal, respectively. A high-precision PacketStorm 8XG WAN emulator was placed in the data pathway of both LANs to pre-process network traffic and inject various network disruptions (e.g., bandwidth throttling, delay, and jitter).

By executing test cases using the DocBox Apiary app and web portal to change in NKV-550 ventilator modes, parameters and alarm settings (i.e., both near-patient and far remote control), the researchers confirmed that:

• The minimum network Quality of Service (QoS) required for the NKDocBox system is ≥12Mb/s bandwidth, ≤ 500ms delay, ≤ 100ms jitter, ≤ 1% data drop rate, and ≤ 1e-6 bit error rate.
• When the minimum QoS was met, both the DocBox app and web portal correctly controlled NKV-550 with minimal appreciable delay.
• When the minimum QoS was not met, transmission of controls to the ventilator was delayed, and visualization of ventilator data was noticeably degraded (e.g., waveforms were skewed, frozen, and/or out of sync with the ventilator), or the connection with the ventilator failed. This performance degradation could affect the remote user’s ability to perform TCC, including remote control (but would be evident to the remote user).

Usability Testing

The NK and MD PnP teams conducted simulated use testing with 15 respiratory therapists, who used the DocBox web portal from their office in Boston to control an NKV-550 ventilator located in Irvine, California, or vice versa. All participants watched a three-minute training video before testing. During the testing, they were asked by a trained facilitator to perform a set of 19 tasks, such as view/interpret ventilator data, adjust ventilator settings, and respond to alarms.

The participants successfully completed all 285 tasks (100%) without any use error observed that might lead to a hazardous situation, and without assistance from the facilitator. And all participants confirmed the ease-of-use of the DocBox web portal, with only two difficulties observed regarding locating the ventilator remote control tab and ventilator numeric measurement sections of the UI.

The researchers concluded, “The testing presented in this poster effectively verified critical system properties that developers, system integrators, and end users need to consider to ensure the safety of these technologies. Manufacturers can use these novel test methods, which the MD PnP program continues to improve and provides as a service, to assess the safety and security of their novel remote-control technologies towards real-world adoption.”