In vivo validation studies, currently the standard for pulse oximeter performance validation, are expensive, time consuming and with significant limitations including the inability to study performance during severe anemia. This study is the first to use a novel in vitro circulation system (designed and built by Kestrel Labs, Inc.) to demonstrate degradation of oximeter performance during severe anemia for three commercial pulse oximeters of varying cost.
Currently there are a small number of in vitro calibration devices that have been reported and few that are commercially available. Attempts to develop an in vitro calibration system date back to as early as the 1990s. Reynolds and colleagues developed an in vitro test system to study the accuracy of 10 different oximeters at low oxyhaemoglobin saturations [17]. De Kock and Tarassenko developed an in vitro blood circuit with a flexible cuvette to investigate theoretical models of optical transmission in whole blood [18]. Hornberg et al. developed a novel pulse oximeter calibration technique utilizing a spectral light modulator as a calibration standard [19].
Commercially available devices (including the Fluke Biomedical ProSim 8 [20] or SPOT Light SpO2 Functional Tester [21] and WhaleTeq AECG100 [22]), are frequently misunderstood and potentially inappropriately utilized by researchers hoping to quickly ‘validate’ the performance on an oximeter. While these devices do play an important role in device development and performance verification, existing devices are intended to validate performance for devices that have calibration curves pre-programmed into the testing device. They provide an optical signal to verify that the electronics within the pulse oximeter probe are functional during preventative maintenance checks on patient monitors in service. In other words, if the oximeter is known to the testing device, then the testing device can assess if the oximeter performs against a simulated signal in an expected way. According to manufacturer documentation, they are not intended to be used to calibrate medical equipment. They should not be used to assess performance of pulse oximeters unknown to the testing device. No commercially available devices use real blood. Current work is underway to better characterize and improve the utility of commercially available in vitro devices.
In our study, the most expensive device tested (Masimo Radical) maintained good accuracy at all but the most extreme anemic hematocrit level. The intermediate cost device (Acare v1.5) was not as accurate as the Masimo, and inaccuracies appeared at relatively higher levels when compared to the Masimo device. The least expensive device studied (CMS 50-DL) was the least accurate with the poorest dynamic range of the three oximeters.
All oximeters tested showed loss of pulse oximeter accuracy with decreasing hematocrit, but notably these error trends did not occur in the same direction for all three oximeters. This may reflect oximeter design as opposed to an issue with IVCS performance, and further investigation with additional oximeters would elucidate this observation.
The study had several limitations, the most significant of which is the performance of the IVCS device. In comparison to previously published IVCS devices, the IVCS used in the current study proved to be the most accurate to date [11], [23]. Nonetheless, performance did not equivalently reproduce in vivo performance. Thus, a zeroing factor was utilized for our analysis, and as was the case for the least expensive oximeter (CMS 50-DL), we had to place a neutral density filter in the optical path for the device to produce a result. The Acare unit displayed the correct heart rate, but the audible pulse indicator worked approximately every other beat. The applicability of our IVCS findings to real world performance of these pulse oximeters in clinical settings is unclear.
Future work is needed to continue to refine the performance of IVCS to eventually produce a system that more precisely mimics in vivo performance. An additional limitation of the study was the necessary deconstruction of the oximeter probes for attachment to the IVCS. Probe positioning and design is an important real-world factor in oximeter performance which we could not completely account for in this study.