Assessing The Performance Characteristics of Handheld Ultrasound in A Rheumatic Heart Disease Screening Program


 Background

Rheumatic heart disease affects 33 million people in low and middle income countries and is the leading cause of cardiovascular death among children and young adults. Penicillin prophylaxis has been shown to improve valvular function among patients with clinically silent or mild disease. Efforts to expand echocardiographic screening are focusing on simplified protocols, non-physician ultrasonographers, and portable ultrasound devices, including handheld ultrasound. Recent advances support the use of single-view screening protocols. With the increasing availability and low cost of handheld devices, prospective studies are needed to evaluate their performance in these settings.
Methods

 We conducted a cross-sectional study among 19 at-risk school-children participating in a rheumatic heart disease screening program in Ethiopia comparing a handheld ultrasound device (Phillips Lumify) to a fully-equipped portable ultrasound machine (Sonosite M-Turbo).
Results

Agreement between devices was similar for expert and non-expert review (84%). However, when reviewed by a non-expert the Lumify identified fewer screen-positive cases (p-value 0.083). We also compared non-expert to expert interpretation by device and found a significant difference in interpretation for the Lumify (p-value 0.025). There was a trend towards shorter jet length by color Doppler in the handheld ultrasound device for both expert and non-expert review.
Conclusions

Our study highlights the importance of using caution when applying proposed single-view criteria as the sensitivity and specificity may be affected by the device.


Background
Rheumatic heart disease affects 33 million people in low and middle income countries and is the leading cause of cardiovascular death among children and young adults. 1 Rheumatic heart disease also has substantial economic consequences and early identi cation could help to narrow disparities within affected populations. 2,3 Rheumatic heart disease develops as a consequence of acute rheumatic fever which primarily affects children and will lead to rheumatic heart disease in 60% of cases. 4 Recurrent attacks of acute rheumatic fever accelerate the progression of valvular damage and secondary prophylaxis with monthly penicillin is an effective strategy to improve survival. [5][6][7] Penicillin prophylaxis has been shown to improve valvular function among patients with clinically silent or mild disease. 8 Echocardiography has become the standard of care for detecting rheumatic heart disease as it can identify ten times the number of affected children when compared to physical exam. 9 Echocardiographic screening of school-aged children in low and middle income countries has demonstrated a large burden of early rheumatic heart disease thus providing a critical time period for early intervention with penicillin and the prevention of rheumatic heart disease related deaths. 10 In 2012, the World Heart Federation updated their guideline for the diagnosis of asymptomatic or latent rheumatic heart disease including criteria for 'de nite' and 'borderline' disease. 11 Using this guideline, cross-sectional echocardiographic studies have found a substantial burden of disease around the globe. [12][13][14][15][16] While the World Heart Federation criteria serve as the gold standard for making a diagnosis of rheumatic heart disease, they are impractical for population-based screening programs due to time, cost, and resource availability. 17 Efforts to expand echocardiographic screening are focusing on simpli ed protocols, non-physician ultrasonographers, and portable ultrasound devices, including handheld ultrasound. 18 For example, Beaton et al. demonstrated that pediatric cardiologists employing a simpli ed handheld ultrasound protocol can achieve high diagnostic accuracy. 19 However, shortages of echocardiographers and cardiologists in many endemic regions limit the expansion of population-based screening programs. 20 Efforts to expand the screening responsibilities to include a non-physician workforce could overcome this limitation. 21 Through focused training, non-physician ultrasonographers using simpli ed screening protocols can achieve a high sensitivity and speci city for detecting early rheumatic heart disease. [22][23][24][25] Studies evaluating single image protocols with color Doppler of the mitral and aortic valves have demonstrated sensitivity of 73-92% and speci city of 75-100% for latent rheumatic heart disease and screening times of 2-4 minutes. 26-28 However, few published studies have prospectively evaluated the real-world feasibility of single-view screening. One study from Sudan, showed that a single view protocol among 1400 children produced only four false positive screens and took an average of four minutes per exam. 13 More recently, an augmented single-view screening protocol was prospectively evaluated in a cohort of school-children in Timor-Leste and demonstrated a sensitivity of 100% and a speci city of 95%. 29 However, rigorous studies have not evaluated the performance of simpli ed screening protocols on representative cohorts of at-risk children. Furthermore, how hand-held devices will perform for the detection of latent rheumatic heart disease has not been adequately explored. While there is increasing evidence that single view screening protocols can adequately detect latent rheumatic heart disease, there is no consensus on the criteria that de ne a positive scren. 27,29,30 Due to inherent limitations in handheld ultrasound technology, criteria that perform well with one device cannot be assumed valid for another. To explore how a hand-held device compares to a fully equipped portable ultrasound device, we conducted a cross-sectional comparative study on a sample of schoolchildren to assess for differences between ultrasound devices when executing a single-view screening protocol.

Study Setting
This cross-sectional study took place in November 2019 in Soddo, Ethiopia. Starting in April of 2019, Soddo Christian Hospital began operating a rheumatic heart disease screening program. The staff of Soddo Christian Hospital coordinate with local school leadership and perform school-based screenings 2-3 days weekly. The children are screened by grade level and gender. Any child found to have a positive screening ultrasound is referred to Soddo Christian Hospital for a con rmatory echocardiogram free of charge. If the con rmatory echocardiogram demonstrates de nite or borderline rheumatic heart disease, the children and their guardian(s) are noti ed. A brief educational session is provided and the patient is provided ongoing monthly injections of penicillin. Six locally trained rheumatic heart disease screeners under the supervision of an onsite physician perform the screening ultrasounds. The screeners were recruited from a wide range of hospital staff including nurses, technicians, receptionists, and sanitation personnel. All screeners completed a mentored training program to become pro cient in executing a single parasternal long-axis view of the heart with and without color-Doppler.

Study Design
Over a two-day period, our study team accompanied the screeners as they performed a school-based screening. On both study days, all children screened were between the ages of 15-18 and from the same school. For the screening, a portable Sonosite M-turbo ultrasound machine was used with a 5-1Mhz phased array ultrasound probe. A Nyquist limit of 72 cm/s was used for color Doppler images with a frame rate of 16.667 Hz. All children determined to be positive by the screening team, as well as a random sample of children undergoing screening, were selected for our study. To minimize differences observed between devices, all children underwent repeat screening echocardiogram using the Sonosite Mturbo by a trained pediatric cardiac sonographer. This same echocardiographer then used a Phillips Lumify hand-held ultrasound device using a S4-1 Mhz phased array probe. For color Doppler imaging, the Lumify device has a xed Nyquist limit of 60 cm/s and an auto-adjusting frame rate. Two and three second video clips were stored on the Sonosite and Philips devices respectively. Images were then transferred onto encrypted ash drives and transferred onto secure hard drives for analysis. Clips contained a parasternal long-axis view of the heart with and without color Doppler. No associated demographic information was stored, and a random number was used to link the individual between devices. Where appropriate, separate images were saved for the aortic and mitral valves. Our study intervention did not alter the recommendations of the screening team and all children determined to be positive by the screening team were referred per protocol for con rmatory echocardiogram.This study was reviewed by the University of Minnesota Institutional Review Board and approved as a non-human research subject study.

Study tool
All identifying information was removed from the stored video clips. Due to obvious differences in image quality and clip duration, interpretation could not be blinded by device. All images were randomly arranged for analysis so that the interpreting study investigator was not able to compare images from one individual to another. All data entry occurred through REDCap. A 16-item interpretation survey was designed to capture the required elements for determining if the ultrasound was screen positive or screen negative. For the purposes of this study, a screening ultrasound was positive if the following criteria were met: 1. A pansystolic and multicolored regurgitation at the mitral valve by color Doppler estimated at a length of more than 1.5 cm. If the regurgitation was eccentric, an estimated length of more than 1 cm was considered positive; 2. Any regurgitation at the aortic valve; 3. Any valvular abnormalities consistent with rheumatic heart disease. These criteria were derived from the ndings of published studies and expert consensus. 26,27,29 Our full survey tool can be reviewed in the supplemental materials. Two study investigators reviewed all ultrasounds, one an experienced cardiologist (R.J., expert) and the other an internal medicine and pediatric hospitalist (Z.K., non-expert) with experience in the use of point-of-care ultrasound. This design was used to capture differences in non-expert interpretation as might occur during routine school-based screenings.

Statistical Analysis
The primary objective was to determine the agreement between devices for each reader. For each reader, the agreement between devices was summarized and compared using McNemar's test for paired samples. Mitral valve (MV) and aortic valve (AV) abnormalities were summarized for positive agreement between devices, negative agreement between devices and disagreement between devices. To investigate the agreement in MV and AV abnormalities between devices for each of these three scenarios and for each reader, McNemar's test for paired samples was used. All reported p-values are two-sided and signi cance level of 0.05 was used. Statistical analyses were performed using R (version 3.6.1) and SAS (version 9.4, SAS Institute Inc., Cary, North Carolina).

Results
Nineteen children underwent ultrasounds with the M-turbo and Lumify devices. All ultrasounds contained adequate visualization of the mitral valve to assess for soft tissue and color Doppler abnormalities. All ultrasounds contained adequate imaging of the aortic valve. Four ultrasounds (3 M-turbo and 1 Lumify) contained inadequate color Doppler of the aortic valve by expert review. Expert interpretation demonstrated 84% agreement between the devices (Table 1). There was no association between screen status and the ultrasound device being used (p-value 0.564). All disagreement was related to differences in the appearance of the mitral or aortic valves (  Five children had screen positive ultrasounds with both devices. Four children screened positive with mitral regurgitation and one child had aortic regurgitation (Table 3). All jet lengths at the mitral valve for the Lumify were between 1-1.5 cm. For the M-turbo, one jet length was 1.5-2 cm while the remaining were between 1-1.5 cm. Among the eleven children with screen negative ultrasounds on both devices, screennegative mitral regurgitation was signi cantly more common with the M-turbo (Supplemental tables).  (Table 1). However, the M-turbo device was more likely to be associated with a positive screen and this trended towards statistical signi cance (p-value 0.083). Disagreement was related to differences observed at the mitral valve ( Table 2). Two children had screen positive ultrasounds by non-expert review on both devices. One child had pathological mitral regurgitation and one child had aortic regurgitation. Findings for these children were similar to expert review (supplemental tables). Among the 14 children with screen negative ultrasounds on both devices, screen-negative mitral regurgitation was more common with the M-turbo but did not reach statistical signi cance (Supplemental tables).
Among screen positive ultrasounds with abnormal mitral regurgitation, there was a trend in the M-turbo device towards longer estimated jet lengths by color Doppler when expert and non-expert interpretation was combined (Figure 1). When comparing non-expert to expert interpretation by device, the Lumify was more likely to be interpreted as screen negative (p-value 0.025, table 4). We undertook a detailed evaluation of the three children in which there was discrepancy in screen status on the Lumify (Table 5). There were important differences between expert and non-expert interpretation of the color Doppler images which impacted the screen status. Review of the observed mitral regurgitation in these children demonstrated more subtle color Doppler ndings when viewed using the Lumify device ( Figure 2).

Discussion
Among a small but representative sample of children at risk for rheumatic heart disease, our study demonstrates that there are important and measureable differences between two ultrasound devices, the handheld Philips Lumify and the Sonosite M-Turbo. When reviewed by an expert cardiologist, these differences were evenly split between devices and involved subtle differences in myocardial tissue imaging. When reviewed by a non-expert, the Lumify was more consistently associated with a screen negative interpretation. This difference trended towards statistical signi cance. When comparing screen status by device, there was a statistically signi cant difference between expert and non-expert interpretation using the Lumify. When images from the Lumify were reviewed by a non-expert, mitral jets were less likely to be categorized as pansystolic and estimated at a shorter length. Overall, mitral jets trended towards appearing shorter when viewed on the Lumify. Among screening ultrasounds that were categorized as negative, screen-negative mitral regurgitation was seen more commonly on the M-turbo.
Single-view screening protocols for the detection of latent rheumatic heart disease are increasingly demonstrating strong sensitivity and speci city. 27,29 However, most studies to date have not evaluated handheld devices in a prospective manner. With the increasing availability and low cost of these devices, teams around the world have already started to employ modi ed versions of published single-view screening criteria. 13 While these efforts are needed to help realize the potential of population based screening for rheumatic heart disease, our study demonstrates that there are important differences between ultrasound devices that may impact the sensitivity and speci city of single-view protocols especially when non-experts are interpreting the images. While expert review did not demonstrate perfect concordance between devices, the disagreement was isolated to myocardial tissue imaging abnormalities. These discordant ndings highlight the importance of myocardial imaging optimization during screening protocols. For example, one discrepancy between the devices occurred because the mid eld gain was too low on the M-turbo. This led to an inability to detect the valvular abnormality, even on expert review. However, prior research demonstrates that follow-up of screen positive ultrasounds for isolated valvular abnormalities are most likely borderline by World Heart Federation criteria and much less likely to progress on follow-up echocardiograms. [30][31][32][33][34] Our study has many important limitations. While all images were captured by the same experienced pediatric cardiac sonographer, differences in image acquisition may explain some of the differences observed. Some differences are expected between expert and non-expert interpretation of echocardiograms for rheumatic heart disease screening. 22,24,35,36 However, it is notable that agreement regarding screen status on the M-turbo device was extremely high between reviewers with no case of screen-positive valvular regurgitation being missed by non-expert review suggesting that differences in appearance between the ultrasound devices explains the discordance in screen status. Given the importance of mitral regurgitation in the early detection of rheumatic heart disease, this trend when evaluated over a larger sample of children, could have signi cant impacts.
There are also inherent differences between the devices. The Phillips Lumify has a xed Nyquist limit of 60 cm/s whereas the Sonosite M-turbo can be adjusted. For the purposes of this study, the Nyquist limit on the M-turbo was set a 72 cm/s. The Nyquist limit on the M-turbo was not adjusted to minimize interruptions to the standard imaging settings already employed by the screening team. Interestingly, our observed trend towards lower jet lengths in the Lumify are not readily explained by the lower Nyquist limit as one would expect increased aliasing and artifact given the high velocities being observed. 37 Reducing the Nyquist limit in the M-turbo would have exacerbated the observed discrepancies in jet length which suggest the jet length discrepancy is related to other factors. There are also differences in the frame rate between devices. Review of the color Doppler and myocardial images from the M-turbo show a xed frame rate of 16.667 hz. In contrast, the Lumify device automatically adjusts the frame rate depending on several factors. In our study, the frame rate ranged from 14-17 Hz during color Doppler acquisition but was much higher, 29-32 Hz, during myocardial imaging. It is possible that a lower frame rate could result in a shorter jet length by color Doppler. On a per second basis, there was a maximum of approximately two fewer frames in the Lumify device. These frame rates are very comparable and unlikely to explain more than trivial differences in jet length or duration. While the frame rates during myocardial imaging are much more substantial, the impacts of this were not apparent on our analysis but may have been limited by the small sample size. Our study sample was too small to evaluate for the impact of the higher frame rates on the Lumify during myocardial imaging.
Single-view screening protocols need to weigh the balance between sensitivity and speci city. The differences observed in our study suggest that the thresholds used to determine the ideal positive screen may be in uenced by unique and unpredictable performance characteristics of the ultrasound device being utilized. Few other studies have directly compared the imaging characteristics of handheld ultrasound devices to a fully equipped device in a real world-screening environment. Our study highlights that the ideal criteria for determining a screen positive ultrasound may vary by ultrasound device. This may be more important among screening programs that utilize non-expert ultrasonographers to carry out school-based screenings. Criteria that move away from length based estimations of color Doppler jets and focus on other characteristics of pathological regurgitation seen in latent rheumatic heart disease may be preferable. 29 Ongoing research efforts examining the Philips Lumify device are underway and may help shed light on the appropriate criteria for single-view screening protocols and handheld ultrasound devices. 38 Until further evidence is available, the use of handheld devices for rheumatic heart disease screening should be approached cautiously to avoid missing cases that would otherwise be referred for con rmatory echocardiograms.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. SupplementalTables.docx