Accuracy of medical student measurements of CT right-to-left ventricular diameter in patients with acute pulmonary embolism

doi:10.21203/rs.3.rs-1443670/v1

Download PDF

Research Article

Accuracy of medical student measurements of CT right-to-left ventricular diameter in patients with acute pulmonary embolism

https://doi.org/10.21203/rs.3.rs-1443670/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background:

Acute pulmonary embolism (PE) is a common disease with varying presentations, necessitating risk stratification to determine management. Measurement of right ventricle (RV) to left ventricle (LV) ratio >=1.0 on computed tomography pulmonary angiogram (CTPA) suggests RV strain, which may indicate a worse prognosis. Two prior studies showed that residents with brief training by a radiologist could accurately measure RV/LV ratio on CTPA. In this study, we assessed whether medical students could accurately measure RV dilatation on CTPA.

Methods:

We conducted a post hoc analysis of a retrospective cohort study of adults undergoing management for acute PE at 21 community Emergency Departments across Kaiser Permanente Northern California (KPNC) from 2013 to 2015. We created a sample of 120 patients from the original 2,387 patients, stratified to contain an equal number of patients from each of the five Pulmonary Embolism Severity Index classes. The sample had a proportional ratio of home/short stay to hospitalized patients. Four medical students measured RV and LV diameter on CTPA after a series of brief training sessions from an emergency medicine physician and an interventional radiologist. We used Cohen’s kappa statistics, Bland-Altman plots, and Pearson correlation coefficients to assess interrater reliability, comparing the student measurements with those of the radiologist.

Results:

Of the 120 CTPAs, 108 images were accessible and constitute the study cohort. Among the 108 CTPAs, 79 (73%) showed RV dilatation and 29 (27%) did not. The kappa statistic for the presence of RV dilatation of the medical students compared to the radiologist showed moderate agreement for 3 medical students (kappa (95% CI): 0.46 (0.21-0.70), 0.49 (0.31-0.68), 0.50 (0.32- 0.68)) and fair agreement for one medical student (kappa (95% CI): 0.29 (0.10- 0.47)). The average interrater differences in RV/LV ratio between a radiologist and each of the four medical students were -0.04, -0.05, 0.04, and 0.24. Pearson correlation coefficients were 0.87, 0.80, 0.74, and 0.78, respectively, indicating moderate correlation (p<0.001 for all).

Conclusion:

With brief training, medical students were able to identify RV dilatation on CTPA in moderate agreement with that of a radiologist. Further study is needed to determine whether medical student accuracy could improve with additional training.

The majority of patients diagnosed in the clinic and emergency department (ED) with acute pulmonary embolism (PE) are normotensive. Evidence of right ventricular (RV) dysfunction can risk stratify this population and help guide site-of-care and treatment decisions.¹ Computed tomography pulmonary angiography (CTPA), used to diagnose PE in approximately 95% of cases, can also contribute to prognostics by providing a concomitant assessment of right ventricular (RV) dilatation.^2,3 Compared with transthoracic echocardiography (TTE), the gold standard for RV evaluation, CTPA has been found to be sensitive for RV dilatation, but less specific.¹ CTPA, however, is often more readily available to outpatient clinicians than TTE. A recent large, patient-level meta-analysis found that RV dysfunction (assessed by any method) was associated with short-term death in otherwise low-risk patients with acute PE. The authors recommended that confirming the absence of RV dysfunction could help identify low-risk patients who may be candidates for outpatient management.⁴

Radiology reports of CTPA results, however, do not routinely include a description of the RV/left ventricle (LV) diameter ratio, often leaving the identification of RV dysfunction to the ordering clinician. Primary care and emergency physicians are not conventionally trained in evaluating RV dilatation on CT images. A retrospective cohort study found that emergency medicine physicians could be trained to accurately measure the RV/LV diameter ratio on CTPAs of ED patients with acute PE.⁵ The measurement of RV/LV diameter ratio has also been found to be reproducible by internal medicine residents with brief training.⁶ We inferred from these resident studies that evaluation of the ratio was relatively easy and that non-radiologists at any level of training could be taught to do this reliably. To test this hypothesis, we undertook a post-hoc analysis using CT images from a prior retrospective cohort study of patients undergoing management of PE at a large network of U.S. community hospitals. If our hypothesis was confirmed, it could have implications for medical education and strengthen the case for inclusion in emergency medicine (EM) residency curriculum.

Study design and setting

We performed post-hoc analysis of a prior retrospective cohort study of adult patients undergoing management of acute PE at 21 community medical centers across Kaiser Permanente Northern California (KPNC).⁷ KPNC is one of the largest community integrated health systems in Northern California, serving over 4 million members and totaling over 1.2 million ED visits per year. Kaiser Permanente members include approximately a third of the population in areas served and are highly representative of the diverse surrounding and statewide population.⁸ KPNC is a healthcare system with an applied research agenda and is supported by a comprehensive electronic health record (Epic, Verona, WI). The Research Determination Committee for KPNC decided that the project did not meet the regulatory definition of research involving human subjects per 45 CFR 46.102(d). Administrative permissions were required, and obtained, to access the raw data. The study was conducted in accordance with the principles of the Declaration of Helsinki.

Study aim

The aim of this study was to evaluate the accuracy of medical student measurements of RV dilatation on CTPA, after brief training, compared with those of an experienced radiologist. Our primary outcome was the presence of RV dilatation as defined by an RV/LV ratio of 1.0 or greater in the setting of acute PE.

Selection of participants

We accessed health records of participants with acute PE managed at 21 EDs within KPNC from January 2013 to April 2015. Inclusion and exclusion criteria were the same as the original retrospective study.⁷ From the original dataset of 2,387 patients, we divided patients into five PE Severity Index (PESI) classes, which correlate with 30-day all-cause mortality and initial site-of-care (home vs observation vs hospital).⁹ Within each class, we determined the proportion of patients who were discharged home either directly from the ED or within 24 hours from ED registration (N = 496).¹⁰ We sought to include 100 CTPAs, so set our sample size at 120 CTPAs to allow for the possibility that some image files would be corrupted or otherwise inaccessible. We then assembled a randomly-selected stratified sample of 24 patients from each PESI class (N = 120). The 120 images were accessed and reviewed.

Training

The study team consisted of two emergency medicine attending physicians, one experienced radiologist, three medical students of various training levels, and one premedical student. None of the students had prior radiology training. Prior to beginning data collection, all students attended three training sessions led by the same principal investigator (PI) to ensure that each was taught the same material in a similar fashion. The first session consisted of a group lecture on measurement of ventricle size on CT scan. Each student then independently completed a series of 10 practice cases. During the second individual training session the sample slide set was reviewed and teaching points were reiterated. The students were then given a second sample set to complete. For the third training session the students met with the PI and radiologist, reviewed the second case set, and learned how to use the Picture Archiving and Communications (PACS) system.

Image review

The student reviewers had access only to images. They did not have access to the electronic health record and remained blind to clinical data, including the original radiology report. The study radiologist reviewed all the images and his review was used as the gold standard.

Statistical analysis

Interrater reliability for the assessment of the primary outcome was determined using Cohen’s kappa statistics, with agreement determined as none (value < 0), slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00). In comparing each of the raters with the radiologist, we used Bland-Altman plots to represent the mean difference in RV/LV ratio measurements between the radiologist and each of the 4 raters. The Bland-Altman analysis is a simple and accurate way to quantify agreement between two raters. The Bland-Altman plot is a scatterplot in which the X-axis represents the average [(M1 + M2)/2], and the Y-axis represents the difference (M1 – M2) of two measurements. The scatterplot can be evaluated according to the scatter pattern. When the scattering of points is relatively close to the horizontal line, it indicates good agreement between two raters. The Pearson correlation coefficients were also used to represent the correlations in RV/LV ratio measurements between the radiologist and each of the 4 raters. We compared patient’s demographic and clinical characteristics by RV dilatation. Comparisons involving categorical variables were performed using the chi-square or Fisher’s exact test. Normally distributed continuous variable was compared using Student’s t test. All analyses were performed using SAS 9.4 (SAS Institute, Cary NC).

From the initial sample of 120 images, 12 studies were excluded because the image files were corrupted or inaccessible. The remaining 108 CTPAs underwent analysis. We report patient demographic and clinical characteristics in Table 2. The study radiologist determined that 79 patients (73%) showed RV dilatation on CT and 29 (27%) did not. Patients with RV dilatation had a higher mean age (64.4 ± 15.8 years) than those without RV dilatation (54.0 ± 16.7 years, p = 0.004). There were no other significant differences in patient characteristics and comorbidities. The proportions of home/short stay patients in each PESI class were shown in Table 1.

Table 1

Disposition of emergency department patients with acute pulmonary embolism stratified by severity index class
	CTPA review 120 pts
PESI class	Total	Home or short stay	Hospital admission	% of home/ short stay
1	24	9	15	37.5%
2	24	8	16	33.3%
3	24	5	19	20.8%
4	24	4	20	16.7%
5	24	2	22	8.3%
Total	120	28	92

Table 2

Characteristics of adult patients with acute pulmonary embolism whose computed tomography images were used in student training, stratified by right ventricular dilatation
		Right ventricular dilatation identified by study radiologist
Variables	Total (N = 108)	No (N = 29)	Yes (N = 79)	P Value
Age				0.004†
Mean ± SD	61.6 ± 16.7	54.0 ± 16.7	64.4 ± 15.8
Min-Max	22.0–91.0	23.0–78.0	22.0–91.0
Median (IQR)	63.5 (51.5–75.0)	58.0 (40.0–67.0)	65.0 (54.0–76.0)
Sex				0.102‡
F	53 (49.1)	18 (62.1)	35 (44.3)
M	55 (50.9)	11 (37.9)	44 (55.7)
Race/ethnicity				0.922§
White	70 (64.8)	18 (62.1)	52 (65.8)
African American	21 (19.4)	6 (20.7)	15 (19.0)
Hispanic	8 (7.4)	2 (6.9)	6 (7.6)
Asia	8 (7.4)	3 (10.3)	5 (6.3)
Cancer				0.780‡
0	73 (67.6)	19 (65.5)	54 (68.4)
1	35 (32.4)	10 (34.5)	25 (31.6)
Chronic lung disease				0.319‡
0	78 (72.2)	23 (79.3)	55 (69.6)
1	30 (27.8)	6 (20.7)	24 (30.4)
Congestive heart failure				0.175§
0	105 (97.2)	27 (93.1)	78 (98.7)
1	3 (2.8)	2 (6.9)	1 (1.3)
Obesity (BMI > 30)				0.159‡
0	64 (59.3)	14 (48.3)	50 (63.3)
1	44 (40.7)	15 (51.7)	29 (36.7)
Smoking				1.000§
0	100 (92.6)	27 (93.1)	73 (92.4)
1	8 (7.4)	2 (6.9)	6 (7.6)
Charlson comorbidity index score				0.938†
Mean ± SD	1.8 ± 2.3	1.9 ± 2.3	1.8 ± 2.3
Min-Max	0.0–9.0	0.0–7.0	0.0–9.0
Median (IQR)	1.0 (0.0–3.0)	1.0 (0.0–3.0)	1.0 (0.0–3.0)
Charlson comorbidity index score				0.965‡
0	46 (42.6)	12 (41.4)	34 (43.0)
1	17 (15.7)	5 (17.2)	12 (15.2)
>=2	45 (41.7)	12 (41.4)	33 (41.8)
PESI class				0.756‡
1	22 (20.4)	8 (27.6)	14 (17.7)
2	21 (19.4)	6 (20.7)	15 (19.0)
3	22 (20.4)	6 (20.7)	16 (20.3)
4	22 (20.4)	5 (17.2)	17 (21.5)
5	21 (19.4)	4 (13.8)	17 (21.5)
† Two group t-test
‡ Chi-square test
§ Fisher’s exact test

In comparing the four raters to the radiologist, all except one showed moderate agreement for the presence of RV dilatation (Table 3). The Pearson correlation coefficients for the measurements of each of the raters compared with the radiologist were 0.87, 0.80, 0.74, and 0.78 for raters 1–4, respectively. The Bland-Altman plots (Table 4 and Fig. 1–4) show that the mean differences in RV/LV ratio between the raters and the radiologist were relatively small for three of the four raters (-0.04, -0.05, 0.04, 0.24 for raters 1–4, respectively).

Table 3

Cohen’s kappa statistics of the experienced radiologist reviewer and the four students
Cohen kappa statistics	N of images	Kappa (95% CI)
Rater 1 vs Radiologist	105	0.50 (0.32–0.68)
Rater 2 vs Radiologist	57	0.46 (0.21–0.70)
Rater 3 vs Radiologist	99	0.49 (0.31–0.68)
Rater 4 vs Radiologist	104	0.29 (0.10–0.47)

Table 4

Difference between rater and radiologist RV and LV ratio measurements (in mm)
	Mean(std dev)	Median	Interquartile range (IQR)
Rater 1 vs Radiologist	-0.04 (0.19)	0.00	-0.10–0.10
Rater 2 vs Radiologist	-0.05 (0.26)	-0.10	-0.20–0.10
Rater 3 vs Radiologist	0.04 (0.29)	0.00	-0.10–0.10
Rater 4 vs Radiologist	0.24 (0.26)	0.20	0.10–0.40

In this retrospective cohort study, we found that after brief training, medical and premedical students could measure RV dilatation with moderate accuracy compared with an experienced radiologist. Our study is the first to evaluate the ability of learners at this level of training to interpret these findings on CT.

Studies of non-radiologist physicians and resident physicians on the evaluation of CT dilation on CTPA have shown near-perfect agreement between measurements performed by non-radiologists and an experienced radiologist. There are several likely reasons why our findings differ. First, our student raters were much earlier in their medical training than the senior resident physicians. Although the residents in the reference study were non-radiologists, basic interpretation of diagnostic imaging would have been part of their training and routine medical practice during residency.

Second, it is not possible to make a direct comparison between the training for RV dilatation in our study and that in the reference study. The residents in the reference study were noted to have undergone “simple instruction,” in addition to receiving written instructions by an experienced radiologist. The extent and content of this initial instruction is unclear. Our training was also limited. It is possible that had we expanded it, our students could have achieved a higher level of accuracy.

Finally, a much higher proportion of our patients (73%) had RV dilatation compared with those in the reference study (42%). Our sample was stratified to contain an equal number of patients in each PESI class, so our cohort contained more patients in the higher PESI classes than a random, non-stratified sample. When looking at a dichotomous outcome (RV dilatation), small differences in measurements can significantly change dichotomous results.

Prior studies evaluating the ability of medical students to interpret specific findings on diagnostic imaging after training have shown mixed results, with generally high accuracy for some ultrasound applications and poor accuracy for interpretation of CT.^11–13 In one study, nearly two-thirds of interns reported being frequently expected to make preliminary imaging study interpretations, but fewer than half felt confident in their ability to interpret basic CT images.¹⁴ There is consensus among residency program directors that adequate radiology skills should be a part of undergraduate medical education.¹⁵ Our study shows that a basic level of proficiency for certain applications of CT can easily be taught to students.

The findings of this study imply that although the identification of RV dilation on CTPA may be relatively accurate when performed by non-radiologists and resident physicians, the same level of accuracy may not extend to medical students. Medical and premedical student learners were able to achieve moderate accuracy with limited training. It is possible that with additional focused training, student accuracy could approach that of resident physicians.

Ethics approval and consent to participate

The Research Determination Committee for KPNC decided that the project did not meet the regulatory definition of research involving human subjects per 45 CFR 46.102(d), so consent for participation was not required. The study was conducted in accordance with the principles of the Declaration of Helsinki.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author upon request.

Competing interests

The authors have no competing interests as defined by BMC, or other interests that might be perceived to influence the results and/or discussion reported in this paper.

Funding

Funding for this study was provided by Kaiser Permanente.

Authors' contributions

ED contributed to study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical expertise, obtaining funding, administrative, technical, or material support, and study supervision. SF contributed to study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. DE contributed to study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. LF contributed to study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. JS contributed to study concept and design, acquisition of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. YH contributed to study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical expertise. JC contributed to study concept and design, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. BSR contributed to study concept and design, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content. DV contributed to study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical expertise, administrative, technical, or material support, and study supervision.

Acknowledgements

The authors would like to thank Juleon Rabbani of the Kaiser Biostatistical Consulting Unit for his help and advice in the early stages of this study.

Dudzinski, D. M., Hariharan, P., Parry, B. A., Chang, Y. & Kabrhel, C. Assessment of Right Ventricular Strain by Computed Tomography Versus Echocardiography in Acute Pulmonary Embolism. Acad Emerg Med 24, 337–343 (2017).
Karri, J., Truong, T., Hasapes, J., Medicine, D. T. of T. & 2020. Correlating computed tomography pulmonary angiography signs of right ventricular strain in pulmonary embolisms to clinical outcomes. ncbi.nlm.nih.gov 15, 64 (2020).
Duffett, L., Castellucci, L. A. & Forgie, M. A. Pulmonary embolism: update on management and controversies. BMJ 18, m2177-23 (2020).
Becattini, C. et al. Right ventricle assessment in patients with pulmonary embolism at low risk for death based on clinical models: an individual patient data meta-analysis. Eur Heart J 42, 3190–3199 (2021).
Cho, S. et al. Assessing the severity of pulmonary embolism among patients in the emergency department: Utility of RV/LV diameter ratio. Plos One 15, e0242340 (2020).
Ende-Verhaar, Y. M. et al. Accuracy and reproducibility of CT right-to-left ventricular diameter measurement in patients with acute pulmonary embolism. PLOS ONE 12, e0188862-9 (2017).
Vinson, D. R. et al. Outpatient Management of Emergency Department Patients With Acute Pulmonary Embolism: Variation, Patient Characteristics, and Outcomes. Annals of emergency medicine 72, 62-72.e3 (2018).
Gordon, N. & Lin, T. The Kaiser Permanente Northern California Adult Member Health Survey. Perm J 20, (2016).
Aujesky, D. et al. Derivation and Validation of a Prognostic Model for Pulmonary Embolism. Am J Resp Crit Care 172, 1041–1046 (2012).
Shan, J. et al. “Outpatient Management” of Pulmonary Embolism Defined in the Primary Literature: A Narrative Review. Perm J 25, 20.303 (2020).
Stokke, T. M. et al. Brief Group Training of Medical Students in Focused Cardiac Ultrasound May Improve Diagnostic Accuracy of Physical Examination. J Am Soc Echocardiog 27, 1238–1246 (2014).
Werth, B. et al. Assessing Medical Student’s Ability to Interpret Traumatic Injuries on Computed Tomography Before and After the Third Year Clerkships. Kans J Medicine 11, 91–94 (2018).
Nguyen, B. et al. Comparisons of Medical Student Knowledge Regarding Life-Threatening CT Images Before and After Clinical Experience. Kans J Medicine 10, 55–58 (2019).
Saha, A., Roland, R. A., Hartman, M. S. & Daffner, R. H. Radiology Medical Student Education An Outcome-based Survey of PGY-1 Residents. Acad Radiol 20, 284–289 (2013).
Kondo, K. L. & Swerdlow, M. Medical Student Radiology Curriculum What Skills Do Residency Program Directors Believe Are Essential for Medical Students to Attain? Acad Radiol 20, 263–271 (2013).

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Accuracy of medical student measurements of CT right-to-left ventricular diameter in patients with acute pulmonary embolism

Status:

Version 1

Abstract

Figures

Introduction

Methods

Study design and setting

Study aim

Selection of participants

Training

Image review

Statistical analysis

Results

Discussion

Conclusion

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Additional Declarations

Status:

Version 1