Comparison of Electronic Medical Records with Self-report in Measuring Cancer Screening

doi:10.21203/rs.3.rs-1990852/v1

Download PDF

Research Article

Comparison of Electronic Medical Records with Self-report in Measuring Cancer Screening

https://doi.org/10.21203/rs.3.rs-1990852/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 25 Jul, 2023

Read the published version in BMC Medical Research Methodology →

You are reading this latest preprint version

Background: Efficient measurement of the receipt of cancer screening has been attempted with electronic health records (EHRs), but EHRs are commonly implemented within a single health care setting. However, health information exchange (HIE) includes EHR data from multiple health care systems and settings, thereby providing a more population-based measurement approach. In this study, we set out to understand the value of statewide HIE data in comparison to survey self-report (SR) to measure population-based cancer screening.

Methods: A statewide survey was conducted among residents in Indiana who had been seen at an ambulatory or inpatient clinical setting in the past year. Measured cancer screening tests included Colonoscopy, Fecal Immunochemical Test (FIT) for colorectal cancer, Human Papilloma Virus (HPV) and Pap tests for cervical cancer, and Mammogram for breast cancer. For each screening test, the self-reported response for receipt of the screening (yes/no) and ‘time since last screening’ were compared with the corresponding information from patient HIE to evaluate the association between the two measures.

Results: Percent agreement of HIE and self-report of screening receipt ranged from 59% to 81%, with the Gwet’s AC ranging from 0.24-0.76, indicating a fair to substantial concordance. For the time since receipt of last screening test, Percent agreement ranged from 71% to 93% along with the Gwet’s AC ranging from 0.47-0.91. In comparison with SR data, HIE data provided relatively more additional information about laboratory-based tests: FIT (15% HIE alone vs. 4% SR alone) and HPV tests (24% HIE alone vs. 12% SR alone) and less additional information about procedures: Colonoscopy (7% HIE alone vs. 34% SR alone), Pap test (10% HIE alone vs. 27% SR alone), or Mammography (8% HIE alone vs. 20% SR alone).

Conclusion: Studies that use a single data source should consider the type of cancer screening test to choose the optimal data collection method. HIE and self-report both provided unique information in measuring cancer screening, and the most robust measurement approach involves collecting screening information from both HIE and patient self-report.

early detection

cancer screening

survey

electronic medical records

Evidence-based screening tests are important for early detection of cancer to reduce the likelihood of a more advanced stage at diagnosis when cancer may be less treatable. Information about the receipt of cancer screening tests is widely available from patient data stored in their electronic health records (EHR), a systematic collection of patient health information in a digital format. Another alternative source of information about cancer screening is from patients’ self-report, an arguably more cost-effective source. EHRs may benefit both clinical and epidemiologic research, patient care, as well as performance measurement. However, structured EHR data inevitably contains incomplete or inaccurate information. Furthermore, EHRs typically are deployed in a single health care practice, hospital, or system. Health information exchange (HIE), which involves the electronic exchange of clinical and administrative information across a variety of health care organizations, provides a more population-based approach to the measurement of cancer screening as it contains EHR data from multiple health care settings or systems.^,

To our knowledge, no prior statewide studies of cancer screening has compared HIE data with patient self-report. Also, previous literature comparing self-report and EHR data showed varying levels of agreement based on factors such as type of clinical condition, type of clinical procedures performed, or data collection methods.^,^,^,^, Given the varying results of previous validation studies and the implications for population-based measurement in the use of HIE, we assessed the association and concordance between self-reported cancer screening by survey among Indiana residents and the individuals’ corresponding information from the statewide Indiana Health Information Exchange (IHIE). We focused on the cancer screening phase of the cancer care continuum for colorectal, cervical, and breast cancers. We hypothesized that the value of information gained from different data sources on cancer screening, might vary according to the types of screening tests.

From January 2018 through February 2018, Indiana residents who had been seen at least once in the previous year at Indiana University Health (IU Health) participated in a cross-sectional, mail-based survey known as the Hoosier Health Survey. A statewide integrated health system, IU Health operates 19 hospitals in Indiana and 178 clinics that provide outpatient care. The purpose of this survey was to better understand the cancer control needs of the community served by the Indiana University Cancer Center. Following HIPAA authorization from respondents to access their electronic health records, cancer screening information obtained through the survey were matched with the participant’s longitudinal EHR data by referencing each patient’s first and last name, birthdate, and place of residence. The electronic information was obtained from the Indiana Network for Patient Care (INPC), which is the clinical data repository for IHIE, a community-wide HIE operating in central Indiana with the support of the Regenstrief Institute. The INPC consists of clinical observations across five major hospital systems, public health departments (both state and county), and Indiana Medicaid.^, The study was approved by the IUPUI Institutional Review Board.

Study cohort

From a list of 284,062 people seen at least once in the past 12 months in the statewide health system and living in one of 34 Indiana counties with higher cancer mortality rates, a random, stratified sample of 8,000 adults was selected. In stratifying the sample, geographic location and race were equally weighted. The initial goal was to sample 2,000 individuals from each of four strata (rural White, rural Black, urban White, urban Black); due to the small number of participants in rural Black, the remaining 2,000 were taken from the rural White strata, resulting in 4,000 individuals from both rural and urban areas. 21 patients were excluded from the sample because their primary care providers declined to authorize their participation in the survey, thus resulting in 7,979 mailed surveys. Out of all mailed surveys, a total of 970 adults aged 18–75 years completed the survey, generating a 12% response rate. Younger adults were included in the sample so as to collect data on cervical cancer screening behavior; the upper age limit of the sample was set at 75 years because guidelines do not routinely recommend cancer screening after age 75. Out of these 970 respondents, a total of 711 individuals provided HIPAA authorization (73.3%), comprising our final study sample. The survey methodology has been described in more detail elsewhere. [11]

Populations eligible for cancer screening

The participants in our study were assessed on their cancer screening behavior for three types of cancer: colorectal, cervical, and breast. Screening guidelines from the U.S. Preventive Services Task Force (USPSTF) were used to determine the sample of survey respondents eligible for appropriate screening tests. For colorectal cancer, the eligible sample included men and women aged 50–75 years to receive a Colonoscopy every ten years or receive a fecal immunochemical test (FIT)/stool test every year. For cervical cancer, the eligible sample included women aged 21–29 years to receive a Pap test every three years and 30–65 years to receive a Pap test every three years or a Human Papilloma Virus (HPV) test every five years. In the case of breast cancer, the eligible sample was taken from women aged 50–75 years to receive a Mammography every two years.

Survey-based cancer screening measures

When eligible for screening, the respondents were asked whether or not they reported receiving one of the three cancer screening approaches with responses being “Yes” or “No” (binary in nature). For colorectal cancer, patients were asked if they ever received a Colonoscopy and whether they had one every ten years, as well as if they ever received a Fecal Immunochemical Test (FIT)/stool test, and whether they had one every year. For cervical cancer, patients were asked if they ever received a PAP test and if they had one every three years, or if they ever received a Human Papilloma Virus (HPV) test and if they had one every five years. For breast cancer, the patients were asked whether they ever received a Mammogram and if they had one every two years.

Respondents were also asked the time since their last screening test with responses being “Within the past year (less than 12 months ago)”, “More than 1 year ago, but less than 2 years ago”, “More than 2 years ago, but less than 3 years ago”, “More than 3 years ago, but less than 5 years ago”, or “5 or more years ago” (ordinal in nature). See Appendix 1 for detailed survey questions. Information on receipt of the screening and time since last screening were measured to assess the degree of association and concordance between survey self-report and HIE data.

STATISTICAL ANALYSIS

Descriptive statistics were performed of individual socio-demographic characteristics, including age, gender, race, educational level, marital status, insurance status, income, home ownership, employment status, rurality based on RUCA codes, and self-reported health status.

For the questions on receipt of cancer screening and time since last screening, as a first step, we conducted bivariate analysis on participants’ responses to the survey and the corresponding information in HIE data using Chi-square tests to check for any significant differences between the two information sources. As a second step, we evaluated the association and concordance between the two measures of screening information. For the questions on time since last screening, we only considered those participants whose HIE data as well self-report indicated receipt of screening.

To measure the strength and direction of association between the two information sources of screening data (survey self-report and EHR from HIE) we used the Spearman’s rank correlation coefficient (ρ). The value of ρ ranges from − 1 to + 1, indicating a perfectly negative to a perfectly positive association, with 0 indicating no association at all.^, The effect sizes were interpreted according to Cohen's (1988) guidelines.

Finally, we used two interrater reliability measures to measure the concordance of screening information obtained from HIE data and survey self-report:

- A) Percentage Agreement, one of the most common interrater reliability measures, is calculated as the number of agreement scores divided by the total number of scores.

- B) The Gwet's agreement coefficient (Gwet's AC), a measure of correlation, is defined as the conditional probability that two randomly chosen observational measurements will agree, assuming no agreement by chance. The agreement coefficients were calculated using Gwet's new chance-corrected inter-rater agreement coefficients weighted ordinally, extending all existing agreement coefficients to include multiple raters, multiple rating categories, any measurement level, and multiple ratings per subject.

We chose Gwet’s AC as a measure of concordance over an alternative measure, Cohen’s Kappa, as the latter suffers from various statistical issues.^,^,^,^, The Gwet's AC is interpreted according to Landis and Koch's guidelines.^, The analyses were performed in Stata (Stata 16.1, StataCorp LLC, College Station, TX).

[1] Cohen/Conger’s Kappa (κ), similar to correlation coefficients ranging from -1 to +1 with 0 representing the amount of agreement expected from random choice, and 1 indicating perfect agreement is an alternative method to measure the level of concordance between raters [21]. We choose Gwet’s AC over Cohen’s Kappa as Cohen’s Kappa suffers from some statistical issues; it assumes independence between the raters, hence frequently generating agreement due to chance which is not entirely correct. However, the Gwet’s AC does not depend upon this assumption. Additionally, the Cohen’s Kappa values suffer from the “Kappa paradox”; they tend to change quite a lot with a change in prevalence, i.e., the values become high and close to the percentage agreement when there is high prevalence (the prevalence problem). Also, the degree to which observers disagree has an impact on the Kappa values (the bias problem). Gwet’s AC tends to lessen the kappa limitations;[22,23] hence we consider Gwet’s AC providing more stable inter-rater reliability coefficients in our study following few recent studies that have also preferred Gwet’s AC over Cohen/Conger’s Kappa for being a more stable inter-rater reliability coefficient.[24,25,26]

Descriptive statistics

Of the 711 patients surveyed, the participants had a mean age of 56.3 years, were most often female (54%), white (78%), not partnered (63%), homeowners (71%), insured (94%), employed (41%), rural (54%), and reported good general health (41%) (Table 1).

Bivariate analysis

With regards to the receipt of screening, bivariate analysis showed statistically significant differences between the two data sources (survey self-report (SR) and EHR from IHIE) for all screening tests (p-value <0.01) (Table 2.a). The participants who reported positive receipt of cancer screening were also asked about their time since last screening. Bivariate analysis showed statistically significant differences between the two information sources for Colonoscopy, Pap test, and Mammogram (Table 3.a).

The proportion of patients for whom both the HIE and self-report data indicated receipt of screening showed the following pattern: Colonoscopy (220/505=43%), FIT test (11/504=2%), HPV test (26/161=16%), Pap test (84/185=45%), and Mammogram (157/255=62%). Comparing the proportion of patients whose HIE data indicated screening (but self-report did not) with the proportion of patients whose self-report indicated screening (but HIE did not), the following patterns emerged: Colonoscopy (7% HIE alone vs. 34% SR alone), FIT test (15% HIE alone vs. 4% SR alone), HPV test (24% HIE alone vs. 12% SR alone), Pap test (10% HIE alone vs. 27% SR alone), Mammography (8% HIE alone vs. 20% SR alone) (Table 2.a).

Association and Concordance (Receipt of cancer screening)

For receipt of cancer screening, Spearman’s rank correlation coefficient showed small positive associations^[²⁰^] between information from HIE data and self-report for all five types of cancer screening tests (ρ range: 0.12-0.27; p-value<0.01 except for FIT test where p-value<0.05).

With regards to the level of concordance of information on receipt of cancer screening between HIE data and survey self-report, percent agreement and Gwet’s AC showed the highest level of agreement for FIT test (Percent agreement: 80.95%, 95% CI: 0.77-0.84; Gwet’s AC: 0.76, 95% CI: 0.71-0.81) and the lowest level of agreement for Colonoscopy (Percent agreement: 58.81%, 95% CI: 0.54-0.63; Gwet’s AC: 0.24, 95% CI: 0.15-0.33).

To summarize, screening receipt information from HIE data and survey-self report showed overall agreement ranging from 59% to 81% according to the Percent Agreement measure, with a fair to substantial concordance ^[²³^] according to Gwet’s AC (0.24-0.76) (Table 2.b).

Association and Concordance (Time since last cancer screening)

For time since last screening, Spearman’s rank correlation coefficient showed small to moderate positive associations ^[²⁰^] among information from HIE data and self-reports for the screening tests (ρ range: 0.22-0.48; p-value<0.01 except for Mammogram where p-value<0.05). ρ could not be calculated for time since last FIT test due to absence of information in self-report (compared to information in HIE data) for the other "time" categories of the variable except for >1 year since the last test.

With regards to the level of concordance of information on time since last screening between HIE data and survey self-report, percent agreement and Gwet’s AC showed the highest level of agreement for Mammogram timing (Percent agreement: 93.50%, 95% CI: 0.91, 0.96; Gwet’s AC: 0.91, 95% CI: 0.87, 0.95) and the lowest level of agreement for FIT test timing (Percent agreement: 70.91%, 95% CI: 0.44, 0.97; Gwet’s AC: 0.47, 95% CI: -0.14, 1.00 (although p-value>0.10)).

To summarize, information on time since last screening from HIE data and survey self-report showed overall agreement from 71%-93% according to the Percent Agreement measure, with a moderate to almost perfect concordance^[²³^] according to Gwet’s AC (0.47-0.91) (Table 3.b).

In our study we focused on the cancer screening phase and evaluated the concordance between HIE data and self-reported responses of surveyed Indiana residents seen in a statewide healthcare system for receipt of screenings and the time since receipt of last screening test. For screening receipt, results indicated the highest level of agreement for FIT test and lowest level of agreement for Colonoscopy. For screening timing, the highest level of agreement between the two data sources was for Mammogram timing and the lowest level of agreement for FIT test timing. Additionally, HIE data provided relatively more information about FIT and HPV tests, which are both laboratory-based screening tests. Self-reported data provided more information about Colonoscopy, Pap test, and Mammography, all of which are medical screening procedures.

In the screening phase of the cancer care continuum, one of the earliest prior studies assessed the concordance between self-report and non-electronic medical record documentation among Kaiser Permanente Medical Care Program participants. Data were collected on the reason and timing for Pap tests, Mammograms, clinical breast exams, fecal occult blood tests (FIT tests), digital rectal examinations, and Sigmoidoscopies. Researchers found that self-reported response and non-electronic medical record documentation generally agreed more for procedures involving a test report (Mammogram, Pap test, fecal occult blood test, and Sigmoidoscopy) than a physician's note (clinical breast examination and digital rectal examination).^[6] The results of our study are similar to their findings, especially for Mammogram timing and receipt of FIT test where we found the most agreement between HIE data and self-report. A relatively recent study conducted among patients in 25 New Jersey Primary Care Practices who participated in the SCOPE program (supporting colorectal cancer outcomes through participatory engagement) indicated evidence of agreement for cancer screening ranging from 61% for Pap and PSA test to 83% for colorectal endoscopy. In this study, self-reports had a higher rate than non-electronic medical records.^[7] Both of these studies compared patients' self-reports against non-electronic medical records. Paper-based documentation may not provide accurate information on screening histories if information is entered incompletely; further, non-electronic medical records also suffer from disorganization, non-integration with other electronic systems, and lack of backups and security issues.^, Hence, our study has the advantage of using EHRs from HIE over non-electronic health records which is more consistent with current medical practice. Moreover, EHRs arguably provide higher quality data with more accessible, accurate, complete and up-to-date patient records with built-in privacy and security features.^,

In the treatment phase of the cancer care continuum, an academic hospital cancer registry study among breast cancer survivors from 2004–2009 evaluated concordance firstly between electronic query and manual review used to extract EHR data, and secondly between survivors' self-reports and the extracted EHR data on post-treatment Mammography. Electronic query identified more mammograms post-treatment than manual review, with high concordance between the two methods (0.90). Fewer days since Mammogram were associated with better concordance between self-reporting and EHR data. In conclusion, Tiro et al. encouraged the use of self-report as a screening tool among cancer survivors for surveillance care delivery.^[10] The advantage of our approach over prior EHR-based studies is the fact that they were studies performed within a single health care setting. On the other hand, EHRs in a state-based HIE, as explored in this study, use clinical data from among patient populations aggregated across multiple health care organizations. In addition to offering a complete, accurate, and holistic view of patient records, HIEs reduce duplication of information on procedures or tests, improving the usefulness of patient health records. HIE also improves the accessibility of medical data across multiple clinical settings, thereby improving the capacity to use population data for public health purposes and other quality improvement activities across consortia of health care organizations.^,^, Hence, using the Indiana Network of Patient Care (INPC), encompassing a community wide HIE, enables the measurement population-based cancer screening behavior at a population level with greater efficiency and completeness.

Limitations

Some study limitations must be considered when interpreting these results. First, our survey response rate was relatively low, at 12%, despite using established methods for survey research. Due to our expectation of a low response rate based upon current survey experience,^,^, enough surveys were delivered, with follow-up postcard reminders and a second copy of the survey to have meaningful population-based estimates. This was to ensure a relatively large absolute number of surveys among the target population. Other data collection methods, such as in-person interviews, might have improved our participation rate but would not have had the same reach as the mailed survey. Nonetheless, we received completed surveys from every surveyed county, and respondents and non-respondents did not differ significantly across available sociodemographic characteristics. Second, we selected only those residents who had been involved with a single health system in Indiana, although IU Health is Indiana's largest integrated health system serving approximately 1 million individuals, community wide, across 19 hospitals and 178 clinics. Nonetheless, the results of our study should be interpreted as a sample of statewide health system with at least some access to healthcare, as opposed to a population-based state sample. Overall, the access to healthcare by participants increased the likelihood of cancer screening receipt being captured in the EHR.

Different data sources yielded different information values about the receipt of cancer screening, depending on the type of cancer screening. The HIE data, for example, provided relatively more information about FIT and HPV tests, both laboratory tests, than about Colonoscopy, Pap tests, or Mammograms, all procedures. To choose the ideal data collection method, studies that use a single data source should consider the type of cancer screening test. Both HIE and self-reports provided unique information about cancer screening. The most robust measurement approach involves collecting both HIE and self-reported screening information. Moreover, the optimal data source may vary depending on the outcome of interest being measured, whether it is clinical decision making, performance measurement, or population surveillance. Future research opportunities include looking at concordance between self-report and EHR data over time, concentrating on vulnerable populations. Data should also be considered from different EHR systems such as single system EHRs, EHRs controlled by patients known as personal health records to draw comparisons with corresponding patient self-report regarding cancer screening.

EHR Electronic Health Record

FIT Fecal Immunochemical Test

Gwet’s AC Gwet’s Agreement Coefficient

HIE Health Information Exchange

HIPAA Health Insurance Portability and Accountability Act

HPV Human Papilloma Virus

IHIE Indiana Health Information Exchange

INPC Indiana Network for Patient Care

IU Indiana University

IUPUI Indiana University Purdue University

SCOPE Supporting Colorectal Cancer Outcomes Through Participatory Engagement

SR Self-report

USPSTF United States Preventive Services Task Force

Ethics approval and consent to participate: This study contains human subject. The survey was conducted in accordance with relevant guidelines and regulations. Informed consent was obtained from all subjects and/or their legal guardian(s). The Indiana University Purdue University Indianapolis (IUPUI) Institutional Review Board (IRB) approved the study.

Consent for publication: Not applicable.

Availability of data and materials: The datasets generated during and/or analyzed during the current study are not publicly available because public posting of database was not approved by the IUPUI IRB. However, datasets are available from the corresponding author on reasonable request.

Competing interests: The authors declare that they have no competing interests.

Funding: This work was aided by National Cancer Institute supplement to Indiana University Cancer Center Grant (P30 CA082709-17S6), Indiana University Melvin and Bren Simon Cancer Center, Indianapolis, Indiana.

Authors' contributions: O.B., D.H., S.R.: Conception and design; O.B., D.H., S.R., S.D.: Development of methodology; S.R., D.H.: Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.); O.B., D.H.: Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis); O.B., D.H., S.R., S.D.: Writing, review, and/or revision of the manuscript; O.B., D.H.: Study supervision.

Acknowledgements: Not applicable.

Centers for Disease Control and Prevention. Available at: https://www.cdc.gov/cancer/dcpc/prevention/screening.htm
Gunter, T. D., & Terry, N. P. (2005). The emergence of national electronic health record architectures in the United States and Australia: models, costs, and questions. Journal of medical Internet research, 7(1), e3.
Weiskopf, N. G., Cohen, A. M., Hannan, J., Jarmon, T., & Dorr, D. A. (2019). Towards augmenting structured EHR data: a comparison of manual chart review and patient self-report. In AMIA Annual Symposium Proceedings (Vol. 2019, p. 903). American Medical Informatics Association.
Dixon, B. E. (2016). What is health information exchange? In Health information exchange (pp. 3-20). Academic Press.
Dixon, B. E., Haggstrom, D. A., & Weiner, M. (2015). Implications for informatics given expanding access to care for Veterans and other populations. Journal of the American Medical Informatics Association, 22(4), 917-920.
Gordon, N. P., Hiatt, R. A., & Lampert, D. I. (1993). Concordance of self-reported data and medical record audit for six cancer screening procedures. JNCI: Journal of the National Cancer Institute, 85(7), 566-570.
Ferrante, J. M., Ohman-Strickland, P., Hahn, K. A., Hudson, S. V., Shaw, E. K., Crosson, J. C., & Crabtree, B. F. (2008). Self-report versus medical records for assessing cancer-preventive services delivery. Cancer Epidemiology and Prevention Biomarkers, 17(11), 2987-2994.
Gupta, V., Gu, K., Chen, Z., Lu, W., Shu, X. O., & Zheng, Y. (2011). Concordance of self-reported and medical chart information on cancer diagnosis and treatment. BMC medical research methodology, 11(1), 1-7.
Ho, P. J., Tan, C. S., Shawon, S. R., Eriksson, M., Lim, L. Y., Miao, H., ... & Li, J. (2019). Comparison of self-reported and register-based hospital medical data on comorbidities in women. Scientific reports, 9(1), 1-9.
Tiro, J. A., Sanders, J. M., Shay, L. A., Murphy, C. C., Hamann, H. A., Bartholomew, L. K., ... & Vernon, S. W. (2015). Validation of self-reported post-treatment mammography surveillance among breast cancer survivors by electronic medical record extraction method. Breast cancer research and treatment, 151(2), 427-434.
Haggstrom, D. A., Lee, J. L., Dickinson, S. L., Kianersi, S., Roberts, J. L., Teal, E., ... & Rawl, S. M. (2019). Rural and urban differences in the adoption of new health information and medical technologies. The Journal of Rural Health, 35(2), 144-154.
McDonald, C. J., Overhage, J. M., Barnes, M., Schadow, G., Blevins, L., Dexter, P. R., ... & INPC Management Committee. (2005). The Indiana network for patient care: a working local health information infrastructure. Health affairs, 24(5), 1214-1220.
Vreeman, D. J., Stark, M., Tomashefski, G. L., Phillips, D. R., & Dexter, P. R. (2008). Embracing change in a health information exchange. In AMIA Annual Symposium Proceedings (Vol. 2008, p. 768). American Medical Informatics Association.
Salzman, B., Beldowski, K., & de La Paz, A. (2016). Cancer screening in older patients. American family physician, 93(8), 659-667.
U.S. Preventive Services Task Force. Available at: https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/colorectal-cancer-screening
U.S. Preventive Services Task Force. Available at: https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/cervical-cancer-screening
U.S. Preventive Services Task Force. Available at: https://www.uspreventiveservicestaskforce.org/uspstf/recommendation/breast-cancer-screening
Kendall MG. Rank Correlation Methods. Oxford, England: Griffin; 1948.
Lange T, Kopkow C, Lützner J, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability. BMC Med Res Methodol. 2020;20(1):28. doi:10.1186/s12874-020-0912-8
Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. New York: Routledge; 1988. doi:10.4324/9780203771587
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276-282.
Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of clinical epidemiology, 43(6), 551-558.
Dettori, J. R., & Norvell, D. C. (2020). Kappa and beyond: is there agreement? Global Spine Journal, 10(4), 499-501.
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC medical research methodology, 13(1), 1-7.
Jimenez, A. M., & Zepeda, S. J. (2020). A Comparison of Gwet’s AC1 and kappa when calculating inter-rater reliability coefficients in a teacher evaluation context. Journal of Education Human Resources, 38(2), 290-300.
Cibulka, M. T., & Strube, M. J. (2021). The Conundrum of Kappa and why some Musculoskeletal Tests Appear Unreliable despite High Agreement: A Comparison of Cohen Kappa and Gwet’s AC to Assess Observer Agreement when Using Nominal and Ordinal Data. Physical Therapy.
Gwet KL. Handbook of Inter-Rater Reliability, 4th Edition: The Definitive Guide to Measuring The Extent of Agreement Among Raters. Advanced Analytics, LLC; 2014.
Klein D. Implementing a General Framework for Assessing Interrater Agreement in Stata. Stata J. 2018;18(4):871-901. doi:10.1177/1536867X1801800408
Hedges, L. (2020). The Pros and Cons of Paper Medical Records (According to Doctors Who Use Them). Retrieved from Software Advice: https://www.softwareadvice.com/resources/pros-cons-paper-charts/
Advantages & Disadvantages of Paper Medical Records. Retrieved from Truenorth: https://www.truenorthitg.com/pros-and-cons-paper-medical-records/
Editorial Team. (2020). The Pros and Cons of Electronic Medical Records (EMRs). Retrieved from Virtru: https://www.virtru.com/blog/electronic-medical-records-pros-cons/
Gallagher Healthcare. (2018). Retrieved from Gallagher Healthcare: https://www.gallaghermalpractice.com/blog/post/advantages-and-disadvantages-of-electronic-health-records
Menachemi, N., Rahurkar, S., Harle, C. A., & Vest, J. R. (2018). The benefits of health information exchange: an updated systematic review. Journal of the American Medical Informatics Association, 25(9), 1259-1265.
Department of Health Care Finance. https://dhcf.dc.gov/page/benefits-health-information-exchange
HealthIT.gov. https://www.healthit.gov/faq/what-are-benefits-health-information-exchange
HealthIT.gov. https://www.healthit.gov/topic/health-it-and-health-information-exchange-basics/hie-benefits
Czajka, J. L., & Beyler, A. (2016). Background paper declining response rates in federal surveys: Trends and implications. Mathematica policy research, 1, 1-86.
Boyle, J., Berman, L., Dayton, J., Iachan, R., Jans, M., & ZuWallack, R. (2021). Physical measures and biomarker collection in health surveys: Propensity to participate. Research in Social and Administrative Pharmacy, 17(5), 921-929.
Lallukka, T., Pietiläinen, O., Jäppinen, S., Laaksonen, M., Lahti, J., & Rahkonen, O. (2020). Factors associated with health survey response among young employees: a register-based study using online, mailed and telephone interview data collection methods. BMC Public Health, 20(1), 1-13.
Rawl SM, Dickinson S, Lee JL, et al. Racial and Socioeconomic Disparities in Cancer-Related Knowledge, Beliefs, and Behaviors in Indiana. Cancer Epidemiol Biomarkers Prev. 2019;28(3):462-470. doi:10.1158/1055-9965.EPI-18-0795
Lee, J. L., Rawl, S. M., Dickinson, S., Teal, E., Baker, L. B., Lyu, C., ... & Haggstrom, D. A. (2020). Communication about health information technology use between patients and providers. Journal of general internal medicine, 35(9), 2614-2620.
https://iuhealth.org/

Table 1: Summary statistics of participants’ sociodemographic variables

Patient Characteristics

Frequency(percentage)

TOTAL (N)

711(100.00)

AGE

Mean (SD)

Journal Publication

published 25 Jul, 2023

Read the published version in BMC Medical Research Methodology →

Editorial decision: Major revision
19 Dec, 2022
Reviewers agreed at journal
08 Dec, 2022
Reviews received at journal
30 Nov, 2022
Reviewers agreed at journal
21 Nov, 2022
Reviewers invited by journal
21 Nov, 2022
Editor assigned by journal
21 Nov, 2022
Editor invited by journal
20 Oct, 2022
Submission checks completed at journal
20 Oct, 2022
First submitted to journal
23 Aug, 2022

You are reading this latest preprint version

Screening Tests (Receipt)	Survey responses (SR)	EMR=No	EMR=Yes	P-value
COLORECTAL CANCER
Colonoscopy (n=505)	SR=No SR=Yes	77(30.68) 174(69.32)	34(13.39) 220(86.61)	<0.001***
FIT test (n=504)	SR=No SR=Yes	397(95.20) 20(4.80)	76(87.36) 11(12.64)	0.006***
CERVICAL CANCER
HPV test (n=161)	SR=No SR=Yes	77(79.38) 20(20.62)	38(59.38) 26(40.63)	0.006***
PAP test (n=185)	SR=No SR=Yes	33(39.76) 50(60.24)	18(17.65) 84(82.35)	0.001***
BREAST CANCER
Mammogram (n=255)	SR=No SR=Yes	27(35.06) 50(64.94)	21(11.80) 157(88.20)	<0.001***

	Spearman’s rank correlation		Measures of concordance
Screening Tests (Receipt)	Coefficient (95% CI)	Scale	A) Percentage Agreement (95% CI)	B) Gwet’s AC (95% CI)	Benchmark scale for reliability (Gwet’s AC)
COLORECTAL CANCER
Colonoscopy (n=505)	0.21*** (0.12,0.29)	Small	58.81*** (0.54,0.63)	0.24*** (0.15,0.33)	Fair
FIT test (n=504)	0.12** (0.02,0.23)	Small	80.95*** (0.77,0.84)	0.76*** (0.71,0.81)	Substantial
CERVICAL CANCER
HPV test (n=161)	0.22*** (0.05,0.38)	Small	63.98*** (0.56,0.71)	0.34*** (0.19,0.50)	Fair
PAP test (n=185)	0.25*** (0.10,0.39)	Small	63.24*** (0.56,0.70)	0.32*** (0.17,0.46)	Fair
BREAST CANCER
Mammogram (n=255)	0.27*** (0.15,0.40)	Small	72.16*** (0.67,0.78)	0.56*** (0.45,0.66)	Moderate

Screening Tests (Time since last screening)	Survey responses	EMR	P-value
COLORECTAL CANCER
Colonoscopy n=220 (SR & EMR) <1yr >1yr-<2yrs >2yrs-<3yrs >3yrs-<5yrs >5yrs-<10yrs >=10yrs	62(28.18) 38(17.27) 39(17.73) 38(17.27) 43(19.55) -	79(35.91) 30(13.64) 37(16.82) 28(12.73) 38(17.27) 8(3.64)	<0.001***
FIT test n=11 (SR & EMR) <1yr >1yr-<2yrs >2yrs-<3yrs >3yrs-<5yrs >5yrs-<10yrs >=10yrs	11(100.0) - - - - -	4(36.36) 3(27.27) 1(9.09) 1(9.09) - 2(18.18)	-
CERVICAL CANCER
HPV test n=26 (SR & EMR) <1yr >1yr-<2yrs >2yrs-<3yrs >3yrs-<5yrs >=5yrs	11(42.31) 9(34.62) 2(7.69) 4(15.38) -	12(46.15) 7(26.92) 2(7.69) 5(19.23) -	0.165
PAP test n=84 (SR & EMR) <1yr >1yr-<2yrs >2yrs-<3yrs >3yrs-<5yrs >=5yrs	40(47.62) 29(34.52) 15(17.86) - -	43(51.19) 14(16.67) 7(8.33) 11(13.10) 9(10.71)	<0.001***
BREAST CANCER
Mammogram n=157 (SR & EMR) <1yr >1yr-<2yrs >2yrs-<3yrs >3yrs-<5yrs >=5yrs	124(78.98) 33(21.02) - - -	133(84.71) 9(5.73) 7(4.46) 2(1.27) 6(3.82)	0.027**

	Spearman’s rank correlation		Measures of concordance
Screening Tests (Time since last screening)	Coefficient (95% CI)	Scale	A) Percentage Agreement (95% CI)	B) Gwet’s AC (95% CI)	Benchmark scale for reliability (Gwet’s AC)
COLORECTAL CANCER
Colonoscopy n=220 (SR & EMR)	0.43*** (0.30,0.56)	Moderate	87.00*** (0.84,0.89)	0.57*** (0.47,0.66)	Moderate
FIT test n=11 (SR & EMR)	-	-	70.91*** (0.44,0.97)	0.47 (-0.14,1.00)	Moderate
CERVICAL CANCER
HPV test n=26 (SR & EMR)	0.48*** (0.18,0.77)	Moderate	80.13*** (0.71,0.89)	0.48*** (0.18,0.78)	Moderate
PAP test n=84 (SR & EMR)	0.40*** (0.19,0.61)	Moderate	85.36*** (0.82,0.89)	0.63*** (0.50,0.76)	Substantial
BREAST CANCER
Mammogram n=157 (SR & EMR)	0.22** (0.04,0.39)	Small	93.50*** (0.91,0.96)	0.91*** (0.87,0.95)	Almost perfect

Comparison of Electronic Medical Records with Self-report in Measuring Cancer Screening

Status:

Journal Publication

Version 1

Abstract

Introduction

Methods

Study cohort

Populations eligible for cancer screening

Survey-based cancer screening measures

STATISTICAL ANALYSIS

Results

Descriptive statistics

Bivariate analysis

Association and Concordance (Receipt of cancer screening)

Association and Concordance (Time since last cancer screening)

Discussion

Limitations

Conclusion

Abbreviations

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1