This study examined the concordance of comorbidity data extracted from administrative data, medical charts and self-reported. Our findings showed that agreement between comorbidity data collected by the Victorian Admitted Episodic Dataset, medical charts, and self-reports by men with CaP who have undergone a radical prostatectomy varied across the analysed comorbidities.
In terms of the comparison of all three data sources, a “slight”, less than “moderate” agreement was observed for the majority of comorbidities when calculated using the kappa. The raw data for this study was also analysed to gain a more holistic understanding of concordance. No studies have previously compared these three data sources, so it is not possible to discuss our findings in context with others. However, studies have assessed concordance between two of these three groups and these are discussed below.
Concordance between self-reports and other data sources
Cancer was one of the most interesting self-reported conditions not reported by the majority of patients with only 26% of respondents reporting the condition. A negative statistic was reported with the kappa when this condition was analysed between the self-reports and other datasets, highlighting “poor” agreement. Additionally, when comparing medical charts and self-reports, 74% (n=121) of patients had cancer in the medical charts, even though they did not self-report the condition themselves. A similar observation was noted with the administrative datasets, with 76% of patients compared with the self-reports failing to state that they had cancer even though this was recorded in the administrative dataset.
The finding that men did not report having cancer may be due to the framing of the SCQ question that asked: “Do you have cancer?”. Given that the selected cohort of patients had undergone surgery it may be that they believe having the prostate gland removed meant they no longer had cancer. However, while for many men this is the case, 27–53% of men will develop prostate-specific antigen (PSA) recurrence up to 10 years after radical prostatectomy and 16–35% of patients receive second-line treatment within five years of surgery.(17)
Other studies have noted that the framing of a question relating to conditions such as cancer can impact reporting. Katz et al analysed concordance of Charlson comorbidities between the medical chart and self-reports, and found that there was only “moderate” concordance (kappa of 0.45 (95% CI: 0.28 to 0.62)) between the medical chart and self-reports in regard to the presence of cancer.(18) The researchers had intentionally omitted asking patients whether their tumour (if present) had been treated within the last 5 years. More interestingly, the Spearman correlation (a statistical measure used to quantify the statistical dependence between the ranking of two variables) between the comorbidity questionnaire and the medical chart-based Charlson index increased from 0.63 (P = 0.0001) to 0.70 (P = 0.0001) after the tumour condition was excluded. This highlights how the explicit wording of questions relating to cancer can result in various levels of recall bias.
There was a three-fold difference in reporting of depression (6 vs 17 reports), a nine-fold difference in reporting of back pain (4 vs 38 reports) and a nearly two-fold difference in reporting of osteoarthritis between medical charts and self-reports. Studies have highlighted that these conditions, in particular depression, are under-diagnosed in hospitals, with physicians not actively investigating whether patients have the condition due to its non-acute nature.(19) This has been especially noted with back pain, a chronic but non-life threatening comorbidity.(20) Patients with osteoarthritis have been noted to view their condition as part of the ageing process (21), with 50% of patients with severe knee pain not reporting to their physician about it (22) and thus reducing their inclination to actively seek medical assistance for their condition. In a study of 2380 community-dwelling patients aged 55–85 years a comparison of medical charts and self-report data showed a kappa of 0.31 (95% CI: 0.27-0.35) for osteoarthritis, with 21.8% of patients stating that they were affected by this condition even though their medical chart did not support the claim.(23) Those with a mobility limitation were more likely to self-report their condition than those without this limitation (OR: 2.68, 95% CI 2.10-3.44).(23) Ultimately, this highlights that patient- and physician- specific views towards certain comorbidities can influence their tendency of being recorded in the medical charts or self-reports.
Concordance between administrative data versus other datasets
In this study, we found that there was strong concordance between the administrative comorbidity data and the other studied data sources based on the reporting statistics used. However, after a more nuanced and deeper investigation, agreement with regard to the reporting of comorbidities was modest, at best. With conditions such chronic pulmonary disease and rheumatic disease, cases were often recorded in the medical chart but not in the administrative datasets.
Nine of the eighteen comorbidities compared between the medical charts and administrative datasets had no cases identified in the administrative data source. Of these nine conditions, chronic pulmonary disease was recorded 26 times in the medical chart but was not coded in the administrative dataset. A similar observation was seen with rheumatic disease and cerebrovascular disease, where it was recorded in the medical chart of 9 patients but was not coded for in the administrative dataset. There may be a few reasons for this observed discrepancy. Several studies have identified that hospital coders prioritise the coding of symptomatic comorbidities over asymptomatic ones, due to the higher level of hospital funding associated with the former (24) or due to Australian guidelines dictating that “additional diagnoses can only be assigned if they affect patient care during admission”.(25, 26) The first point was somewhat proven in the case of chronic pulmonary disease in this study, as it is a condition that may only manifest depending on certain environmental and physiological stimuli.(27) The Australian guidelines surrounding coding comorbidities is interesting, given that in the Australian Modification of the ICD-10AM codes, coders are provided with ample space (fifty slots) to state any secondary diagnoses of a particular patient.(26) Other studies have highlighted how the level of experience of the hospital coder can impact the accuracy of the ICD-10AM codes.(28) Differences in coding practice between inexperienced and experienced coders have been shown to exist.(24, 29, 30, 31)
The “substantial” agreement reported by the kappa for diabetes may be attributed to the highly scrutinized nature of this condition in clinical settings.(32, 33) Clinical coders are required to document ICD codes for conditions which require health services resources.(24) The codes are used to assign a Diagnosis Related Group (DRG), which in turn translates to funding for the health service. For patients with Type I diabetes, blood sugar levels are usually required at least twice daily, and insulin must be administered, usually by nursing staff. Type II diabetes is a condition that is monitored strictly within hospitals.(34)
Strengths and limitations
This study has several strengths. This is the first time that comorbidities have been compared across medical charts, administrative data and patient self-reports. Given the increasing focus on the use of patient reported data, this study enhances our knowledge on the reliability and accuracy of such data. This is particularly important as self-reported comorbidity data is a cost-effective way of collecting comorbidity data compared to the extraction of data from medical charts or administrative datasets.(35)
While the SCQ survey has been validated, with good test-retest reliability being reported (36, 37), this is the first time that it has been examined in a prostate cancer population. This is despite the fact that it has been recommended by ICHOM as the preferred method for collecting comorbidities in men with localised prostate cancer (38). However, more research is required before we can use it to risk adjust health outcomes. While pre-operative administration of the SCQ will likely improve the likelihood that patients will self-report cancer, there remain other comorbidities such as heart disease, which were reported by patients but not documented in the other data sources.
This study has a number of limitations which will impact the interpretation of the findings. One relates to the potential discrepancy in interpretation of the definitions for each comorbidity. A list of ICD-10AM codes pertaining to certain conditions such as heart disease, lung disease and kidney disease were used to identify whether a patient was regarded to have the condition or not. These same criteria were not used uniformly in the other data sources. Indeed, the SCQ purposely diluted the complexity of the comorbidity labels to allow the conditions to be understood by patients “without any prior medical knowledge”.(14) This likely impacted concordance. For example, in this analysis, ischaemic heart disease (IHD) was not classified as “heart disease” as the ICD-10AM codes pertaining to the two Charlson comorbidities of heart disease (myocardial infarction and congestive heart failure) did not consider IHD. However, patients with the condition may have self-reported it as heart disease.
Non-response bias may have influenced our results, given that we only had a response rate of 55.3%. It is not possible to know whether the non-responders would differ systematically from responders in terms of their self-reported comorbidities. Another bias likely introduced into this study relates to recall bias as men may have had trouble recalling comorbidities.
The sample size for this study was relatively small (N=217), preventing sub-group analysis, such as whether there were differences based on type of hospital (public/private), where documentation practices may differ. Also, more nuanced findings were unable to be revealed, as shown in the difference in CCI distributions for the administrative datasets for n=112 (population whom had data across all three data sources) and n=201 (sample who had data across administrative dataset and medical charts). Incomplete datasets also prevented more nuanced investigations into the concordance of data.