Inter-observer agreement among specialists in the diagnosis of oral potentially malignant disorders and oral cancer using store-and-forward technology

Oral cancer is a leading cause of morbidity and mortality. Screening and mobile Health (mHealth)-based approach facilitates early detection remotely in a resource-limited settings. Recent advances in eHealth technology have enabled remote monitoring and triage to detect oral cancer in its early stages. Although studies have been conducted to evaluate the diagnostic efficacy of remote specialists, to our knowledge, no studies have been conducted to evaluate the consistency of remote specialists. The aim of this study was to evaluate interobserver agreement between specialists through telemedicine systems in real-world settings using store-and-forward technology. The two remote specialists independently diagnosed clinical images (n=822) from image archives. The onsite specialist diagnosed the same participants using conventional visual examination, which was tabulated. The diagnostic accuracy of two remote specialists was compared with that of the onsite specialist. Images that were confirmed histopathologically were compared with the onsite diagnoses and the two remote specialists. There was moderate agreement (k= 0.682) between two remote specialists and (k= 0.629) between the onsite specialist and two remote specialists in the diagnosis of oral lesions. The sensitivity and specificity of remote specialist 1 were 92.7% and 83.3%, respectively, and those of remote specialist 2 were 95.8% and 60%, respectively, each compared with histopathology. The diagnostic accuracy of the two remote specialists was optimal, suggesting that “store and forward” technology and telehealth can be an effective tool for triage and monitoring of patients. Telemedicine is a good tool for triage and enables faster patient care in real-world settings.


Introduction
Cancer of the lip and oral cavity is the 16 th most common cancer worldwide, with 377,713 new cases and 177,757 deaths annually.India is the country with the second highest number of oral cancer cases [1].India accounts for approximately 100,000 new cases, accounting for nearly a quarter of the total burden and making oral cancer a leading cause of death among men [2,3].
The stage of disease at the time of diagnosis is the most important determinant of the of patient outcome [4].The 5-year survival rate for localized cancers is 54.3-60.2%,while it is as low as 3.1-3.3% in advanced stages [5].Detection at an advanced stage lowers the chances of cure, decreases the quality of life, and imposes significant costs on the patient [3].In India, 70% of cases are reported at advanced stages (American Joint Committee on Cancer, Stage III-IV) [2] due to lack of access to a specialist in oral cancer, resulting in the five-year survival rate of 20% [2,5].Regrettably, early-stage cancers are often asymptomatic, decreasing the chances of patients seeking medical attention they need [3].
Oral cancer is preceded by a group of visible mucosal lesions called Oral Potentially Malignant Disorders (OPMDs), which have oral epithelial dysplasia [6].Early detection and surveillance of such OPMDs has the potential not only to reduce the incidence but also to improve the survival of those who develop oral cancer.Conventional screening provides a means for early detection and surveillance [3].However, disadvantages include inefficient data management, poor follow-up after screening, lack of knowledge and practice among dentists in detecting and diagnosing OPMDs, and delayed communication with the specialist [7].The constant challenge of effective communication between specialist and patient and the physical inaccessibility of health services increase the number of advanced-stage diagnoses and cancer mortality [8].
A communication strategy such as telemedicine, which includes real-time technology, store-and-forward technology, remote monitoring and mhealth approaches, is reliable, acceptable, accessible, reduces travel cost and unnecessary patient referrals, can serve as an effective triaging tool, and can improve the education of primary care physicians [4,8,9].Nowadays, telemedicine systems are openly accepted by health professionals in low-income countries [8].The telemedicine program for remote diagnosis of OPMDs and oral cancer screening has already been piloted in India.It was found that telemedicine-based oral cancer screening and surveillance is feasible in low-resource settings [4].
To date, most studies of telemedicine system for OPMDs and oral cancer have evaluated the agreement between remote (single diagnosis) and onsite diagnosis.Agreement and consistency between remote specialists has not been thoroughly investigated.The purpose of this study was to determine the interobserver agreement between the onsite, two remote specialists and histopathology and to evaluate the diagnostic accuracy of remote specialists in a realworld setting.The study used store-and-forward technology in which the clinical photographs of these lesions taken by onsite Frontline Healthcare Providers (FHPs) were digitally transmitted via a secure server for remote diagnosis and recommendations.

Methodology
This prospective, double-blinded study was conducted to evaluate the diagnostic efficacy of two remote oral medicine specialists in diagnosing OPMDs and oral cancer and to determine interobserver agreement between the two remote oral medicine specialists in store and forward technology and telemedicine.The two remote oral medicine specialists were trained in remote diagnosis.The Institutional Review Board approved the study (KIDS/IEC/Nov-2018/18).The onsite specialist performed the conventional visual examination of the participants and made a specific diagnosis.Lesions were biopsied based on the onsite specialist's recommendation.FHPs captured intraoral photographs using a smartphonebased mHealth application.A convenient sampling (n=822) clinical images were accrued from the image archive, which included both onsite diagnosis and remote diagnosis by the specialists based on our earlier study [10].Remote diagnosis was performed based on the image data without considering information on associated risk factors such as age, sex, habits, and underlying systemic diseases/medications.The remote specialists were blinded and independently made a diagnosis for each image in the following four categories - The diagnostic accuracy of the remote specialists was compared with that of the onsite specialist, whom we considered the reference standard.

Fig. 1 Photograph of benign exophytic growth on left buccal mucosa
A total of (n=102) biopsies were performed on the recommendation of the onsite specialist, with histopathology as the gold standard.The diagnoses of the onsite specialist, and the two remote specialists were compared with histopathology.Diagnostic accuracy in terms of sensitivity, specificity, positive predictive values, and negative predictive values was determined, and interobserver agreement was estimated using Cohen's kappa.Cohen suggested interpreting the Kappa result as follows: values ≤ 0 as indicating no agreement, 0.01-0.20 as none to slight, 0.21-0.40fair, 0.41-0.60moderate, 0.61-0.80substantial, and 0.81-1.00almost perfect agreement [11].

Results
A total of (n=822) images were included in the study to assess interobserver agreement between the two remote specialists.Based on the onsite specialist's diagnosis, which is considered the reference standard, category 1 included (n=228), category 2 (n=147), category 3 (n=326), and category 4 (n=121) images were considered.

Agreement between the two remote specialists and the onsite specialist
Table 1 shows substantial agreement (k= 0.682) between the two remote specialists in diagnosing clinical images.Moderate agreement (k= 0.603) was found between the   2), with sensitivity and specificity of 83.1% and 90.6%, respectively.The sensitivity and specificity between the onsite specialist and remote specialist 2 were 79% and 82.6%, respectively with moderate agreement (k= 0.605) in the diagnosis of oral lesions, as shown in Table 3. Table 4 shows the concordance between the onsite specialist and the two remote specialists in the diagnosis of oral lesions with moderate agreement (k= 0.629).Diagnostic accuracy and concordance between the onsite specialist, remote Specialist 1, remote Specialist 2 are shown in Table 5.

Agreement between the histopathology, two remote specialists, and onsite specialist
A total of (n=102) biopsies were taken to compare the efficacy of the onsite specialist, remote specialist 1, and remote specialist 2 with histopathology (the gold standard).Table 6 shows the diagnostic accuracy and agreement between histopathology, onsite specialist, remote specialist 1, and 2. Onsite specialist showed a sensitivity of 94.8% and a specificity of 83.3% for diagnosis of OPMDs and oral cancer with histopathology as the gold standard.Remote specialist 1 showed a sensitivity and specificity of 92.7% and 83.3%, respectively; in comparison, remote specialist 2 showed a 95.8% sensitivity but a lower specificity of 60%.

Discussion
Sustainable technology-enabled healthcare practice is essential.One such concept of an ideal technology is telemedicine.Not long ago, the global pandemic crisis highlighted the need for telecommunication technology for physicianpatient communication strategies, which in turn led to the need to evaluate existing facilities and the reliability of telemedicine in reducing cancer-specific morbidity and mortality.The main goal of telehealth technology is to provide access to healthcare in rural areas where the need for early diagnosis of OPMDs and oral cancer is high but access to healthcare is limited.
The mHealth-based remote oral cancer surveillance program was adopted to aid in remote early detection of oral cancer by primary care dental practitioners in a resource-constrained setting has been presented previously [5].Store and forward technology could eliminate the need for an onsite consultation in 50% of cases.In two isolated studies [5,9], the agreement between the onsite specialist and the remote specialist in detecting oral lesions was 97% and 92.7%, respectively.These studies proved that the telediagnosis and telemedicine module is suitable for low resource settings.In another study [10], the remote specialists showed a sensitivity and specificity of 95% and 84%, respectively, in diagnosing OPMD and oral cancer compared to onsite specialist.To date, no study has evaluated the agreement between the remote specialists in diagnosing oral lesions, and this is the first such study.
In our study, we evaluated diagnostic accuracy and interobserver agreement between two remote specialists in a storeand-forward technology and telemedicine module.Remote diagnosis was performed for clinical photographs taken by low-skilled FHWs in low-income and low-infrastructure settings.The FHWs recorded patients' demographic data, medical history, habits, and captured clinical photographs of oral lesions.In the regular telemedicine module, the specialist remotely makes the diagnosis based on structured data (patient demographics, including de-identified data, and habit history) and unstructured data (clinical images).Currently, the application of artificial intelligence models is widely used in image-based diagnosis.These models are mainly developed using unstructured data.To simulate this process, in this study remote diagnosis was performed based on lesion morphology (unstructured data) without patient clinical data or risk factors to maintain uniformity of data input A study comparing the diagnostic accuracy of remote and onsite specialist using a novel, low-cost telemedicine platform consisting of a smartphone-based remote intraoral camera and a custom software application found that on-site diagnosis had a high sensitivity (94%) and a specificity of 69.2% compared with histopathologic diagnosis, which was not significantly different from the accuracy of the remote specialist (sensitivity: 94%; specificity: 62.5%) [12].Our study showed a specificity of 83.3% for remote specialist 1 but a specificity of 60% for remote specialist 2, which may be attributed to the remote specialist 1 having more years of experience with remote diagnoses than the remote specialist 2.
The variability of diagnosis made by remote specialists may be due to misdiagnosis of the obscured lesions, inability to perform a physical examination, or overdiagnosis.Sometimes the predefined clinical data and images of the lesions collected at the remote site are not sufficient for remote specialist to make a diagnosis.In such a scenario, the diagnosis may be based on perception and intuition rather than analytical reasoning, a human cognitive factor that may increase the incidence of misdiagnoses [13].A lesion with typical clinical features such as oral leukoplakia or oral lichen planus is easier to recognize on an image than a case of oral submucous fibrosis and other malignant lesions (such as salivary gland tumors), which require a thorough physical examination by the onsite specialist.The low specificity of the remote diagnosis compared with histologic confirmation may be due to overdiagnosis of the lesions.The remote specialists were calibrated to over diagnose in regular workflow, and the remote specialist diagnosis was tentative.Without the usual inspection and palpation of the lesion, the remote diagnosis relied on the clinical data and morphology of the lesions.It has been shown that early borderline lesions can be classified as malignant to avoid the consequences of the misdiagnosis of more aggressive cancers [14].Although not all misdiagnoses cause harm, the malignant transformation of OPMDs and the aggressiveness of OSCCs are highly variable and unpredictable in reality, and the relative contribution of overdiagnosis bias in different populations remains to be elucidated [15].
Assessing and addressing mHealth and store and forward telemedicine concerns is a critical step toward fully integrating telemedicine into clinical practice and outreach programs.All of this work is being done in collaboration with community healthcare workers and general dentists.Patients, the specialist, and the general dentist do not need to be available at the same time, so this model is convenient and improves efficiency, reduces patient travel and waiting time, allows for quick second opinions, and rapid retrieval of specialist reports-these are some of the benefits of store and forward technology.Limitations include good internet connectivity with high download and upload speeds at the remote site and overdiagnosis by the specialist [16].Remote specialist diagnosis depends on the quality of the photographs taken and the data collected from the remote site.The oral cavity is more difficult to diagnose in a teleconsultation because it is more difficult to photograph.FHPs should be trained to identify the lesion and ensure that they focus the lesion properly to obtain sharper images for diagnosis [9].If the quality of the image is nondiagnostic, smartphones with AI-driven applications may be deployed to alert FHWs to retake the photo.In this study, the FHWs were prompted by the remote specialists to retake the photo.Store and forward technology is an asynchronous service model based on sharing data and information outside of realtime consultations.

Conclusion
Store and forward technology can be an effective tool for patient triaging and monitoring, and can strengthen the healthcare system in low-and middle-income countries.
Specialist training is recommended for remote diagnosis to improve its efficiency.Despite all its limitations, telemedicine allows specialized clinicians to treat a larger number of cases than geographic distance would allow.It also improves the process by avoiding delays in diagnosing oral cancer.Image repository also serves as an important tool for documenting visual changes over time.

Fig. 2 Fig. 3
Fig. 2 Photograph of Speckled leukoplakia (mixed red and white lesion) on right buccal mucosa

Table 1
Agreement between the two remote specialists

Table 2
Agreement between the onsite specialist and remote specialist 1

Table 3
Agreement between the onsite specialist and remote specialist 2

Table 4
Agreement between the onsite specialist and two remote specialists

Table 5
Diagnostic accuracy and agreement between onsite specialist and two remote specialists