The efficiency, inter-rater reliability and validity of a mobile application to aid periodontal classification.

doi:10.21203/rs.3.rs-2617883/v1

Download PDF

Article

The efficiency, inter-rater reliability and validity of a mobile application to aid periodontal classification.

https://doi.org/10.21203/rs.3.rs-2617883/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Introduction

Periodontitis is the sixth most common disease worldwide, with almost half of the UK adult population having a degree of irreversible periodontitis. Forming an accurate diagnosis is paramount in making decisions regarding treatment planning. The aim of this study was to assess the inter-rater reliability, validity and efficiency of a smartphone application to aid in forming a periodontitis diagnosis compared with conventional methods.

Methods

Twenty-five Dental Core Trainees (DCTs) were randomly allocated into two groups. All dentists were asked to diagnose 10 periodontal cases and time taken to complete was measured. In addition, participants recorded their confidence in forming a periodontal diagnosis. The experimental group undertook the exercise with the aid of a smartphone application to aid in forming a diagnosis. The control group used a diagnostic flow diagram published by the British Society of Periodontology.

Results

The mean of correct responses was 90% in the app group and 85% in the control group. The time to complete was 15.5 minutes in the app group and 20 minutes in the control group. These findings were statistically significant. Inter-rater reliability for both groups was comparably high.

Conclusion

The use of the mobile application increased the accuracy and efficiency in forming a periodontal diagnosis compared to traditional diagnostic aids.

Health sciences/Health care/Dentistry/Periodontics

Health sciences/Health care/Dentistry/Dental education

Mobile application

periodontal disease

periodontitis

education

Periodontal disease requires the establishment of an accurate diagnosis before considering the viability and appropriateness of invasive or non-invasive treatments. The ability to make definitive decisions based on objective, empirical data is a cornerstone of clinical practice, however, this may not always be a consistently reliable process due to multiple patient, operator and environmental factors. The classification of periodontal disease has changed continually, largely driven by changes in definitions rather than disease pathology. As we move toward a more minimally invasive era, these definitions can be the defining factor of when to treat and when to monitor. Conflicting views in treatment planning can arise because of individual variations in patient anatomy, physiology and disease process, but also because of the subjectivity of the operator’s decisions based on previous experience, education and the interpretation of clinical data. The need for objectivity and consistency in the diagnostic process is important to ensure that evidence-based treatments are provided to the correct patient, with the correct condition at the correct time.

Periodontitis is the sixth most common disease worldwide [1], with almost half of the UK adult population having a degree of irreversible periodontitis. [2] Conventional tools for the diagnosis of periodontal disease include probing pocket depths, bleeding on probing and radiographic interpretation of the levels of horizontal or vertical bone. How we use and interpret these measures has varied over time to offer different diagnoses. Diagnostic aids have been created mainly geared towards accuracy, simplicity and a better understanding of the disease process for both the clinician and the patient. In 2017, the British Society of Periodontology (BSP) implemented new classifications for periodontal disease. This classification was based on conventional measures (bleeding on probing, level of horizontal bone loss and probing pocket depths) but with a focus on understanding the stability of a patient’s disease and identification of risk factors [3]. The goal is to ensure that stability is established before more complex treatment is undertaken on possibly unsuitable patients and conversely, ensure that stable patients are not undergoing erroneous treatment.

These diagnostic criteria were published into a flowchart by the BSP for the ease of the clinician. However, this may not be the definitive solution to the issue of subjective diagnoses. It has been found that assessment of dental student responses to historic periodontal cases showed around a quarter (25.2–27.6%) of participants answered correctly when trying to use this classification system while using the 2017 classification schematic. [4] The prescriptive nature of a periodontal classification system is more well suited to a digital tool. With the almost ubiquitous use of smartphones, a digital periodontitis diagnostic aid for general dental practitioners could present an array of benefits. In 2021, the Periobrain app was developed by SHOme Digital Ltd (Fig. 1) to improve classification of periodontal disease utilising the British Society of Periodontology interpretation of European S3 Treatment Guideline document. This app is available from the Apple App Store and Android Google Playstore. At the time of writing this research, the Periobrain application has been downloaded by 2261 users and used 14,542 times (Data provided by SHOme Digital Ltd.

The aim of this study is to assess the inter-rater reliability, validity and efficiency of a smartphone application to aid in forming a periodontitis diagnosis compared with conventional methods.

Ethical approval was obtained from King’s College London [Registration number MRSU-21/22-33666]. This study was conducted as a pilot study with a randomised controlled design.

Participant recruitment

Dental Core Trainees (DCT) are qualified dentists within 4 years of graduation. DCTs were invited to take part in this study over a one-month period. A decision was made to continue the study until a minimum of 10 DCTs per group was reached. At this stage, the study was concluded. Participation was voluntary, without penalty if refused. No identifiable information was held on participants other than their training grade (DCT1, DCT2, DCT3) and their gender. A written information sheet was provided to all subjects.

Design of the study and procedure

Ten orthopantomograms (OPG) of adult patients with varying degrees of periodontal disease were identified from a local oral surgery clinic. All patient data were kept anonymous other than age. Each OPG was then utilised to produce a six-point pocket chart (6PPC) that correlated to the bone levels illustrated by the radiograph, noting pocket depths and bleeding on probing. The 10 cases were then assessed by 3 experienced senior clinicians and a diagnosis was concluded. Any conflict between diagnoses was resolved by discussion. Each OPG with the corresponding 6PCCC and BOP were recorded on an A4 participant worksheet (Figure 2).

The worksheets were placed in a brown envelope in combination with a preliminary questionnaire intending to capture the subject’s gender, year of DCT and confidence in the diagnosis of periodontal disease. Confidence was self-reported using a 10cm visual analogue scale labelled from ‘not confident’ to ‘confident’.[5]

Envelopes also contained instructions on what diagnostic aids were available to complete the worksheet – either using the mobile application (experimental group) or using the provided flowchart (control group).

Following recruitment, participants were allocated to either an experimental group or a control group through simple randomisation (flipping a coin). The subject was then instructed to complete the preliminary questionnaire followed by providing a classification for the 10 periodontal cases. The time taken to complete the periodontal cases was measured in seconds.

All results were recorded on an Excel worksheet and subjected to further analysis.

Interpretation of results and data analysis

Statistical analysis was carried out by a single author (**) who was blinded to the groups. Descriptive statistics were used to describe the findings from the preliminary questionnaire.

Periodontal diagnoses were assessed for accuracy to the correct diagnosis. Results within the experimental group (app group) were assessed for inter-rater reliability using Fleiss’ Kappa.

Comparisons between the groups for the time-taken to complete was analysed for significance by calculating a provisional P- value using Z-scores.

Statistical analysis of collected data was done with construction of a Two Tailed Mann-Whitney U Test within an Excel model.

25 subjects were recruited for the study from August to September 2022. Of the participants who took part, 14 males (56%) and 11 females (44%) completed the study. 21 were DCT1 (68%), 2 DCT2 (16%) and 2 DCT3 (16%).

The mean confidence VAS scores for all subjects were found to be 68.16% (SD 17.1).

Due to the nature of randomisation, 14 were allocated to the experimental group and 11 to the control group. The groups were considered well matched for gender, DCT level and VAS scores; 70.2% (SD 16.87) for experimental and 65.9% (SD 17.81) for control.

Accuracy

The arithmetic mean of correct responses was found to be 71.6% (SD 1.749) in the control group and 87.6% (SD 1.535) in the app group. Results of the Mann- Whitney U Test showed a statistically significant difference between the two groups (z-score = 2.63805, p-value = 0.0083).

Inter-rater reliability

Following the calculation of values of agreement between correct and incorrect answers between the experimental and control groups, Fleiss’ Kappa for each group was calculated for inter-rater reliability. This was calculated as 1.000087 (perfect agreement) for the control group and 0.999734 (perfect agreement) for the experimental group. This indicated a high degree of inter-rater reliability in both groups.

Efficiency

The mean time to complete was 948.8 seconds in the app group and 1273.3 seconds for the control. A statistically significant difference was detected between the two groups (P = 0.040) (Table 1).

Table 1

Mean time to completion measured in seconds.
App group	Control group	P-value	Significance
948.8	1273.3	0.040	SIGNIFICANT

This study has demonstrated that a mobile application designed to aid in providing a periodontal diagnosis is more accurate and more efficient than a conventional method of formulating a diagnosis. A high level of inter-examiner reliability was also found, comparable to the conventional method.

Previous literature on implementing the 2017 Periodontal Classification found that a lack of simplicity was a barrier to adoption. Many dentists reported that they reverted back to the 1999 classification, implying limiting day-to-day application of the classification system. [6] Additionally, it has been shown that the simplification of conventional flowcharts can allow for improved clinical diagnostic accuracy. [7] The findings from this study reinforce the concept that simplicity and ease of use improves the clinician's ability to determine a correct diagnosis.

The inclusion of mobile applications in the arsenal of clinical tools can provide dentists with an easy and efficient means of gaining accurate information. Additional benefits include portability and accessibility, as well as applications being a more environmentally conscious method of distributing information, as they avoid the need to print and post paper formats. In addition, mobile applications are easily editable, negating the need to redistribute individual flowcharts, and lowering transport and production costs. The digital medium also allows for a more accessible user interface, overcoming certain additional needs such as decreased visual acuity, and the need to zoom or manage the screen colour for those with dyslexia. Finally, other formats, such as flowcharts or posters, are often stored openly in the surgery with large exposure to aerosols. Though this could be the case for mobile devices, recent advances in waterproofing have allowed for easier smartphone and tablet sanitisation.

Limitations

Best practices were followed when conducting this study, however, we, the authors, are aware of a number of limitations. The study was carried out in a single environment, primarily with dentists at DCT1 level. Therefore, results cannot be generalised to the entire dental population. Due to the recency in graduation, none of the participants would have been likely to have had extensive experience in formulating a periodontal diagnosis compared with the general population.

The decision was made to use this as a pilot study using the idea of “the rule of 12” [8] for the estimation of average values and variability. The authors have a future goal to plan a larger study using assessment criteria and metrics that have been previously applied. Future studies will allow for a better understanding of using digital tools and should incorporate a qualitative element to validate the app in a more realistic environment.

This study has demonstrated that mobile applications can improve accuracy and efficiency when diagnosing periodontal disease. The reliability of these tools is comparable to conventional methods. This pilot study has demonstrated the need for consideration of evolving technologies within dentistry as they could lead to benefits for both the clinician and patient.

Data availability statement:

The data that supports the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Kassebaum NJ, Bernabé E, Dahiya M, Bhandari B, Murray CJL, Marcenes W. Global burden of severe periodontitis in 1990-2010: a systematic review and meta-regression. Journal of dental research. 2014;93(11):1045-1053. doi:10.1177/0022034514552491.
UKHSA. Delivering Better Oral Health: An Evidence-Based Toolkit for Prevention- Chapter 5: Periodontal Diseases. UK Health Security Agency. 2021.
British Society of Periodontology. The Good Practitioner’s Guide to Periodontology. 2017; p 23-26.
Gandhi, K. Katwal, D., Chang, J., Blanchard, S., Shin D., Maupome, G, Eckert, G. and John, V. Diagnosis and treatment planning using the 2017 classification of periodontal diseases among three dental schools. J. Dent. Ed. 2022.
Turner, N.M., Van De Leemput, A.J., Draaisma, J.M., Oosterveld, P. and Ten Cate, O.T.J., Validity of the visual analogue scale as an instrument to measure self‐efficacy in resuscitation skills. Medical education, 2008. 42(5), pp.503-511.
Smitha, K., Maria Thomas, J., Shankareswari, T., Affsha, S. and Anvitha, D. Critical appraisal of the classification of periodontal and peri‐implant diseases and conditions 2017. Oral Diseases, 2017. 26(5), pp.1090-1091.
Pakdeesettakul, S., Charatkulangkun, O., Lertpimonchai, A., Wang, H. and Sutthiboonyapan, P. Simple flowcharts for periodontal diagnosis based on the 2018 new periodontal classification increased accuracy and clinician confidence in making a periodontal diagnosis: a randomized crossover trial. Clinical Oral Investigations, 2022.
Julious, S. Sample size of 12 per group rule of thumb for a pilot study. Pharmaceutical Statistics, 2005. 4(4), pp.287-291.

There is a conflict of interest Mr Julian Woolley: This author could be argued to have a conflict of interest due to being the creator of the PERIOBRAIN application. Therefore they hold a financial and non-financial incentive in regards to the contents of this manuscript. However as this author was not involved in the collection of data, data analysis or any input into our results or conducted methodology. Our aim to remain impartial has been maintained as much as possible.

Download PDF

Version 1

posted

You are reading this latest preprint version

The efficiency, inter-rater reliability and validity of a mobile application to aid periodontal classification.

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods

Results

Accuracy

Inter-rater reliability

Efficiency

Discussion

Limitations

Conclusion

Declarations

References

Additional Declarations

Status:

Version 1