Evaluation of a structured e-learning-based approach to CT anatomy of the paranasal sinuses for medical students

Background: Computed tomography (CT) anatomy is not an integral part of undergraduate medical training in many countries. Radiology seems to be well suited for new online based teaching methods. The aim of the study was to evaluate whether e-learning is appropriate to introduce complex learning contents such as sinus CT anatomy in novices and to assess the usefulness of a sinus CT checklist to identify relevant anatomical variants in medical students. Material: Medical students were asked to assess sinus CT scans for anatomical variants before and after the implementation of the CLOSE mnemonic. Sinus CT anatomy and the CLOSE mnemonic were introduced by e-learning. The rate of correctly identi�ed variants and the results of the individual CLOSE items were recorded and compared with those of otolaryngology residents. A questionnaire was distributed for subjective evaluation of the usefulness of the checklist and e-learning. Results: Ten students took part in the study. The rate of correctly identi�ed variants improved signi�cantly from 33.3% to 61.1%. The analysis of the individual CLOSE items showed a signi�cant improvement for C, S and E in students and C, L and S in otolaryngology residents. The students identi�ed more anatomical variants compared to the residents. The subjective evaluation of the CLOSE mnemonic and e-learning was very positive. Conclusion: E-learning was able to transfer complex learning contents in previously non-trained medical students and was evaluated as an appropriate introduction to the topics. The structured assessment of paranasal sinus CT scans using the CLOSE criteria can signi�cantly improve the recognition of anatomical variants.

paranasal sinus CT scans using the CLOSE criteria can signi cantly improve the recognition of anatomical variants.

Background:
One of the most common diseases of the paranasal sinuses is chronic rhinosinusitis (CRS).First line treatment for CRS is medical, but in refractory cases, surgery may be indicated.In endoscopic sinus surgery (ESS), major (e.g.skull base injuries with CSF leak, intracranial and orbital complications and extensive bleeding) and minor complications (e.g.adhesions, hyposmia, infections and mild bleeding) are rare, but can be devastating.The extent of the disease, of surgery and nasal polyps, anatomical variants, previous surgery with missing landmarks, increased bleeding and concomitant diseases are risk factors for major complications [1][2][3].To minimize the risk of complications, detailed knowledge of the individual anatomy is crucial [4].In order to assess the individual anatomy preoperatively, computed tomography (CT) is currently the preferred imaging technique [5].
Formal teaching of radiologic anatomy in general and sinus anatomy in particular in medical school is insu cient in many countries [6][7][8].Interpretation of medical images is usually introduced to medical students during their clinical years or even during residency.Radiology seems to be well suited for new online teaching methods.In times of the corona pandemic with closed Universities and suspended classroom teaching, online teaching formats become indispensable.Not every University and medical school has been prepared for this scenario.E-Learning has been widely used in medical education.It provides maximum independence through exible organization of time and place.Learning contents can be repeated as often as necessary.Studies have shown that e-learning could be equally effective as traditional teaching methods [9][10][11].It has also been shown, that radiological anatomy and image interpretation can successfully be introduced by e-learning.These studies mainly examined teaching of plain radiographic image interpretation.This is different to the interpretation of CT scans as they are only two dimensional, whereas in CT images a third dimension is added by scrolling through the images in different planes.
Checklists, as described in aviation, have become widely accepted in medicine.They are being used throughout medicine and are supported by the World Health Organization (WHO).They represent an effective and systematic approach to patient safety in surgery [12,13].
Checklists are used successfully in medical education, mainly in performance-based examinations such as the OSCE (Objective Structured Clinical Examination) [14,15].
There are checklists speci cally designed for sinus surgery [16][17][18].Most of them contain general and speci c safety aspects.In 1997 Simmen and Schuknecht de ned 8 important anatomical structures and variants -potential danger points during surgery -as the core component of a preoperative CT checklist [19].This checklist is very detailed.The CLOSE mnemonic for the evaluation of sinus CT scans was rst proposed by Weitzel et al [20].It is simple and easy to reproduce.It has been shown that the identi cation of both anatomy and anatomical variants in CT images can be improved by these criteria [21][22][23].
Cribriform plate = Keros classi cation, asymmetry, bony dehiscence of the skull base Lamina papyracea: dehiscence, prolaps of orbital content, infraorbital cell (Haller cell), uncinate process in contact with lamina papyracea Onodi cell (sphenoethmoidal cell): present/not present, optic nerve coursing through sphenoethmoidal cell Sphenoid sinus = pneumatization, dehiscence of carotid and/or optic canal, sphenoid sinus septum attached to carotid canal Ethmoidal artery (anterior) = identi cation of entry into nasal cavity (anterior ethmoidal notch along medial orbital wall), course of artery through ethmoid sinus (within skull base, travelling freely through ethmoid sinus) Driven by these positive results we wanted to develop and evaluate an e-learning tool to teach CT anatomy of the paranasal sinuses to medical students.
The aim of the study was to evaluate whether complex contents such as CT anatomy can be introduced by e-learning and if the CLOSE mnemonic, used as a CT checklist, would improve the identi cation of anatomical variants in novices in sinus CT anatomy by providing a structured approach.In addition, we wanted to compare the results with those of otolaryngology residents and evaluate the subjective usefulness of the CLOSE criteria and e-learning for medical students.

Methods:
The e-learning was speci cally designed for the study.It consisted of two parts.The rst part incorporated a short repetition of sinus anatomy, basic principles of imaging techniques of the paranasal sinuses and CT anatomy and anatomical variants of the paranasal sinuses and their clinical relevance.The e-learning consisted of slides with theoretical content and imaging examples (Fig. 1a,b).
The second part of the e-learning introduced the CLOSE mnemonic and the different structures were illustrated on different CT images.
Medical students were asked to participate in the study.All students were 5th -year medical students.We chose senior students as they have su cient knowledge of anatomy and an advanced level of theoretical medical knowledge, comparable to junior doctors, but only little professional experience.Therefore, they are well suited to assess the effectiveness of e-learning to transfer complex learning contents such as CT anatomy of the paranasal sinuses.They were previously only taught basic CT anatomy during the lecture on nose and paranasal sinuses, which is held as a combined lecture of anatomy, radiology and clinical otorhinolaryngology.
After the students gave their informed consent, the rst part of the e-learning was distributed by e-mail.During a rst session, ten anonymized CT scans of patients in two to three planes (axial, coronal and sagittal if available) were presented to the students to be checked for anatomical variants.The scans were presented in a regular CT viewer, where the students could scroll through the images in different planes.A total of 18 anatomical variants were present in these scans.The students could scroll through the images on an individual computer and there was no time restriction for the assessment of the scans.
After this session, the second e-learning module was distributed by e-mail.In the following session, the students were asked to review the same 10 CT scans again, utilizing the CLOSE mnemonic as a CT checklist.To reduce the risk of a learning effect assessing the same CT scans, the students were not told if their answers during the rst session were correct and there were at least 4 weeks in between the two session.
After the second session, the students completed a subjective evaluation form regarding the usefulness of the checklist (modi ed from [21]).The included items are presented in Table 1.The items were rated on a 4 point Likert scale (1 = strongly agree, 2 = agree, 3 = disagree, 4 = unsure).
Table 1 Subjective evaluation of the usefulness of the CLOSE criteria and e-learning.

Item 1
The checklist is useful Item 2 The checklist makes sure, that I spend enough time reviewing the CT scans Item 3 The checklist makes me more con dent in dealing with sinus anatomy and CT scans of the paranasal sinuses Item 4 The e-learning was appropriate to introduce the topics The study was conducted according to national regulations and the declaration of Helsinki.The study was approved by the local ethics committee of the University.
The participants were given an information sheet about the study and gave written informed consent.
They were also informed about their rights according to current data protection regulation and gave written consent for the collection, analysis and storage of their personal data.
The percentage of correctly identi ed anatomical variants before and after implementation of the CLOSE criteria were compared.Eighteen variants were rated as 100%.We also evaluated each CLOSE item separately.The 10 CT scans included two variants of the cribriform plate, six of the lamina papyracea, four Onodi cells, two variants of the sphenoid sinus and two of the anterior ethmoidal artery.There were two scans with concha bullosa, which were rated in the total score of anatomical variants but not the individual items as they are not included in the CLOSE mnemonic.
We previously performed the same study with junior and senior otolaryngology residents in our institution [23].Twelve residents were shown the same ten CT scans as the students before and after the implementation of the CLOSE criteria.In contrast to the students, they were not given an introduction to the topic before the rst session as we wanted to evaluate the knowledge they acquired during their training.The CLOSE criteria were introduced during a formal teaching session.We further interpreted the residents' results by analyzing the individual CLOSE items and compared the results of students and residents.Comparison between data was computed using the Wilcoxon-test and statistical signi cance was set at a p-value of < 0.05.The comparisons were calculated with the Levene-test to de ne homo-versus heterogeneous variance and the T-test to de ne statistically signi cant differences.Graphs were compiled with Excel (Microsoft).
Descriptive data analysis was performed for the results of the evaluation form.Minimum, maximum, mean and standard deviation for each item of the evaluation form were calculated.

Results:
Ten students were included in the study and completed both sessions.Each student reviewed 10 CT scans prior and after implementation of the CLOSE mnemonic.All 10 students completed the evaluation form.
33.3% of anatomical variants were correctly identi ed during the rst session.After the implementation of the CLOSE criteria, 61.1% of the anatomical variants were correctly identi ed by the students.There was a statistically signi cant improvement in correctly identi ed anatomical variants after the implementation of the CLOSE criteria (p < 0,0001).
The results for the individual CLOSE items are summarized in Table 2.There was a signi cant improvement for C (cribriform plate), S (sphenoid sinus) and E (ethmoidal artery).The biggest percentage improvement was seen for the recognition of anatomic variants of the ethmoidal artery.The results for the subjective evaluation are summarized in Table 3.All of the students thought the CLOSE mnemonic is useful and 70% that they think that it makes sure that they spend enough time reviewing the CT scans.90% of the students strongly agreed and 10% agreed that it makes them more con dent in dealing with sinus anatomy and CT scans of the paranasal sinuses.Half of the students strongly agreed and the other half agreed that the e-learning is appropriate for the introduction of the topics.Table 3 shows the evaluation results for all items with means and standard deviations.Comparing the correctly identi ed anatomical variants for ENT residents and student we found that before the implementation of the CLOSE criteria, there was no signi cant difference in correctly identi ed anatomical variants between residents (23.6%) and students (33.3%) (p = 0.085).Afterwards the students identi ed signi cantly more anatomical variants than the residents (61.1% vs. 45.3%, p = 0.041) (Fig. 2).
The results for the individual CLOSE items for the residents are summarized in Table 2. Residents showed a signi cant improvement for C (cribriform plate), L (Lamina papyracea) and S (sphenoid sinus) with the biggest percentage improvement for anatomical variants of the sphenoid sinus. Discussion: CT anatomy is not an integral part of undergraduate training in Germany and many other countries [6,7].
Most students have little prior knowledge of CT interpretation in general and sinus CT anatomy in particular.On a national level, the German Society of Radiology has developed a curriculum for radiological teaching at German medical schools [24].For the nose and paranasal sinuses, knowledge of the most important structures of normal anatomy and basic knowledge of the most common radiological ndings in traumatic and in ammatory diseases is included but not further speci ed.The European Society of Radiology has proposed key components that should be included in undergraduate training aiming to improve and harmonize educational standards in radiology throughout Europe.For diseases of the ear, nose and throat, diseases of the paranasal sinuses are one topic that should be covered [8].Basic CT anatomy of the paranasal sinuses is required to be able to interpret abnormal ndings.The European Society of Radiology also released a statement on new approaches to undergraduate teaching in radiology [25].It recommends adopting new methods like e-learning, ipped classroom and problem based learning to teach radiology to undergraduate students.
Different studies have evaluated the feasibility of e-learning in teaching radiology to medical students.
Most of them assessed the interpretation of plain radiographs like chest x-rays.They could show that the interpretive skills can be improved and that the students rated the e-learning to be an appropriate means to transfer the contents [10,11,26].Positive results have been shown in only a few studies on more complex radiologic imaging techniques like emergency CT scans of the brain and PET/CT scans in lung cancer [27,28].
Our study assessed the interpretation of sinus CT scans.This is more complex than plain radiographs as a third dimension is added by scrolling through the images in different plains.In addition, the available studies mostly evaluated the ability to detect abnormal results.The identi cation of anatomical variants is much more di cult, as they are not as obvious as abnormal ndings.
Our results show that that e-learning seems to be an adequate tool to convey even complex contents such as CT anatomy.It's interesting to see that the students, who can be seen as novices in this eld, can transfer learning contents from xed images in the e-learning slides to moving images in three planes.
A structured approach to CT scans of the paranasal sinuses using the CLOSE criteria can further improve identi cation of critical anatomical variants signi cantly.These positive results have previously been shown in otolaryngology residents [22].Error et al showed a signi cant improvement in identi cation of critical anatomical structures in sinus CT scans after the implementation of the CLOSE criteria in 9 junior and 9 senior residents.In a single blinded study, residents were interviewed by a staff surgeon at the beginning of a surgical case to identify individual critical anatomical structures before and after the implementation of the CLOSE mnemonic.They analyzed 57 preoperative encounters (28 before and 29 after implementation).With a total of 120 CTs analyzed before and after implementation of the checklist, our study evaluates a greater number of cases.In a previous study, we also demonstrated this effect in junior and senior residents [23].
There are not many studies evaluating the usefulness of checklists in medical education.One study by Hofer could show that the implementation of a checklist in a gross anatomy laboratory improved learning outcomes and dissection quality [29].Other studies assessed their use in performance-based examinations like the OSCE.This form of exams is validated and well established in medical education.Checklists in OSCEs are used for the structured and objective evaluation of the students' practical performance [15].In contrast, we used the CLOSE criteria as a checklist to impart and apply medical knowledge.CT anatomy is much more complex than practical tasks, tested during OSCE.The structured approach with checklists is validated in several studies on OSCE and in conclusion we think that it is not necessary to evaluate the feasibility in a large cohort of students and residents [14].Our study therefore focuses on a group of 10 students and 12 residents in order to show, that a structured approach facilitated by checklists can improve the transfer of knowledge in trained and untrained subjects.
We included a subjective evaluation of the usefulness of the CLOSE criteria and the e-learning.The checklist was rated very positive by the students.A study by Yao assessed the educational value and effectiveness of a preoperative sinus CT checklist from the perspective of otolaryngology residents [21].
Their results were comparable to ours.Most residents agreed that the checklist used was useful, made them more comfortable with radiologic sinus anatomy and helped identify critical anatomical structures.Junior and senior residents gave a similar feedback in our previous study [23].The students also thought that the e-learning was an appropriate introduction to the topics.Students in other studies on e-learning for radiological imaging interpretation showed positive subjective evaluations [11,26].
Error et al also analyzed the individual CLOSE items and found a signi cant improvement postimplementation for all items except lamina papyracea [22].We found an improvement for all items in residents and all items in students except for lamina papyracea.Our results preimplementation are comparable to that of Error's study for students.Our residents showed inferior results for L and S. Our results in students and residents after the implementation of the CLOSE criteria stayed well below the identi cation rate seen in the study by Error.One explanation could be that the residents in Error's study might have been in a rhinology rotation and therefore would have been more exposed to sinus CT scans.This is the rst study comparing results in students residents.The lack of professional experience of the students compared to the residents was not re ected in the results.The students outperformed the residents before and after the implementation of the CLOSE criteria, although the difference was only statistically signi cant postimplementation.The e-learning before the rst session seems to have compensated the lack of professional experience and put the students on the same level as residents.It also seems to be better suited than traditional formal teaching sessions to transfer new learning contents like the CLOSE mnemonic.The better performance of the students could also be explained by a higher learning motivation, strategy and time.The daily workload of residents leaves little time for extensive study.The presentation of new learning contents online would enable exible study times.As we wanted to evaluate the performance of residents with their knowledge acquired during their daily work and training, they were not given a formal teaching of CT anatomy of the paranasal sinuses prior to the rst session.There is no formal teaching of sinus CT anatomy in our unit so far and no speci c rhinology rotation during which residents would gain more experience reviewing sinus CT scans, which is similar to most German otolaryngology units.German residents usually start training in sinus surgery during their 2nd and 4th year of training and most resident attend a sinus dissection course during their training to get familiar with the anatomy and surgical technique.Anatomy and CT anatomy of the sinuses is complex and sinus surgery requires the ability to mentally rebuilt the two-dimensional CT scans and textbook anatomy into a three-dimensional compilation and transfer this to the patient [20].This has been shown to be one of the main obstacles in sinus surgery training [30].Therefore, formal teaching of sinus CT anatomy should also be included in the curriculum for otolaryngology residents and the use of radiological checklists can be bene cial for a structured approach.The presentation as e-learning could ensure that studying and repetition remain exible, which might produce better results than formal teaching sessions during daily working hours.
Lack of IT skills have been identi ed to be a barrier to the success of e-learning in health care education.
Most students in developed countries can be seen as digital natives and are therefore well equipped for new online teaching methods [31].
We want to use the e-learning modules implemented in this study to develop a skills lab for training undergraduate students and otolaryngology residents in anatomy and diseases of the paranasal sinuses and endoscopic sinus surgery.The e-learning modules tested here will be included and a training model for endoscopic skills will be added.
One limitation of this study is that we only included students of one medical school.The students might have volunteered as they might have a special interest in ENT and extracurricular learning.Further studies are necessary to show whether the results can be reproduced in a different student population.We cannot exclude that there is a certain learning effect through repetition between the rst and second session of the study.As we saw a highly statistically signi cant improvement we are con dent that the improvement is not only driven by repetition but also by the structured approach facilitated by the CLOSE mnemonic.

Conclusion:
CT anatomy of the paranasal sinuses is important as CRS is a common disease and sinus surgery requires a sound knowledge of individual anatomy.CRS is the only otolaryngological disease deemed important and representative enough to make it into the new national competence-based catalogue of learning objectives for undergraduate medical training in Germany as a so-called focus disease.Our results suggest that complex contents like CT anatomy can be successfully introduced by e-learning in previously non-trained individuals with little professional experience.The identi cation of anatomical variants in sinus CT scans can be further improved by the use of a CT checklist that provides a structured approach to the assessment of critical anatomical structures.The use of e-learning to transfer radiological learning contents is positively evaluated by the students.
Examples from the rst (1a) and second (1b) e-learning module.
Statistical analysis was performed with IBM SPSS Statistics (IBM Corp. Released 2015.IBM SPSS Statistics for Windows, Version 24.0, Armonk, NY: IBM Corp.).

Table 2
Results for individual CLOSE items for students and residents before and after the implementation of the CLOSE criteria (in % correctly identi ed).

Table 3
Results for the subjective evaluation of the CLOSE mnemonic (SD: standard deviation).