2.1.1 Ethical approval
The study was carried out in accordance with relevant guidelines and regulations. The study was conducted after institutional review committee approval from Ethical Committee for Institute of Life and Health Sciences (CEICVS) of School of Medicine, University of Braga, Braga, Portugal (Chairperson Prof. Dr Cecília Leão) on 15th November 2020 (CEICVS15/2020) (available Supplementary material Ethical file). Participation in this study was absolutely voluntary and all participants and assessors gave an oral and written informed.
In addition, all were aware that they could leave the study at any time without any problem.
2.1.2 Study dates
The study was conducted at School of Medicine, University of Minho, Braga, Portugal, from November 2020 to June 2021.
2.1.3 Study Design
This is a prospective experimental, rater and principal investigator double-blinded study to determine OSAUS’s psychometric properties when the scale is used for the ultrasound measurement of hyomental distance in head neutral and extended position.
Figure 1 represents the study design (Figure 1).
The study has developed in 4 steps. In the first step an educational moment was organized with a theoretical presentation of 1 hour. Two experienced airway ultrasound anesthesiologists presented the OSAUS scale (17) and its applicability to measuring ultrasound hyomental distance in neutral and extended positions. The protocol for HMD measurement in neutral and extended position was very well-defined and given to participants and was available for consult as a guide for practice. After this session, 3 hours of practical training of the HMD ultrasound measurements were done, tutorized by the same trainers.
Two weeks later, in the second step, participants completed the ultrasound measurement of ultrasound HMD in neutral and extended position, and the performance was video recorded. Each participant evaluated ultrasound HMD from three standardized patients, generating a total of 45 videos.
In step 3, six assessors were recruited and received online guidance on the OSAUS scale and its applicability for assessing ultrasound HMD measurement. A concrete preparation analyzing a pilot video study was done.
In step 4, assessors blindly rated participants videorecords according to OSAUS and a global rating scale (GRS) with 5-Likert points (1 point - unacceptable; 2 points – weak; 3 points – acceptable; 4 points – Very good; 5 points – excellent performance). For further analysis, the time needed to complete participants’ tasks was also collected. The assessors' evaluation was sent in an anonymous excel file.
Volunteer participants provided informed consent and self-reported their experience with airway ultrasound before their enrolment. According to participants experience, three categories were created: novices, intermediates and experts. A novice participant had up to six months experience in airway ultrasound, and an expert uses airway ultrasound for more than two years. The intermediate group enrolled participants within the two groups.
Standardized patients (SP) volunteers: Nine SP were recruited for the study and participated in both moments (training and validation) and provided informed consent before participating on the study.
2.1.5 Equipment and environment
Steps 1 and 2 were realized at School of Medicine, University of Minho, and the study equipment and environment was the same for the practical session and the assessment time. Ultrasound measurements were obtained using a SonoSite®, portable ultrasound machine (Fujifilm, SonoSite® Edge II and SonoSite® SII, Ultrasound System, Inc Bothell, WA, USA), using a curvilinear, multifrequency 3-8 MHz ultrasound transducer probe.
Participant measurement of HMD in neutral and extended head position was video recorded. The angle of video records provided a global overview of the technique, including the face of the SP; both hands of the sonographer and the all ultrasound machine. In all recordings, both the practitioner’s technique and the ultrasound were visible.
Physicians were anonymized by not recording their faces or voices. Once the procedure was finalized, the film clips were all stored and referenced by order of collection. An anonymous link to a folder with the videos was sent to each assessor.
2.2.1 Psychometric study
The analysis of OSAUS’s psychometric properties was based on the Standards for Educational and Psychological Testing (American Educational Research Association - AERA, American Psychological Association & National Council on Measurement in Education, 2014)(28), following the category framework articulated by Messick (29) (content, internal structure, relation to other variables, response process and consequences).
The internal structure or construct validity was analyzed by internal consistency and interrater reliability.
The relation to other variables included the criterion-related analysis (concurrent validity), by comparing OSAUS with GRS, where raters were instructed to rate the overall participant´s performance and by comparing time to complete the task from different experience levels (expert, intermediate and novice).
The response process of OSAUS scale evaluation focused on data collection methods; rater instructions, training and performance; how scores were reported and summarized and in the methods responsible for the lack of bias in the process.(30)
2.2.2 Statistical analysis
The internal consistency was assessed through Cronbach’s alpha for each item used (items 2, 3, 4 and 5). For each participant, we calculated the mean score of the 3 ultrasound measurements and the mean time in seconds to complete the task. Intraclass Correlation (ICC) estimates and their 95% confidence interval were calculated. The ICC two-way-random effect model was used to evaluate consistency between raters based on the mean value of the OSAUS score from 5 raters (k=5), consistency, 2-way random effects model. (31) An analysis of variance (ANOVA) for repeated measurements was also done. Convergent validity was assessed by a Pearson´s correlation between OSAUS and Global Rating Scale.
One-way ANOVA explored differences in OSAUS rating scores and differences in time to complete the task between different levels of competence.
The statistical analysis was done using SPSS version 27 (IBM Corp, Armonk, NY) with P values below 0.05 were interpreted as statistical significance and the strength of agreement were interpreted according to Portney and co-workers (31) where values under 0.5 represent poor reliability, values between 0.5 to 0.69 considered moderate, values between 0.7 to 0.9 indicate strong and over 0.9 represent excellent reliability.