Development and Validation of the Anesthesia Critical-Thinking Test (ACTT)

Background: Despite critical thinking being a crucial skill in anesthesia practice, no specic tool exists to evaluate the skill in anesthesia training. The present study measured the validity, reliability and practicality of the Anesthesia Critical-Thinking Test (ACTT) for measuring the stages of critical thinking in anesthesia settings. Methods: We developed the ACTT to evaluate critical-thinking skills by ranking learners into 6 critical-thinking stages: 1) unreective thinker, 2) beginning critical thinker, 3) practicing critical thinker, 4) advanced critical thinker, 5) accomplished critical thinker and 6) challenged thinker. Content validity was assessed by 5 experts. Concurrent validity, reliability and practicality were tested by 2 evaluators. We retrospectively and randomly assessed 47 long-essay assignments written by 1 st –3 rd year anesthesia residents in 2018. The assignments were subsequently blindly reviewed by 2 researchers to assess the critical-thinking stages. Results: The ACTT’s items were adjusted for content validity until the item-objective congruence was > 0.5. Concurrent validity between the scores and the critical-thinking stages showed moderate correlation by both evaluators (Evaluator 1 r s 0.574, P < 0.001; Evaluator 2 r s 0.603, P < 0.001). High-performance learners (stages 3, 4 and 5) had signicantly higher scores than low-performance learners (stages 1 and 2), with P = 0.001. Interrater reliability was strongly correlated (r s 0.866, P < 0.001). Both evaluators were satised with the tool’s practicality. Conclusions: The ACTT provides a valid, reliable and practical means of evaluating the critical-thinking skills of anesthesia residents.


Background
The cognitive processes and decision-making involved during the conduct of anesthesia are dynamic and complex in both crisis and non-crisis situations. 1,2 Critical thinking, the ability to apply higher cognitive skills, and the disposition to be deliberate about the thinking that leads to the performance of logical and appropriate actions, are the mainstays underlying the cognitive processes of clinical reasoning and decision-making. [3][4][5][6][7] In the context of anesthesia practice, inadequate critical thinking related to interpretation, analysis, evaluation, explanation, inference and self-regulation may lead to incorrect approaches to problems, misdiagnoses, delayed diagnoses or mismanagement. [8][9][10][11] The development of the critical-thinking skills of anesthesia residents is usually based on guiding their decision-making processes during clinical work by focusing on their behavior and the rationale of decision-making. 11 However, it is di cult to categorize trainees into different stages of critical-thinking milestones, such as novice, intermediate and master. 4 When the performance of expert anesthesiologists is analyzed, it becomes apparent that metacognition (the ability to think about one's own thinking) has played an important part in the development of the critical-thinking skills that underpin their clinical approaches. 12,13 Papp et al. classi ed the stages of development of critical thinking for the undergraduate, graduate and continuing-education levels into 6 milestone stages, ranging from "unre ective thinker" through to "accomplished critical thinker". 3 The matrix of attributes for each stage comprised "metacognitive ability" (an awareness of one's own thinking processes), "attitudes" (disposition towards critical thinking) and "skills" (referring primarily to cognitive skills).
The transition from an unre ective thinker to an accomplished critical thinker involves clinical reasoning skills throughout the progression. According to the dual process theory, clinical reasoning relies on two major systems. System 1-nonanalytical process or intuition-is automatically triggered and quickly responded to by pattern recognition. In contrast, System 2-analytical process or hypothetico-deductive reasoning-is a slower response for less familiar problems. 14,15 Most decision-making in anesthesia draws upon System 1 11 , but this process alone is not feasible for junior learners. Adams et al. found a constant dialectic between the nonanalytical and analytical processes throughout the reasoning employed by junior doctors in emergency medicine. 16 Clinical reasoning by Systems 1 and/or 2 occurs interchangeably, and it depends on an individual learner's personal experience and the complexity of the problem. The higher-stage critical thinker with a higher level of metacognition will be able to recognize biases that might occur during the nonanalytical process and know when to appropriately choose Systems 1 or 2 for each clinical situation. 15,17 The current study aimed to develop a tool based on this 6-stage framework in order to measure the stages of critical thinking in anesthesia residents. The validity, reliability and practicality of the instrument were subsequently evaluated.

Methods
Ethical approval was provided by the Siriraj Institutional Review Board. All methods were performed in accordance with the relevant guidelines and regulations.

Description of Tool Development
We developed a tool named the Anesthesia Critical-Thinking Test (ACTT) to measure the stages of critical thinking in anesthesia, utilizing the framework devised by Papp et al. 3 The original matrix is summarized in the table, see Additional le 1. Content validity was assessed by 5 educational experts in anesthesia for item-objective congruence. The educational experts were assistant professors or associate professors in anesthesia who had experience with various levels of anesthesia learners: medical students, anesthesia residents and nurse anesthetists. Items with a congruence of < 0.5 were discussed and adjusted until the congruence was > 0.5. The six stages of critical thinking in the ACTT are described in Table 1. -Any stage of critical thinker that regresses due to an internal or external stressor, e.g., time-constraints or limited resources

Context
The study population was composed of anesthesia residents undergoing the 3-year curriculum-training program at the Faculty of Medicine Siriraj Hospital, Mahidol University. We retrospectively and randomly graded 47 written assignments completed by residents during September-November 2018, which corresponded with the 3 rd -5 th months of the academic year. The randomization was done in 9 blocks, involving 3 cases and the 3 training years.

Description of the Assignment Question Format
Writing a long essay incorporates the background and rationale of thinking into decision-making; thus, we chose the scores for the written assignments to compare with the tool. The assignments consisted of 3 serial, constructed-response questions (long-essay cases with key-feature problems; Table 2), and they were scored following the normal grading procedures by staff anesthesiologists (full score: 100). Prior to this, the staff had been standardized during the rater-training process. All essays were scored using a structured scoring system along with exempli cation of answers in each subsection. The scores for each section and subsection were weighted, based on the importance of the speci c situation. The 3 topics assessed were (see Additional le 2): 1) the induction and intubation techniques for an obese patient with potential di cult airway and a full stomach who is to undergo a ruptured globe injury repair; 2) the choice of anesthesia (with rationale and technique) for a patient with coronary artery disease who is receiving dual anti-platelet therapy and has end-stage renal disease, a history of cerebrovascular disease and di cult airway, and who is scheduled to undergo an open reduction and internal xation of the ankle; and 3) the anesthetic considerations for a patient with a prosthetic heart valve and pacemaker who is to undergo an emergency gastroscopy due to upper gastrointestinal bleeding. Table 2 The format of constructed-response questions (long-essay cases with key-feature problems) Problem lists (10) Anesthetic consideration (30) Anesthetic management (60) 1.

4.
Patient Before using the ACTT for the evaluations, 2 evaluators (SS and KR) were trained for agreement on the speci c learner characteristic that related to each stage of critical thinking. The evaluators rated 4 sample assignments and compared scores to the point of reaching agreement prior to commencing the formal rating of each topic. The expected characteristics of each topic are described in Table 3. The evaluators were blinded to each resident's year of training, sequence of assignments and assignment scores. Table 3 The expected characteristics of critical-thinking stages 1-5 for the three written assignments concern for increased IOP or di cult airway GA with concern for di cult airway, or PNB with bleeding concern GA with RSI; concern for bleeding, or concern for pacemaker Stage 3 GA with RSI; concern for increased IOP and di cult airway; prepare equipment for di cult airway GA with concern for di cult airway; We did not propose an expected characteristic gathered from the written assignment for stage 5; however, learners could achieve stage 5 if they explained more fully than the stage-4 response outlined above.
Abbreviations: GA, general anesthesia; RSI, rapid-sequenced induction; IOP, intraocular pressure; PNB, peripheral nerve block * In our institution in 2018, peripheral nerve block was not routinely done under real-time ultrasound guidance. If ultrasound-guidance was not speci cally expressed in the essay, it was presumed to be done with an anatomical landmark or a nerve stimulator.

Data Collection
Concurrent validity was tested by the correlation between the assignment scores and the rankings for the stages of critical thinking. Based on the problem-solving skill levels, the 5 stages of critical thinking were further divided into 2 groups: a low-performance group (stages 1 and 2); and a high-performance group (stages 3, 4 and 5). The learners in stages 3 and higher demonstrated an awareness of the potential medical complications and were able to propose appropriate treatment plans to deal with them. The assignment scores were analyzed for each stage of critical thinking. The interrater reliability of the two evaluators was also tested. The practicality of the tool was also rated by each evaluator upon completion of the assessments, see Additional le 3.

Sample Size Calculation
The sample size was calculated using Spearman's correlation (ρ or r s ) for concurrent validity between the stages of critical thinking and the scores for the assignments. Accepting a type I (alpha) error (two-tailed) of 0.05, a type II error (beta) of 0.2 and an r of 0.7 for strong correlation 18 , the calculated sample size was 13. Since our population was large enough for a r s of 0.4 for moderate correlation, we proceeded with a calculated sample size of 47.

Statistical Analysis
The content validity was determined by item-objective congruence (IOC), with data presented as means.
An IOC value of more than 0.5 was considered acceptable. The concurrent validity between the stages of critical thinking and the assignment scores was calculated by Spearman's correlation (r s ). Scores for the high-and low-performance stages of critical thinking were compared by an independent t-test. The differences in the assignment scores relating to each stage of critical thinking were further analyzed by one-way ANOVA, with post-hoc analysis using Bonferroni's method. The interrater reliability was

Results
The tool's items were adjusted for content validity by 5 educational experts until the IOC was > 0.5. The major concern of the educational experts was how to differentiate stages 4 and 5 as some characteristics are not explicit via written assignments. Details of the IOC and comments are provided in Table 4. Other comments: This tool would be feasible to evaluate from written assignment for stages 1-3. Stages 4 and 5 should be evaluated from simulation or oral examination.
In all, 78 1st-3rd year anesthesia residents performed the writing tasks related to the 3 serial constructedresponse questions, resulting in a total of 231 scripts. Of those, 177 were retrieved, with 47 subsequently being randomly selected and graded for tool-validation purposes.
The concurrent validity between the stages of critical thinking and the assignment scores showed moderate correlation by both evaluators (Evaluator 1: r s 0.574, P < 0.001; Evaluator 2: r s 0.603, P < 0.001).

Discussion
The critical thinking stages in the ACTT were valid and correlated with the assignment scores. We divided the stages into low-performance (stages 1 and 2) and high-performance (stages 3, 4 and 5) groups, based on problem-solving skills. This milestone is important in anesthesia training as we are working in a high-stakes, fast-paced environment. 12 It is our opinion that the goal to be achieved by learners (that is, the milestone) by the end of their rst year of residency training is stage 3 ("practicing critical thinker").
This milestone is in keeping with the differentiation of junior (1 st -year) and senior (2 nd -and 3 rd -year) residents on the basis of their problem-solving skills (speci cally, on senior residents' ability to recognize potential complications that might occur and to plan problem-solving actions ahead of time).
Critical thinking is tightly linked to knowledge. Thus, by virtue of having more content knowledge, more senior residents are expected to perform at higher stages of critical thinking. However, we cannot measure only critical-thinking skills without a measure of content knowledge. Those learners with scores in the 70s could still have critical-thinking skills in either stage 2 or 3 ( Table 6). This highlights that learners with high cognitive scores do not necessarily possess the high critical-thinking skills needed for problem-solving in real clinical situations. Baker et al. also found that clinical performance scores were independently associated with board examination scores. 19 The correlation of concurrent validity demonstrated moderate correlation. The possible explanation could be drawn from the hypothesis that each tool has been designed to measure different constructs. A long-essay written assignment is designed to measure mostly content knowledge and decision-making, whereas the critical-thinking stage in the ACTT is designed to assess content knowledge, reasons, decision-making, back-up plans and problem-solving skills.
In the current study, the ACTT showed a strong correlation for interrater reliability (r s 0.866, P < 0.001) because both evaluators had prior discussions. Applying the ACTT to summative evaluations (for instance, end-of-year assessments or high-stakes board examinations) mandates ne-tuning or agreement between evaluators. The ACTT provides only broad criteria for any situation in anesthesia practice. A speci c description of the learners' characteristics that relate to the clinical situation being considered must be clari ed by evaluators before assessments are conducted.
We compared the ACTT with the standardized instruments used to evaluate critical thinking in the nursing curriculum. These comprised the Health Science Reasoning Test (HSRT), the California Critical Thinking Disposition Inventory (CCTDI), the California Critical Thinking Skill Test (CCTST), the Watson-Glaser Critical Thinking Appraisal (WGCTA), the Performance-Based Development System (PBDS) and the Critical Thinking Diagnostic (CTD). 20,21 Although our tool is as time-consuming to use as most of those instruments, it was more speci c to the professional specialty of anesthesia. Carter et al. developed 2 versions of a midwifery-speci c tool, named Carter Assessment of Critical Thinking in Midwifery (CACTiM), that provide qualitative assessments. One-CACTiM (preceptors/mentors)-assesses critical thinking in practice, whereas the other-CACTiM (re ection)-assesses critical thinking through re ective writing that requires midwifery preceptors to give feedback. 22,23 Compared with the 6 non-speci c, standardized instruments listed at the beginning of this paragraph, the ACTT requires higher levels of expertise-speci c human resources for the provision of the qualitative feedback intended to guide learners' future performance.
Application of the ACTT facilitates formative assessments for longitudinal development in that it can guide teachers in the giving of feedback to learners about the latter's current knowledge and performance statuses, and the speci c training milestones that the learners are expected to achieve. The ACTT can be implemented as a tool for teaching in the operating theatre, simulations, oral discussions or workplacebased assessments (e.g., Entrustable Professional Activity, or EPA). During interactive sessions, learners must demonstrate their rationale and decision-making should the patient's condition or the course of the disease not be proceeding as well as planned.

Limitation and Future Research
This study was retrospective, gathering data from long-essay assignments. Only one area in each case was selected to evaluate the learners' critical-thinking skills. If the assignments had been short-answer questions, the written answers would have been more explicitly expressed and better suited to criticalthinking evaluation. Due to one-way communication, the maximum stage of critical thinking reported by our study was stage 4 ("advanced critical thinker"). Unfortunately, we were unable to evaluate stage 5 ("accomplished critical thinker") from the written assignments only. Future research may focus on feasibility of the ACTT implementation on clinical performance either in the operating theatre or in a simulation. As the ACTT makes critical-thinking skills more explicit and assessable, instruction strategies should be designed to improve critical-thinking skills in the anesthesia curriculum.

Conclusions
We developed the ACTT to evaluate the stages of critical-thinking skills involved in an anesthesia setting and validated with a written assignment. ACTT has the capability to serve both teaching and evaluation purposes. -Consent for publication: Not applicable -Availability of data and materials: The datasets generated and/or analysed during the current study are not publicly available due as this may compromise participant anonymity but are available from the corresponding author on reasonable request.

List Of Abbreviations
-Competing interests: The authors declare that they have no competing interests.
-Funding: No funding was received.
-Authors' contributions: TK, KR, SS conceived, designed the study, and developed the Anesthesia Critical-Thinking Test. TK conducted a class of written assignments. KR and SS rated the stages of critical thinking. TK analyzed data and wrote the rst draft of this manuscripts and all authors contributed critically to revisions. All authors read and approved the nal manuscript.
-Acknowledgements: The authors thank Miss Julaporn Pooliam of the hospital's Research Department for her statistical support.