Development and Validation of the Anesthesia Critical-Thinking Test (ACTT)

Abstract

Background: Despite critical thinking being a crucial skill in anesthesia practice, no specific tool exists to evaluate the skill in anesthesia training. The present study measured the validity, reliability and practicality of the Anesthesia Critical-Thinking Test (ACTT) for measuring the stages of critical thinking in anesthesia settings.

Methods: We developed the ACTT to evaluate critical-thinking skills by ranking learners into 6 critical-thinking stages: 1) unreflective thinker, 2) beginning critical thinker, 3) practicing critical thinker, 4) advanced critical thinker, 5) accomplished critical thinker and 6) challenged thinker. Content validity was assessed by 5 experts. Concurrent validity, reliability and practicality were tested by 2 evaluators. We retrospectively and randomly assessed 47 long-essay assignments written by 1^st–3^rd year anesthesia residents in 2018. The assignments were subsequently blindly reviewed by 2 researchers to assess the critical-thinking stages.

Results: The ACTT’s items were adjusted for content validity until the item-objective congruence was > 0.5. Concurrent validity between the scores and the critical-thinking stages showed moderate correlation by both evaluators (Evaluator 1 r_s 0.574, P < 0.001; Evaluator 2 r_s 0.603, P < 0.001). High-performance learners (stages 3, 4 and 5) had significantly higher scores than low-performance learners (stages 1 and 2), with P = 0.001. Interrater reliability was strongly correlated (r_s 0.866, P < 0.001). Both evaluators were satisfied with the tool’s practicality.

Conclusions: The ACTT provides a valid, reliable and practical means of evaluating the critical-thinking skills of anesthesia residents.

Background

The cognitive processes and decision-making involved during the conduct of anesthesia are dynamic and complex in both crisis and non-crisis situations.^1,2 Critical thinking, the ability to apply higher cognitive skills, and the disposition to be deliberate about the thinking that leads to the performance of logical and appropriate actions, are the mainstays underlying the cognitive processes of clinical reasoning and decision-making.^3–7 In the context of anesthesia practice, inadequate critical thinking related to interpretation, analysis, evaluation, explanation, inference and self-regulation may lead to incorrect approaches to problems, misdiagnoses, delayed diagnoses or mismanagement.^8–11

The development of the critical-thinking skills of anesthesia residents is usually based on guiding their decision-making processes during clinical work by focusing on their behavior and the rationale of decision-making.¹¹ However, it is difficult to categorize trainees into different stages of critical-thinking milestones, such as novice, intermediate and master.⁴ When the performance of expert anesthesiologists is analyzed, it becomes apparent that metacognition (the ability to think about one’s own thinking) has played an important part in the development of the critical-thinking skills that underpin their clinical approaches.^12,13 Papp et al. classified the stages of development of critical thinking for the undergraduate, graduate and continuing-education levels into 6 milestone stages, ranging from “unreflective thinker” through to “accomplished critical thinker”.³ The matrix of attributes for each stage comprised “metacognitive ability” (an awareness of one’s own thinking processes), “attitudes” (disposition towards critical thinking) and “skills” (referring primarily to cognitive skills).

The transition from an unreflective thinker to an accomplished critical thinker involves clinical reasoning skills throughout the progression. According to the dual process theory, clinical reasoning relies on two major systems. System 1—nonanalytical process or intuition—is automatically triggered and quickly responded to by pattern recognition. In contrast, System 2—analytical process or hypothetico-deductive reasoning—is a slower response for less familiar problems.^14,15 Most decision-making in anesthesia draws upon System 1 ¹¹, but this process alone is not feasible for junior learners. Adams et al. found a constant dialectic between the nonanalytical and analytical processes throughout the reasoning employed by junior doctors in emergency medicine.¹⁶ Clinical reasoning by Systems 1 and/or 2 occurs interchangeably, and it depends on an individual learner’s personal experience and the complexity of the problem. The higher-stage critical thinker with a higher level of metacognition will be able to recognize biases that might occur during the nonanalytical process and know when to appropriately choose Systems 1 or 2 for each clinical situation.^15,17

The current study aimed to develop a tool based on this 6-stage framework in order to measure the stages of critical thinking in anesthesia residents. The validity, reliability and practicality of the instrument were subsequently evaluated.

Methods

Ethical approval was provided by the Siriraj Institutional Review Board. All methods were performed in accordance with the relevant guidelines and regulations.

Description of Tool Development

We developed a tool named the Anesthesia Critical-Thinking Test (ACTT) to measure the stages of critical thinking in anesthesia, utilizing the framework devised by Papp et al.³ The original matrix is summarized in the table, see Additional file 1. Content validity was assessed by 5 educational experts in anesthesia for item-objective congruence. The educational experts were assistant professors or associate professors in anesthesia who had experience with various levels of anesthesia learners: medical students, anesthesia residents and nurse anesthetists. Items with a congruence of < 0.5 were discussed and adjusted until the congruence was > 0.5. The six stages of critical thinking in the ACTT are described in Table 1.

Table 1 The description of stages of critical thinking of the Anesthesia Critical-Thinking Test (ACTT). Adapted with permission from Papp KK, Huang GC, Lauzon Clabo LM, et al. Milestones of critical thinking: a developmental model for medicine and nursing. Acad Med. 2014; 89(5):715-20.

Context

The study population was composed of anesthesia residents undergoing the 3-year curriculum-training program at the Faculty of Medicine Siriraj Hospital, Mahidol University. We retrospectively and randomly graded 47 written assignments completed by residents during September–November 2018, which corresponded with the 3^rd–5^th months of the academic year. The randomization was done in 9 blocks, involving 3 cases and the 3 training years.

Description of the Assignment Question Format

Writing a long essay incorporates the background and rationale of thinking into decision-making; thus, we chose the scores for the written assignments to compare with the tool. The assignments consisted of 3 serial, constructed-response questions (long-essay cases with key-feature problems; Table 2), and they were scored following the normal grading procedures by staff anesthesiologists (full score: 100). Prior to this, the staff had been standardized during the rater-training process. All essays were scored using a structured scoring system along with exemplification of answers in each subsection. The scores for each section and subsection were weighted, based on the importance of the specific situation. The 3 topics assessed were (see Additional file 2):

1) the induction and intubation techniques for an obese patient with potential difficult airway and a full stomach who is to undergo a ruptured globe injury repair;

2) the choice of anesthesia (with rationale and technique) for a patient with coronary artery disease who is receiving dual anti-platelet therapy and has end-stage renal disease, a history of cerebrovascular disease and difficult airway, and who is scheduled to undergo an open reduction and internal fixation of the ankle; and

3) the anesthetic considerations for a patient with a prosthetic heart valve and pacemaker who is to undergo an emergency gastroscopy due to upper gastrointestinal bleeding.

Table 2 The format of constructed-response questions (long-essay cases with key-feature problems)

Rater Training

Before using the ACTT for the evaluations, 2 evaluators (SS and KR) were trained for agreement on the specific learner characteristic that related to each stage of critical thinking. The evaluators rated 4 sample assignments and compared scores to the point of reaching agreement prior to commencing the formal rating of each topic. The expected characteristics of each topic are described in Table 3. The evaluators were blinded to each resident’s year of training, sequence of assignments and assignment scores.

Table 3 The expected characteristics of critical-thinking stages 1–5 for the three written assignments

Abbreviations: GA, general anesthesia; RSI, rapid-sequenced induction; IOP, intraocular pressure; PNB, peripheral nerve block

* In our institution in 2018, peripheral nerve block was not routinely done under real-time ultrasound guidance. If ultrasound-guidance was not specifically expressed in the essay, it was presumed to be done with an anatomical landmark or a nerve stimulator.

Data Collection

Concurrent validity was tested by the correlation between the assignment scores and the rankings for the stages of critical thinking. Based on the problem-solving skill levels, the 5 stages of critical thinking were further divided into 2 groups: a low-performance group (stages 1 and 2); and a high-performance group (stages 3, 4 and 5). The learners in stages 3 and higher demonstrated an awareness of the potential medical complications and were able to propose appropriate treatment plans to deal with them. The assignment scores were analyzed for each stage of critical thinking. The interrater reliability of the two evaluators was also tested. The practicality of the tool was also rated by each evaluator upon completion of the assessments, see Additional file 3.

Sample Size Calculation

The sample size was calculated using Spearman’s correlation (ρ or r_s) for concurrent validity between the stages of critical thinking and the scores for the assignments. Accepting a type I (alpha) error (two-tailed) of 0.05, a type II error (beta) of 0.2 and an r of 0.7 for strong correlation¹⁸, the calculated sample size was 13. Since our population was large enough for a r_s of 0.4 for moderate correlation, we proceeded with a calculated sample size of 47.

Statistical Analysis

The content validity was determined by item-objective congruence (IOC), with data presented as means. An IOC value of more than 0.5 was considered acceptable. The concurrent validity between the stages of critical thinking and the assignment scores was calculated by Spearman’s correlation (r_s). Scores for the high- and low-performance stages of critical thinking were compared by an independent t-test. The differences in the assignment scores relating to each stage of critical thinking were further analyzed by one-way ANOVA, with post-hoc analysis using Bonferroni’s method. The interrater reliability was calculated using Spearman’s correlation. The strength of correlation (r_s) was interpreted as follows: < 0.1, negligible; 0.10–0.39, weak; 0.40–0.69, moderate; 0.70–0.89, strong; and 0.90–1.0, very strong correlation.¹⁸ A P-value of < 0.05 was considered statistically significant. The practicality of the tool was analysed with descriptive statistics. All data were analysed using PASW Statistics for Windows (version 18.0; SPSS Inc., Chicago, IL, USA).

Results

The tool’s items were adjusted for content validity by 5 educational experts until the IOC was > 0.5. The major concern of the educational experts was how to differentiate stages 4 and 5 as some characteristics are not explicit via written assignments. Details of the IOC and comments are provided in Table 4.

Table 4 Item-objective congruence (IOC) for stages of critical thinking by 5 educational experts

Other comments: This tool would be feasible to evaluate from written assignment for stages 1–3. Stages 4 and 5 should be evaluated from simulation or oral examination.

In all, 78 1st–3rd year anesthesia residents performed the writing tasks related to the 3 serial constructed-response questions, resulting in a total of 231 scripts. Of those, 177 were retrieved, with 47 subsequently being randomly selected and graded for tool-validation purposes.

The concurrent validity between the stages of critical thinking and the assignment scores showed moderate correlation by both evaluators (Evaluator 1: r_s 0.574, P < 0.001; Evaluator 2: r_s 0.603, P < 0.001). When we further separated the learners into 2 subgroups (low performance = critical-thinking stages 1 and 2; high performance = critical-thinking stages 3, 4 and 5), learners in the high-performance group had statistically significantly higher scores than the low-performance group (Table 5). The differences in the assignment scores relating to each stage of critical thinking are presented in Table 6.

Table 5 Concurrent validity between the stage of critical thinking and the assignment scores, divided into low- and high-performance stages

Table 6 Differences in the scores for each stage of critical thinking, as assessed by the 2 evaluators

* P-values for stages 1-2, 1-3, and 1-4 = 0.018, < 0.001, and < 0.001, respectively; P-values for stages 2-3 and 2-4 = 0.055 and 0.01, respectively; P-value for stages 3-4 = 1.0.

** P-values for stages 2-3, 2-4 and 3-4 = 0.001, <0.001, and 0.709, respectively.

The interrater reliability for the stages of critical thinking was strongly correlated (r_s 0.866, P < 0.001). The tool’s practicality was expressed by both evaluators using a Likert scale (ranging from 1, “least,” through to 5, “most”). The data of Evaluator 1 and Evaluator 2 were calculated as means. The 5 aspects they evaluated were “simple to use” = 4.5 (4, 5); “time-wasting” = 1.5 (2, 1); “difficult to perform assessments” = 3 (2, 4); “able to differentiate residents” = 3.5 (3, 4); and “appropriate for regular assessment” = 4 (4, 4).

Discussion

The critical thinking stages in the ACTT were valid and correlated with the assignment scores. We divided the stages into low-performance (stages 1 and 2) and high-performance (stages 3, 4 and 5) groups, based on problem-solving skills. This milestone is important in anesthesia training as we are working in a high-stakes, fast-paced environment.¹² It is our opinion that the goal to be achieved by learners (that is, the milestone) by the end of their first year of residency training is stage 3 (“practicing critical thinker”). This milestone is in keeping with the differentiation of junior (1^st-year) and senior (2^nd- and 3^rd-year) residents on the basis of their problem-solving skills (specifically, on senior residents’ ability to recognize potential complications that might occur and to plan problem-solving actions ahead of time).

Critical thinking is tightly linked to knowledge. Thus, by virtue of having more content knowledge, more senior residents are expected to perform at higher stages of critical thinking. However, we cannot measure only critical-thinking skills without a measure of content knowledge. Those learners with scores in the 70s could still have critical-thinking skills in either stage 2 or 3 (Table 6). This highlights that learners with high cognitive scores do not necessarily possess the high critical-thinking skills needed for problem-solving in real clinical situations. Baker et al. also found that clinical performance scores were independently associated with board examination scores.¹⁹ The correlation of concurrent validity demonstrated moderate correlation. The possible explanation could be drawn from the hypothesis that each tool has been designed to measure different constructs. A long-essay written assignment is designed to measure mostly content knowledge and decision-making, whereas the critical-thinking stage in the ACTT is designed to assess content knowledge, reasons, decision-making, back-up plans and problem-solving skills.

In the current study, the ACTT showed a strong correlation for interrater reliability (r_s 0.866, P < 0.001) because both evaluators had prior discussions. Applying the ACTT to summative evaluations (for instance, end-of-year assessments or high-stakes board examinations) mandates fine-tuning or agreement between evaluators. The ACTT provides only broad criteria for any situation in anesthesia practice. A specific description of the learners’ characteristics that relate to the clinical situation being considered must be clarified by evaluators before assessments are conducted.

We compared the ACTT with the standardized instruments used to evaluate critical thinking in the nursing curriculum. These comprised the Health Science Reasoning Test (HSRT), the California Critical Thinking Disposition Inventory (CCTDI), the California Critical Thinking Skill Test (CCTST), the Watson–Glaser Critical Thinking Appraisal (WGCTA), the Performance-Based Development System (PBDS) and the Critical Thinking Diagnostic (CTD).^20,21 Although our tool is as time-consuming to use as most of those instruments, it was more specific to the professional specialty of anesthesia. Carter et al. developed 2 versions of a midwifery-specific tool, named Carter Assessment of Critical Thinking in Midwifery (CACTiM), that provide qualitative assessments. One—CACTiM (preceptors/mentors)—assesses critical thinking in practice, whereas the other—CACTiM (reflection)—assesses critical thinking through reflective writing that requires midwifery preceptors to give feedback.^22,23 Compared with the 6 non-specific, standardized instruments listed at the beginning of this paragraph, the ACTT requires higher levels of expertise-specific human resources for the provision of the qualitative feedback intended to guide learners’ future performance.

Application of the ACTT facilitates formative assessments for longitudinal development in that it can guide teachers in the giving of feedback to learners about the latter’s current knowledge and performance statuses, and the specific training milestones that the learners are expected to achieve. The ACTT can be implemented as a tool for teaching in the operating theatre, simulations, oral discussions or workplace-based assessments (e.g., Entrustable Professional Activity, or EPA). During interactive sessions, learners must demonstrate their rationale and decision-making should the patient’s condition or the course of the disease not be proceeding as well as planned.

Limitation and Future Research

This study was retrospective, gathering data from long-essay assignments. Only one area in each case was selected to evaluate the learners’ critical-thinking skills. If the assignments had been short-answer questions, the written answers would have been more explicitly expressed and better suited to critical-thinking evaluation. Due to one-way communication, the maximum stage of critical thinking reported by our study was stage 4 (“advanced critical thinker”). Unfortunately, we were unable to evaluate stage 5 (“accomplished critical thinker”) from the written assignments only. Future research may focus on feasibility of the ACTT implementation on clinical performance either in the operating theatre or in a simulation. As the ACTT makes critical-thinking skills more explicit and assessable, instruction strategies should be designed to improve critical-thinking skills in the anesthesia curriculum.

Conclusions

We developed the ACTT to evaluate the stages of critical-thinking skills involved in an anesthesia setting and validated with a written assignment. ACTT has the capability to serve both teaching and evaluation purposes.

List of abbreviations

ACTT: Anesthesia Critical-Thinking Test; GA: general anesthesia; RSI: rapid-sequenced induction; IOP: intraocular pressure; PNB: peripheral nerve block; IOC: item-objective congruence; MEQ: Modified Essay Question; HSRT:Health Science Reasoning Test; CCTDI: California Critical Thinking Disposition Inventory; CCTST: California Critical Thinking Skill Test; WGCTA: Watson–Glaser Critical Thinking Appraisal; PBDS: Performance-Based Development System; CTD: Critical Thinking Diagnostic; CACTiM: Carter Assessment of Critical Thinking in Midwifery; EPA: Entrustable Professional Activity

Declarations

- Ethical approval and consent to participate: Ethical approval for this study (Si526/2019) was provided by the Siriraj Institutional Review Board, Bangkok, Thailand. Informed consent was waived and approved.

- Consent for publication: Not applicable

- Availability of data and materials: The datasets generated and/or analysed during the current study are not publicly available due as this may compromise participant anonymity but are available from the corresponding author on reasonable request.

- Competing interests: The authors declare that they have no competing interests.

- Funding: No funding was received.

- Authors’ contributions: TK, KR, SS conceived, designed the study, and developed the Anesthesia Critical-Thinking Test. TK conducted a class of written assignments. KR and SS rated the stages of critical thinking. TK analyzed data and wrote the first draft of this manuscripts and all authors contributed critically to revisions. All authors read and approved the final manuscript.

- Acknowledgements: The authors thank Miss Julaporn Pooliam of the hospital’s Research Department for her statistical support.

References

1. Gaba, D.M. Dynamic decision-making in anesthesiology: cognitive models and training approaches. In: Evans DA, Patel VL, editors. Advanced models of cognition for medical training and practice. Berlin, Heidelberg: Springer Verlag; 1992. p. 123-47.
2. Croskerry P. The theory and practice of clinical decision-making. Can J Anesth. 52(1), R1–R8 (2005). https://doi.org/10.1007/BF03023077.
3. Papp K.K., Huang G.C., Lauzon Clabo L.M., Delva D., Fischer M., Konopasek L., et al. Milestones of critical thinking: a developmental model for medicine and nursing. Acad Med. 89(5), 715-20 (2014). https://doi.org/10.1097/ACM.0000000000000220.
4. Huang G.C., Newman L.R. & Schwartzstein R.M. Critical thinking in health professions education: summary and consensus statements of the Millennium Conference 2011. Teach Learn Med. 26(1), 95-102 (2014). https://doi.org/10.1080/10401334.2013.857335.
5. Huang G.C., Lindell D., Jaffe L.E. & Sullivan A.M. A multi-site study of strategies to teach critical thinking: “Why do you think that?”. Med Educ. 50(2), 236-49 (2016). https://doi.org/10.1111/medu.12937.
6. Sharples J.M., Oxman A.D., Mahtani K.R., Chalmers I., Oliver S., Collins K., et al. Critical thinking in healthcare and education. BMJ. 357, j2234 (2017). https://doi.org/10.1136/bmj.j2234.
7. Victor-Chmil J. Critical thinking versus clinical reasoning versus clinical judgment: differential diagnosis. Nurse Educ. 38(1), 34-6 (2013). https://doi.org/10.1097/NNE.0b013e318276dfbe.
8. Facione N.C. & Facione P.A. Critical Thinking and Clinical Judgement. In: Facione N.C. & Facione P.A., editors. Critical Thinking and Clinical Reasoning in the Health Sciences: A Teaching Anthology. Millbrae, CA: California Academic Press; 2008. p. 1-13.
9. Facione P. Critical Thinking: What It Is and Why It Counts. Insight Assessment [Internet]. (2015) Accessed 15 December 2019. Available from: www.researchgate.net/publication/251303244_Critical_Thinking_What_It_Is_and_Why_It_Counts.
10. Hayes M.M., Chatterjee S. & Schwartzstein R.M. Critical Thinking in Critical Care: Five Strategies to Improve Teaching and Learning in the Intensive Care Unit. Ann Am Thorac Soc. 14(4), 569-75 (2017). https://doi.org/10.1513/AnnalsATS.201612-1009AS.
11. Stiegler M.P. & Tung A. Cognitive processes in anesthesiology decision making. Anesthesiology. 120(1), 204-17 (2014). https://doi.org/10.1097/ALN.0000000000000073.
12. Gaba D.M., Fish K.J., Howard S.K. & Burden A.R. Crisis Management in Anesthesiology. 2 ed. Philadelphia, PA: Elsevier Saunders; 2015.
13. Stiegler M.P., Gaba D.M. Decision-making and cognitive strategies. Simul Healthc. 10(3), 133-8 (2015). https://doi.org/10.1097/SIH.0000000000000093.
14. Audetat M.C., Laurin S., Dory V., Charlin B. & Nendaz M.R. Diagnosis and management of clinical reasoning difficulties: Part I. Clinical reasoning supervision and educational diagnosis. Med Teach. 39(8), 792-6 (2017). https://doi.org/10.1080/0142159X.2017.1331033.
15. Pelaccia T., Tardif J., Triby E. & Charlin B. An analysis of clinical reasoning through a recent and comprehensive approach: the dual-process theory. Med Educ Online. 16 (2011). https://doi.org/10.3402/meo.v16i0.5890.
16. Adams E., Goyder C., Heneghan C., Brand L. & Ajjawi R. Clinical reasoning of junior doctors in emergency medicine: a grounded theory study. Emerg Med J. 34(2), 70-5 (2017). https://doi.org/10.1136/emermed-2015-205650.
17. Marcum J.A. An integrated model of clinical reasoning: dual-process theory of cognition and metacognition. J Eval Clin Pract. 18(5), 954-61 (2012). https://doi.org/10.1111/j.1365-2753.2012.01900.x.
18. Schober P., Boer C. & Schwarte L.A. Correlation Coefficients: Appropriate Use and Interpretation. Anesth Analg. 126(5), 1763-8 (2018). https://doi.org/10.1213/ANE.0000000000002864.
19. Baker K., Sun H., Harman A., Poon K.T. & Rathmell J.P. Clinical Performance Scores Are Independently Associated with the American Board of Anesthesiology Certification Examination Scores. Anesth Analg. 122(6), 1992-9 (2016). https://doi.org/10.1213/ANE.0000000000001288.
20. Carter A.G., Creedy D.K. & Sidebotham M. Evaluation of tools used to measure critical thinking development in nursing and midwifery undergraduate students: a systematic review. Nurse Educ Today. 35(7), 864-74 (2015). https://doi.org/10.1016/j.nedt.2015.02.023.
21. Zuriguel Perez E., Lluch Canut M.T., Falco Pegueroles A., Puig Llobet M., Moreno Arroyo C. & Roldan Merino J. Critical thinking in nursing: Scoping review of the literature. Int J Nurs Pract. 21(6), 820-30 (2015). https://doi.org/10.1111/ijn.12347.
22. Carter A.G., Creedy D.K. & Sidebotham M. Development and psychometric testing of the Carter Assessment of Critical Thinking in Midwifery (Preceptor/Mentor version). Midwifery. 34, 141-9 (2016). https://doi.org/10.1016/j.midw.2015.12.002.
23. Carter A.G., Creedy D.K. & Sidebotham M. Critical thinking evaluation in reflective writing: Development and testing of Carter Assessment of Critical Thinking in Midwifery (Reflection). Midwifery. 54, 73-80 (2017). https://doi.org/10.1016/j.midw.2017.08.003.