Ethical approval was provided by the Siriraj Institutional Review Board. All methods were performed in accordance with the relevant guidelines and regulations.
Description of Tool Development
We developed a tool named the Anesthesia Critical-Thinking Test (ACTT) to measure the stages of critical thinking in anesthesia, utilizing the framework devised by Papp et al.3 The original matrix is summarized in the table, see Additional file 1. Content validity was assessed by 5 educational experts in anesthesia for item-objective congruence. The educational experts were assistant professors or associate professors in anesthesia who had experience with various levels of anesthesia learners: medical students, anesthesia residents and nurse anesthetists. Items with a congruence of < 0.5 were discussed and adjusted until the congruence was > 0.5. The six stages of critical thinking in the ACTT are described in Table 1.
Table 1 The description of stages of critical thinking of the Anesthesia Critical-Thinking Test (ACTT). Adapted with permission from Papp KK, Huang GC, Lauzon Clabo LM, et al. Milestones of critical thinking: a developmental model for medicine and nursing. Acad Med. 2014; 89(5):715-20.
Stage
|
Description
|
Stage 1: unreflective thinker
(only one approach)
|
– Unaware of different approaches to thinking
– Single approach to gathering and processing information
|
Stage 2: beginning critical thinker
(more than one approach)
|
– Aware of different approaches to thinking, but limited
– Unaware of potential complications of each approach
|
Stage 3: practicing critical thinker
(plans for problem-solving)
|
– Aware of multiple approaches to thinking
– Aware of complications that might occur
– Able to plan for problem-solving ahead of time
|
Stage 4: advanced critical thinker
(plans for complicated problem-solving)
|
– Consciously performs critical thinking
– Aware of multiple approaches to thinking and potential complications
– Able to plan for problem-solving ahead of time
– Uses intuitive and analytical strategies to change plans, based on the current situation
|
Stage 5: accomplished critical thinker
(innovation)
|
– Consciously performs critical thinking
– Aware of multiple approaches to thinking and potential complications
– Able to plan for problem-solving ahead of time
– Uses intuitive and analytical strategies to change plans, based on the current situation
– Able to identify cognitive biases that might occur
– Able to create plausible hypotheses to explain observed phenomena, and to create new knowledge or understanding by reasoning inductively in this way
|
Challenged thinker
(temporary)
|
– Any stage of critical thinker that regresses due to an internal or external stressor, e.g., time-constraints or limited resources
|
Context
The study population was composed of anesthesia residents undergoing the 3-year curriculum-training program at the Faculty of Medicine Siriraj Hospital, Mahidol University. We retrospectively and randomly graded 47 written assignments completed by residents during September–November 2018, which corresponded with the 3rd–5th months of the academic year. The randomization was done in 9 blocks, involving 3 cases and the 3 training years.
Description of the Assignment Question Format
Writing a long essay incorporates the background and rationale of thinking into decision-making; thus, we chose the scores for the written assignments to compare with the tool. The assignments consisted of 3 serial, constructed-response questions (long-essay cases with key-feature problems; Table 2), and they were scored following the normal grading procedures by staff anesthesiologists (full score: 100). Prior to this, the staff had been standardized during the rater-training process. All essays were scored using a structured scoring system along with exemplification of answers in each subsection. The scores for each section and subsection were weighted, based on the importance of the specific situation. The 3 topics assessed were (see Additional file 2):
1) the induction and intubation techniques for an obese patient with potential difficult airway and a full stomach who is to undergo a ruptured globe injury repair;
2) the choice of anesthesia (with rationale and technique) for a patient with coronary artery disease who is receiving dual anti-platelet therapy and has end-stage renal disease, a history of cerebrovascular disease and difficult airway, and who is scheduled to undergo an open reduction and internal fixation of the ankle; and
3) the anesthetic considerations for a patient with a prosthetic heart valve and pacemaker who is to undergo an emergency gastroscopy due to upper gastrointestinal bleeding.
Table 2 The format of constructed-response questions (long-essay cases with key-feature problems)
Problem lists (10)
|
Anesthetic consideration (30)
|
Anesthetic management (60)
|
1.
2.
3.
4.
|
Patient factor
-
-
Surgical factor
-
-
Anesthetic factor
-
-
|
Preoperative evaluation
Preoperative preparation
- general
- specific
Premedication
Intraoperative plan
- choice of anesthesia
- monitoring
- induction, intubation, maintenance
- fluid, ventilation, position
- special concern
- emergence
Postoperative care
- general
- specific
|
Rater Training
Before using the ACTT for the evaluations, 2 evaluators (SS and KR) were trained for agreement on the specific learner characteristic that related to each stage of critical thinking. The evaluators rated 4 sample assignments and compared scores to the point of reaching agreement prior to commencing the formal rating of each topic. The expected characteristics of each topic are described in Table 3. The evaluators were blinded to each resident’s year of training, sequence of assignments and assignment scores.
Table 3 The expected characteristics of critical-thinking stages 1–5 for the three written assignments
Topic
|
1
|
2
|
3
|
Area of evaluation
|
The induction and intubation techniques for an obese patient with potential difficult airway, full stomach and a ruptured globe
|
The choice of anesthesia, with rationale and technique, for a patient with coronary artery disease receiving dual anti-platelet therapy, end-stage renal disease, a history of cerebrovascular disease and difficult airway who is scheduled to undergo an open reduction and internal fixation of the ankle
|
The anesthetic considerations for a patient with a prosthetic heart valve and pacemaker who is to undergo emergency gastroscopy due to upper gastrointestinal bleeding
|
Stage 1
|
GA with RSI
|
GA or PNB* alone
|
GA with RSI
|
Stage 2
|
GA with RSI; concern for increased IOP or difficult airway
|
GA with concern for difficult airway, or PNB with bleeding concern
|
GA with RSI; concern for bleeding, or concern for pacemaker
|
Stage 3
|
GA with RSI; concern for increased IOP and difficult airway; prepare equipment for difficult airway
|
GA with concern for difficult airway; post-op pain control; perioperative stroke; and prepare equipment, PNB with ultrasound guided
|
GA with RSI; change pacemaker mode to DOO; prepare for temporary pacemaker in case of pace failure; prepare blood and blood component
|
Stage 4
|
Similar to stage 3; provide alternative plan if unable to intubate on first attempt
|
Similar to stage 3; perioperative stroke surveillance; PNB with sedation to control tourniquet pain. Provide back-up plan if first plan did not go well.
|
Similar to stage 3; discuss extubation; necessity for post-operative critical care
|
Stage 5
|
We did not propose an expected characteristic gathered from the written assignment for stage 5; however, learners could achieve stage 5 if they explained more fully than the stage-4 response outlined above.
|
Abbreviations: GA, general anesthesia; RSI, rapid-sequenced induction; IOP, intraocular pressure; PNB, peripheral nerve block
* In our institution in 2018, peripheral nerve block was not routinely done under real-time ultrasound guidance. If ultrasound-guidance was not specifically expressed in the essay, it was presumed to be done with an anatomical landmark or a nerve stimulator.
Data Collection
Concurrent validity was tested by the correlation between the assignment scores and the rankings for the stages of critical thinking. Based on the problem-solving skill levels, the 5 stages of critical thinking were further divided into 2 groups: a low-performance group (stages 1 and 2); and a high-performance group (stages 3, 4 and 5). The learners in stages 3 and higher demonstrated an awareness of the potential medical complications and were able to propose appropriate treatment plans to deal with them. The assignment scores were analyzed for each stage of critical thinking. The interrater reliability of the two evaluators was also tested. The practicality of the tool was also rated by each evaluator upon completion of the assessments, see Additional file 3.
Sample Size Calculation
The sample size was calculated using Spearman’s correlation (ρ or rs) for concurrent validity between the stages of critical thinking and the scores for the assignments. Accepting a type I (alpha) error (two-tailed) of 0.05, a type II error (beta) of 0.2 and an r of 0.7 for strong correlation18, the calculated sample size was 13. Since our population was large enough for a rs of 0.4 for moderate correlation, we proceeded with a calculated sample size of 47.
Statistical Analysis
The content validity was determined by item-objective congruence (IOC), with data presented as means. An IOC value of more than 0.5 was considered acceptable. The concurrent validity between the stages of critical thinking and the assignment scores was calculated by Spearman’s correlation (rs). Scores for the high- and low-performance stages of critical thinking were compared by an independent t-test. The differences in the assignment scores relating to each stage of critical thinking were further analyzed by one-way ANOVA, with post-hoc analysis using Bonferroni’s method. The interrater reliability was calculated using Spearman’s correlation. The strength of correlation (rs) was interpreted as follows: < 0.1, negligible; 0.10–0.39, weak; 0.40–0.69, moderate; 0.70–0.89, strong; and 0.90–1.0, very strong correlation.18 A P-value of < 0.05 was considered statistically significant. The practicality of the tool was analysed with descriptive statistics. All data were analysed using PASW Statistics for Windows (version 18.0; SPSS Inc., Chicago, IL, USA).