Minho Suture Assessment Scale – Development and Validation of a scale using Item Response Theory

doi:10.21203/rs.3.rs-4675874/v1

Download PDF

Research Article

Minho Suture Assessment Scale – Development and Validation of a scale using Item Response Theory

https://doi.org/10.21203/rs.3.rs-4675874/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Purpose:

This study introduces and validates the Minho Suture Assessment Scale (Minho-SAS), a detailed instrument designed to evaluate fundamental suturing competencies among medical students. Through the application of Item Response Theory (IRT) and a dichotomous yes/no response format, the scale aims to offer a comprehensive yet specific assessment framework for suturing abilities.

Methods:

The development process involved collaboration with multidisciplinary surgical teams and experienced practitioners to ensure content validity. Validation methods incorporated IRT techniques, employing the Rasch model and the 2-PL model, to assess reliability, construct validity, and measurement precision. Data from a cohort of medical students were utilized for rigorous psychometric evaluation.

Results:

The Minho-SAS demonstrated strong internal consistency, reliability, and construct validity across both Rasch and 2-PL models. Item analyses revealed a nuanced understanding of individual item performance, highlighting varying levels of difficulty and discrimination among the 15 assessment components. The scale's focus on fundamental suturing skills was evident in its detailed evaluation, contrasting with broader assessment tools.

Conclusion:

The Minho-SAS emerged as a promising instrument specifically tailored for assessing fundamental suturing skills among medical students. Its rigorous validation through IRT methodologies underscored its precision and reliability, distinguishing it as a valuable addition to existing evaluation scales. While demonstrating considerable promise, further exploration across diverse populations and educational settings is crucial to affirm its broader applicability and impact in medical education and clinical practice.

Assessment

Surgery

Suture

IRT

Validation

Medical Education

Suture skill is an essential competence for every young doctor who is expected to be capable of treating autonomously a patient with a regular skin wound. For this reason, this skill must be developed and assessed during undergraduate medical training to ensure that medical students are prepared for this practice immediately after graduation. However, despite most medical educators recognizing this as an essential skill, little is known about how medical schools are assessing suture skills, leading to a problem in acquiring essential competencies. Despite the availability of various formats for assessing technical and surgical competencies in the literature, these tools are targeted to more complex procedures, mostly applicable to surgery residents and not designed for undergraduates. To the best of our knowledge a suture skill assessment scale targeted specifically at medical students is lacking. To address this gap, the authors propose to develop and validate a new scale using Item Response Theory (IRT) as a novel approach to suture scale validation.

The proper assessment of competency is a challenging task. This can be caused in part by the struggle to define and isolate the competency to be evaluated since there are several dimensions for professional competency ¹ and even a hierarchy amongst them ². In the past decades, efforts have been made to progress to competencies-based curricula, such as the creation of CanMEDS ³, Accreditation Council for Graduate Medical Education (ACGME) core competencies (Derstine et al.) or Entrustable Professional Activities ⁵. Epstein et al. explained that competence is contextual, reflecting the relationship between a person’s abilities and the tasks they are required to perform in a particular situation in the real world ⁶. Regarding undergraduate performance assessment, students can be assessed in a simulated environment or a real-life one, and, for both, multiple instruments can be used to assess the students, depending on the competence being measured.

Regarding the surgical undergraduate curriculum, surgical skills are unquestionably one of the core components and one of the most relevant abilities expected and required for a medical graduate. Amongst these surgical skills, the ability to perform an acceptable basic suture that can be autonomously replicated in any scenario should be a requirement for practice. However, recent reports show a concerning trend in the decline of suture competence among recent medical graduates ⁷.

Several suture assessment scales are available in the literature ⁸, mostly directed to laparoscopic/robotic sutures ⁹, microsurgical ^10–12, post-graduate surgical residents ^11,13–15, or even task-oriented ^16–18. There are others that address basic suture skills and are targeted at medical students but include the time to perform as one of the components of competence ¹⁹. The factor of time to perform has been very controversial in the literature since authors believe the expected competency for the undergraduate should be focused on the quality of the technique, not its speed.

Since competence is contextual and developmental, the authors believe assessment must be tailored to the context. However, when the assessed competence is a technique, most assessment methods are designed to be achieved by experts or at least by performers that are expected to be of good quality, not differentiating the progression of expertise ⁹. For this reason, the authors believe there is still space to discuss what can be done when the assessment purpose is to discriminate those who are competent, disregarding the level of expertise, from those who are unable to perform. Additionally, the authors aim to improve the specificity, detail, and standardization of the assessment tool, reducing subjectivity and making it feasible for less experienced assessors to be used in summative exams, reducing the burden and logistic problems for faculty and the medical school.

To properly validate and understand the measurement process, it is crucial to utilize one of the two test theory models available: IRT and classic test theory (CTT). CTT estimates a standard error of measurement for the tested group of students ²⁰. However, CTT has some drawbacks, such as being sample-dependent and usually assuming constant reliability of the entire group, regardless of their performance levels ²⁰. IRT offers an alternative to CTT by estimating a candidate's true ability inferred from their exam score, using item characteristics such as discrimination and difficulty value. The two most common IRT models are the one-parameter logistic (1-PL)/Rasch, which accounts for item difficulty, and the two-parameter logistic (2-PL), which utilizes item difficulty and discrimination. IRT is sample-independent, meaning that the scale can be used for multiple expertise levels and provides an estimate of measurement error at each point along the ability scale. This is especially important when setting a cut score or transitioning from non-competent to competent ²⁰. IRT yields more information about the item throughout the latent scale, ensuring that the scale is equally effective in assessing individuals regardless of their level of expertise, which is of great advantage when creating an assessment tool since it can help to select the most effective items for assessment, it can find problematic or biased items, ensuring better student differentiation, it provides a more accurate estimation of a student’s ability, by considering item difficulty and discrimination, increasing measurement precision. To the best of our knowledge, the use of IRT is of complete novelty for surgical skills assessment.

With this work, the authors intend to address the problem previously exposed by developing and validating a dichotomic suture assessment scale (Minho Suture Assessment Scale - Minho-SAS) that addresses this gap to differentiate competent from not competent undergraduate medical students in the basic suture skills in the summative context, using Item Response Theory as the validation method.

Step 1: Identification of the purpose and domains of the assessment

The authors intended to create an assessment instrument, in Portuguese language, that could distinguish undergraduates who could perform a correct basic suture. The instrument should be user-friendly, and adapted to the context: summative assessment for 5th-year medical students in a simulated scenario. To the best of our knowledge, no other scale with this purpose and context has been validated.

For guidance on the best practices for scale validation, the work from Boateng et al. and the Standards for educational and psychological testing were used ^21,22.

The study was approved by the Ethics Committee for Research in Life and Health Sciences (CEICVS) (CEICVS 103/2022).

Step 2: Minho Suture Assessment Scale Creation

The authors of the study utilized both inductive and deductive approaches simultaneously to generate the items. Fourteen skilled surgeons from various backgrounds, such as general surgery, pediatric surgery, gynecology, orthopedics, and urology, were carefully selected to form a focus group. The group used Delphi methodology to suggest a list of items to be included in the initial Minho-SAS prototype. Each specialist contributed with suggestions and feedback and revised their opinions based on the group’s discussion. In the primary round, the items were submitted for discussion. After removing any duplicates, standardizing the wording, and reaching a consensus on the items, the final draft was revised and approved by the group. The final draft contained 15 yes/no items and a global performance scale with a Likert-type rating ranging from 1 to 5 (Table 1).

Step 3: Scale Testing and Sample Size

The authors followed a 10:1 rule of thumb²³ for determining the sample size, which implies a minimum of 150 assessments. They used the Minho-SAS to evaluate the performance of 269 fifth-year medical students who underwent an Objective Structured Clinical Examination (OSCE) during a surgical clerkship. The assessments were carried out by six experienced medical doctors who were trained in the technique and had ample experience in assessment.

Step 4: Statistical Analysis – Item Response Theory

To perform the validation of our assumptions of the Minho-SAS results, the authors propose that the test score is a reliable approximation of the true ability, meaning that the authors believe the Minho-SAS addresses only one competency, not being influenced by other variables and that scale takers with high grades would be more successful on an advanced course than students with low grades. To verify this unidimensionality, the Q3 correlation matrix was performed to analyze residuals (differences between observed and predicted values). In our study, we also utilized PCA (Principal Component Analysis) as a powerful method to explore the underlying structure of our dataset. By extracting the principal components and examining the component loadings, we aimed to uncover patterns and identify the key variables that contribute to the observed variance. Additionally, the authors plan to integrate the following validity of evidence:

Test Content: To assure content validity, adding to the approval of the previously described focus group, the scale was approved by additional expert medical professionals with experience in suture and psychometrics, assuring scale alignment with the content.

Response Processes: To ensure the fit between the assessment process and the construct, assessors were posteriorly questioned about their thought process, if they were evaluating suture skills, and if they thought they were assessing other competencies. Additionally, assessors were accompanied by surgical experts during the first assessments.

Internal Structure: The study authors utilized Item Response Theory (IRT). IRT measures the item as the unit of analysis and calculates a latent trait (ability) by estimating specific item parameters, such as difficulty (location parameter), discrimination (slope), and item-guessing (when implementing 3-parameter logistics) indices. These indices were also employed for the item reduction analysis. ^22,24–26. For this analysis, the Rasch model and the 2-parameter logistic model (2-PL) were utilized. The Rasch model is a one-parameter logistic model that assumes that the probability of answering an item correctly depends on the ability, which is treated as a latent trait, as well as the item’s difficulty. On the other hand, the 2-PL model includes the discrimination (as the slope of the curve) and the difficulty of the item to assess the ability (theta – θ). Theta has been standardized, following a normal distribution from − 3 to + 3, with 0 representing the average score. ^27,28. The authors used Cronbach’s alpha and McDonald's omega to assess internal consistency, as reliability estimates, of the scale items, in which an alpha coefficient of 0.70 is regarded as an acceptable threshold. ²².

Relations with other variables: To ensure the accuracy of the results, we used an established and validated technical assessment scale called OSATS ²⁹ in combination with Minho-SAS. This approach provided additional validity evidence to Minho-SAS and helped to enhance the validity of the findings. We conducted a correlation analysis and calculated Cohen's d between the test scores of the two assessed years to further validate the results.

Validity of Consequences of Testing: Posteriorly to the analysis of the test scores, the faculty members responsible for the curricular unit blueprint and mapping can modify the curriculum to address possible gaps in students learning, as intended consequences of this exam. In the future, the scale can also be tested for feedback delivery.

For the analysis software, the authors used R version 4.3.1 with the mirt package, JASP ((JASP Team (2023). JASP (Version 0.18.1) [Computer software].) and jamovi (v2.3.28), retrieved from https://www.jamovi.org. The used R script is available on annex 1.

The final Minho-SAS, composed of 15 dichotomous yes/no items and a global performance score ranging from 1 to 5 is exposed in Table 1, with additional information regarding the number of positive assessments in the OSCE.

Table 1

Minho Suture Assessment Scale translated into the English language.
Question	Item	Yes/No	Nº of positive assessments
1	Confirms and prepares the necessary material	Y/N	242
2	Hold the needle correctly with the needle holder	Y/N	251
3	Handles the needle holder correctly	Y/N	259
4	Handles the tissue forceps correctly	Y/N	245
5	Suture at the correct distance from the edge AND at the same distance on both sides	Y/N	221
6	Demonstrates surgical dexterity in the entry and exteriorization and handling of the needle at the exit of the skin	Y/N	181
7	Performs knot, counter-knot and knot correctly.	Y/N	249
8	Correctly cuts the thread with proper length and technique.	Y/N	206
9	Places the knot lateral to the wound	Y/N	199
10	Places the appropriate degree of tension on the knot.	Y/N	209
11	Correct distance between sutures	Y/N	209
12	Performs suturing safely	Y/N	236
13	Safely store suture equipment and needle at the end of the procedure.	Y/N	166
14	Performs the suture with dexterity, economy and fluidity of movements.	Y/N	164
15	Respects the tissues	Y/N	228
Global	Global Assessment Score	1 to 5	-

For the initial validation, the face-validation process was performed by showing the final prototype to experienced surgeons, with expertise on the skill to be assessed, with a unanimous positive judgement.

In order to use Item Response Theory (IRT) and particularly Rasch analysis, the unidimensionality must be confirmed. Using jamovi, we assessed the residuals analysis and the principal component analysis (PCA), displayed on Table 2 and 3, respectively. Values of correlation of coefficient residuals > 0,3 indicate the assumption of independence is compromised, which does not occur in our data, as can be seen in Table 2. Our results from the PCA revealed a single dominant component, as evidenced by the substantial loadings of variables onto this component. This finding aligns with the assumption of unidimensionality indicating that a single factor can explain most of the variance in our dataset.

Table 2

Dichotomous Rasch Model - Residuals Analysis using Q3 Correlation Matrix
Q3 Correlation Matrix
	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10	x11	x12	x13	x14	x15
x1	—
x2	-0.070	—
x3	-0.094	-0.047	—
x4	0.015	0.183	0.101	—
x5	-0.047	-0.006	-0.107	-0.063	—
x6	-0.062	0.017	0.059	0.019	-0.244	—
x7	-0.015	-0.178	0.007	-0.073	-0.141	0.036	—
x8	-0.206	0.002	-0.004	-0.169	-0.002	-0.071	-0.085	—
x9	-0.233	-0.121	-0.001	-0.111	-0.182	-0.114	0.222	-0.106	—
x10	-0.175	-0.005	-0.050	-0.042	-0.002	-0.186	-0.011	-0.088	0.079	—
x11	-0.055	-0.086	-0.040	-0.028	0.165	-0.165	0.002	-0.097	-0.016	-0.081	—
x12	0.072	-0.142	-0.049	0.025	-0.104	0.024	-0.105	-0.147	-0.123	-0.043	-0.169	—
x13	-0.008	0.051	0.055	-0.130	-0.062	-0.211	-0.012	-0.167	0.068	-0.050	-0.157	0.036	—
x14	-0.108	-0.048	0.016	-0.134	-0.074	0.326	-0.034	-0.079	-0.166	-0.130	-0.110	0.028	-0.035	—
x15	-0.022	-0.071	-0.056	-0.079	-0.088	0.069	0.002	0.042	-0.012	0.016	-0.002	-0.034	-0.312	-0.013	—

Table 3

Principal Component Analysis
	Component
	1	Uniqueness
x1		0.961
x2	0.440	0.807
x3	0.490	0.760
x4	0.424	0.820
x5	0.425	0.819
x6	0.628	0.606
x7	0.563	0.683
x8	0.362	0.869
x9	0.546	0.702
x10	0.546	0.702
x11	0.503	0.747
x12	0.456	0.792
x13	0.469	0.780
x14	0.640	0.590
x15	0.575	0.669

Regarding model fit analysis using Item Response Theory (IRT), Tables 4 and 5 resume the findings. It is interesting to observe the values of AIC (Alkaike Information Criterion) and SABIC (Sample-Size Adjusted Bayesian Information Criterion), statistical measures used to compare different models. When comparing, models with lower values indicate better fit. According to Baker et al., item discrimination values of 0.01–0.34 are considered very low; 0.34–0.64 low; 0.65–1.34 moderate; 1.35–1.69 high; and 1.70 and above very high ²⁷.

Table 4

Model Fit Indices for Rasch and 2PL Analyses of the Minho-SAS. AIC: Akaike Information Criterion; SABIC: Sample-Size Adjusted Bayesian Information Criterion; HQ: Hannan-Quinn Information Criterion; BIC: Bayesian Information Criterion; Df: Degrees of freedom
	AIC	SABIC	HQ	BIC	logLik	X2	df	p
Rasch	3282.212	3288.997	3305.310	3339.727	-1625.106	NA	NA	NA
2PL	3261.493	3274.215	3304.802	3369.334	-1600.746	48.7191	14	< .0001

Analyzing Table 4, which translates the assessment of the relative fit of the two models, the 2-PL analysis outperforms the Rasch model, with lower AIC, SABIC, HQ, and BIC values, which means the 2PL model is a significantly better fit, with better trade-of and likelihood of better predictions, adding with the chi-square value with a significant p-value. The findings of both models support the validity, based on their internal structure.

Table 5

Difficulty and Discrimination Parameters for Each Item in Rasch and 2-PL Models
	Rasch model	2-PL model
	Difficulty	Difficulty	Discrimination
1	2.8	2.301	0.525
2	3.32	3.37	1.38
3	4.02	5.08	2.16
4	2.95	2.8	1.14
5	1.98	1.768	0.919
6	0.948	1.34	2.52
7	3.18	3.92	2.06
8	1.55	1.327	0.774
9	1.37	1.34	1.25
10	1.63	1.58	1.23
11	1.63	1.52	1.08
12	2.52	2.51	1.31
13	0.628	0.573	1.013
14	2.21	0.925	2.935
15	1.75	2.47	1.7

Table 5 reveals interesting insights. In the Rasch model, item difficulty parameters range from 0.628 (item 13) to 4.02 (item 3), reflecting a wide distribution, suggesting that the Minho-SAS possesses items with progressive difficulty levels, which can be useful if we intend to assess different levels of proficiency. In the 2-PL analysis, the item difficulty values range from 0.573 (item 13) to 5.08 (item 3), corroborating Rasch's analysis. Interestingly, there is a shift in the difficulty of item 14 from Rasch to 2-PL analysis, which could indicate that the item is very challenging. As for discrimination parameters, ranging from 0.525 (item 1) to 2.935 (item 14), items with higher values are more effective in distinguishing suture skills and contribute more to the Minho-SAS accuracy. In this case, we have a broad spectrum of low to very discrimination values, according to current literature ³⁰.

The Item Characteristic Curves (ICCs) for each item as estimated by the Rasch Model and the 2-PL are depicted in Figs. 1 and 3, respectively. Each curve illustrates the probability of a correct response to the item at different levels of the latent trait. The x-axis represents the latent trait continuum, indicating increasing levels of proficiency, while the y-axis shows the probability of endorsing the item. In these curves, it can be observed that, given a student whose value of Θ (theta) = 0, which would be an average student, the probability of having the item correctly would be 0.6 in item 14 (v14). When the probability is -2 Θ the value is almost 0.2 and when Θ is + 4 the probability would be almost 1. The Item Information Curves as estimated by the Rasch Model and the 2-PL are depicted in Figs. 2 and 4, respectively.

In ICC curves, item 3 exhibits a tall and narrow curve, indicating that it is very precise in measuring abilities, while item 6 has a broader ICC, suggesting that it provides information across a more comprehensive range of skills. Items with higher peaks are well suited for discriminating between individuals with similar abilities. As for items with steeper curves, as item 3, exhibit higher discrimination values, being effective in separating students for the suture skill level. Items with shallower curves, like item 5, have lesser discrimination values. The peak of the ICC represents the item's difficulty, item 2 peak is at a lower ability level indicating that it is an easy item while item 4 is at a higher ability level indicating a more challenging item. ICC curves the probability of students with varying skills having a positive item assessment: item 6 ICC curve shows a higher probability of success for students with higher skill levels, reflecting a more challenging item. In contrast, the item 15 curve suggests a less challenging item.

As for the IIC, it can be observed that the amount of information obtained from the Minho-SAS has its peak around Θ values of -2 at the Rasch Analysis and Θ = 0 at the 2-PL analysis, decreasing information when going out of this range of ability. This means that in the Rasch analysis, the Minho-SAS acquires the most informative measure for students with lower suture skill proficiency. The 2-PL analysis captures more detail in students with more average ability.

In the context of Rasch Analysis, item fit statistics, as displayed in Table 6, indicate how accurately the data fit the model. Outfit could be understood as a measure of how well an item matches a pattern of responses expected by the Rasch model. If the Outfit value is too high or too low, it could indicate that the item might be too difficult or too easy when compared with the prediction. Infit focuses on the consistency of responses, being more sensitive to the pattern of responses. If the value of infit is too high or too low, it should raise awareness that the item could be confusing or not aligned with the others. Fit values should be between 0,5 to 1,5 ³¹. Items with infit or outfit > 2.0 should be excluded ^32,33. In our work, we observed that the outfit statistics range from 0.479 to 1.253, which indicates acceptable variability in the behavior of the items. Similarly, the infit statistics range from 0.954 to 1.247, suggesting good levels of response consistency. The mean values for outfit (0.782) and infit (0.971) indicate an overall alignment of items with the Rasch model, though with some variability in their fit.

Table 6

– Item Fit Statistics for Rasch Model
	outfit	infit
Minimum	0,479	0,954
Maximum	1,253	1,247
Mean	0.782	0.971
Standard Deviation	0,195	0,118

The scale demonstrated good internal consistency, as evidenced by McDonald's ω estimate of 0.776 (95% CI [0.736, 0.815]) and Cronbach's α of 0.765 (95% CI [0.723, 0.803]), seen in Table 7. These values suggest a robust level of reliability, indicating that the items comprising the scale reliably measure the intended construct. The reliability coefficient obtained for the Rasch model (0.71) and 2-PL model (0.74) reflects an acceptable range for assessment ³⁰.

Table 7

Reliability Statistics for Minho-SAS
Estimate	McDonald's ω	Cronbach's α
Point estimate	0.776	0.765
95% CI lower bound	0.736	0.723
95% CI upper bound	0.815	0.803

The assessment of reliability coefficients yielded values of 0.71 for the Rasch Model and 0.74 for the 2-PL Model. These coefficients provide insights into the internal consistency of the respective models in measuring the latent trait.

Figures 5 and 6 reveal the patterns concerning the reliability of the Minho-SAS across varying ability models, in the Rasch model (Fig. 5) and the 2-PL model (Fig. 6).

The conditional reliability plots obtained from both Rasch and 2PL analyses (Figs. 5 and 6, respectively) give insights into the measurement precision of Minho-SAS across varying ability levels. In the Rasch model, the reliability is moderate level (around 0.4) for individuals with extremely low abilities (around − 6 Θ). As abilities progress towards − 3 to -0.5 Θ, the reliability significantly increases, stabilizing at a high plateau of around 0.8. This indicates that the Rasch model is robust in measuring abilities consistently within this range. However, as abilities increase beyond this range, there is a gradual but expected decrease in reliability, reflecting a natural attenuation in precision for individuals at higher proficiency levels. On the other hand, the 2-PL model demonstrates a notably improved trend but starts with lower reliability (less than 0.2) for those with the lowest abilities (-6 Θ). As abilities increase, the reliability swiftly improves, exceeding 0.8 around − 2 Θ, and maintains this high precision up to Θ = 0. This shows that the 2-PL model consistently and accurately measures abilities at these proficiency levels. Nonetheless, as abilities extend beyond Θ = 0 and reach higher levels (4 Θ), the reliability slowly decreases, indicating some challenges in accurately estimating abilities at these extreme levels. These distinct reliability trends across ability levels highlight the validation mechanisms' effectiveness in ensuring precise measurement and offer valuable guidance for future refinements in assessment methodologies.

As for the correlation between the latent traits (theta), which represents the student ability, and the sum of the items from the Minho-SAS, there was an extremely high correlation level of 0,96 from 2PL and 0,982 from the Rasch model, as can be seen on Table 8.

Table 8

Correlation Matrix
		Rasch	2PL	Sum of MinhoSAS	OSATS	MinhoSAS Global Score
Rasch	R Pearson	—
	gl	—
	p-value	—
2PL	R Pearson	0.975	—
	gl	267	—
	p-value	< .001	—
Sum of MinhoSAS	R Pearson	0.982	0.960	—
	gl	267	267	—
	p-value	< .001	< .001	—
OSATS	R Pearson	0.670	0.698	0.656	—
	gl	135	135	135	—
	p-value	< .001	< .001	< .001	—
MinhoSAS Global Score	R Pearson	0.800	0.811	0.793	0.931	—
	gl	267	267	267	135	—
	p-value	< .001	< .001	< .001	< .001	—

Regarding the correlation matrix, it provides a strong positive correlation (r = 0.975, p < 0.001) between the Rasch and 2PL models, indicating a high degree of agreement between the two models in measuring the latent trait (suture skill). The also high correlation (r = 0.982, p < 0.001 and r = 0.960, p < 0.001) between the two models and the Sum of Minho-SAS scores, indicates that students with higher Rasch and 2PL ability values tend to receive higher total scores, which corroborates the construct validity of the scale. While comparing with an existing validated scale, the OSATS, both Rasch (r = 0,670, p < 0.001) and 2-PL (r = 0.698, p < 0.001) have good positive correlations, which also adds to the validation of the scale. However, the values are not so high, and since Minho-SAS has more items, it could mean that it could be assessing more aspects of the skill. As for the global score, the strong associations also validate the use of the scale.

When searching for differences between the two assessed years, we conducted an independent samples t-test and a search for effect size using Cohen’s d coefficient, displayed in Table 9.

Table 9

Effect Sizes (Cohen's d) for Minho-SAS
		Statistics		gl	p		Cohen’s d
MinhoSAS Global Score	Studen’s t	3.64	ᵃ	267	< .001	d de Cohen	0.443
Sum of MinhoSAS	Studen’s t	2.80		267	0.005	d de Cohen	0.342
2PL	Studen’s t	2.93		267	0.004	d de Cohen	0.357
Rasch	Studen’s t	2.98		267	0.003	d de Cohen	0.363

The comparison between the assessed years, seen in Table 9, revealing no significant difference in the Minho-SAS scores suggests that the scale demonstrates consistency across different cohorts of students. The observed effect sizes (Cohen's d) of 0.34 to 0.44, representing small to moderate effects, support this finding. Interestingly, there is a higher effect size on the global scale scores than the latent scores.

As mentioned earlier, suture skills are necessary for all medical graduates as they are a part of autonomous practice post-graduation. Therefore, it is essential to assess this skill properly. However, many medical schools teach suture skills and need to evaluate them. Despite the availability of several scales for technical skills, including surgical skills, there was still a need for a scale that objectively assesses basic suture skills for medical students while also serving as a possible feedback mechanism in the future. A multidisciplinary surgical team discussed the topics integrated into the Minho-SAS to avoid bias towards any specific surgical areas. The final prototype was shown to surgical experts, who agreed with the proposal. The next step was determining whether the scale should be assessed on a Likert scale or a dichotomous yes/no approach. Current literature suggests that checklist scales can be used for assessing competencies without loss of measurement precision when compared to overall rating scales, especially when referring to specific competencies, such as suture, and can even have advantages such as improving the quality of feedback and more user-friendly ^34,35. Since the focus group determined that this competency should be a required one to complete graduation, a yes/no approach was preferred, similar to other required skills in graduation, such as basic life support.

Regarding validation methods, since the Minho-SAS is a dichotomous scale, conventional validation techniques such as exploratory and confirmatory factor analysis are not optimal. Item Response Theory offers critical advantages for scale validation, as it models the relationship between an individual's latent trait (in this case, suture skill) and their response to the individual items of Minho-SAS. This contrasts with classical test theory, which considers all items as equivalents, not considering item difficulty and discrimination. For this investigation, we employed two IRT models, the Rasch model and the 2-PL, which are well-suited for validating dichotomous assessment scales. The Rasch model is a one-parameter model that assumes unidimensionality and focuses on a single latent trait, not influenced by other factors. Each item in this model has a difficulty parameter (b) that represents the point on the ability continuum at which individuals have a 50% probability of having a positive assessment. The 2-PL model is an extension of one-parameter models, having a discrimination (a) parameter, which allows quantifying the item's ability to discriminate between students with different ability levels.

The next step was to perform several analyses of the items to unfold which items better measure the suture skill and if the scale is more appropriate to assess students with lower or higher proficiency. Items should ideally cover a range of difficulty levels to increase the measuring efficacy of the scale and should have higher discrimination parameters since they are more effective in distinguishing between high and low performers. Our results confirm the wide range of difficulty and high discrimination parameters throughout the items in Minho-SAS.

The Minho-SAS stands as a promising addition to the landscape of suture skill assessment scales, offering different views over existing ones. While other scales such as the OSATS, the most commonly used technical assessment scale, assess surgical skills comprehensively, they often lack specificity in evaluating basic suture skills crucial for medical students ⁷. Unlike these broader assessment tools, the Minho-SAS focuses explicitly on the foundational aspects of suturing, offering a detailed and granular evaluation framework. Its dichotomous yes/no response format aligns with the need for clear-cut evaluations of essential skills. Additionally, the authors believe this granularity could enhance curricula deficiency detection and motivate changes in the following classes to tackle those deficits. Since there is a high correlation between Minho-SAS and OSATS, the increased number of items in the Minho-SAS demonstrated not being a prejudice in the assessment reliably. As previously highlighted, other established and validated scales, such as those proposed by Sundhagen et al., incorporate time as one of the parameters of evaluation. However, we argue that it could be misleading in the correct assessment of the student’s true ability ¹⁹.

The extensive validation of the Minho-SAS emphasizes its potential as a versatile tool applicable across diverse contexts. Despite the thorough validation process, this study has certain limitations that warrant acknowledgment. The study's reliance on a specific cohort or limited sample size might constrain the generalizability of findings to a broader population. Additionally, this study was conducted as a convenience sample and its validation remains context-specific, potentially influencing the representation of diverse learner populations, necessitating caution in extrapolating results beyond the study's confines.

Adding all, it suggests the Minho-SAS is valid and suitable for assessing suture skills.

Our study extensively examined the scale's internal consistency, precision, and performance across various dimensions of evaluation. This thorough validation underscores its potential to serve as a reliable and accurate assessment mechanism. The Minho-SAS addresses a critical gap in evaluating basic suture skills for medical students, offering a standardized and objective approach. The integration of IRT, by providing helpful information into an item’s performance, within the validation process of the Minho-SAS has fortified its standing as a reliable and precise instrument for assessing suture skills.

In conclusion, the amalgamation of IRT methodology in validating the Minho-SAS illuminates its potential as a reliable assessment tool for suture skills. Yet, further research, especially through multicentric studies and with larger, more diverse samples, could increase its validation through invariance measurements that would bring more evidence to the internal structure and its universal applicability. Recognizing the limitations, this study lays a foundation for future endeavors aimed at refining and extending the utility of the Minho-SAS in diverse educational and clinical settings. This validated tool holds promise for enhancing medical education by providing an objective means of evaluating and improving suture skills, contributing to the overall competency and proficiency of future healthcare professionals.

Ethics Approval: Approved. File in annex.

Acknowledgments: Not applicable

Consent For Publication: Not applicable

Availability of data and material: All data is present in this document.

Competing Interests: None to Report

Funding: No direct funding was used for this work.

Authors Contribution: All authors contributed for the work. NG, MBA, VHP and JMP were responsible for the project's design. NG, RS and CC were responsible for the statistical analysis. All authors revise the document.

Epstein RM. Defining and Assessing Professional Competence. JAMA. 2002;287(2):226. doi:10.1001/jama.287.2.226
Miller GE. The assessment of clinical skills/competence/performance. Acad Med J Assoc Am Med Coll. 1990;65(9 Suppl):S63-67. doi:10.1097/00001888-199009000-00045
Frank J. A history of CanMEDS - chapter from Royal College of Physicians of Canada 75th Anniversary history. In: ; 2004.
Derstine P, Leach DC, Teherani A, Reich LM. Improvement Through the Application of the Competencies.
ten Cate O. Entrustability of professional activities and competency-based training. Med Educ. 2005;39(12):1176-1177. doi:10.1111/j.1365-2929.2005.02341.x
Epstein RM. Assessment in Medical Education. Cox M, Irby DM, eds. N Engl J Med. 2007;356(4):387-396. doi:10.1056/NEJMra054784
Emmanuel T, Nicolaides M, Theodoulou I, Yoong W, Lymperopoulos N, Sideris M. Suturing Skills for Medical Students: A Systematic Review. In Vivo. 2021;35(1):1-12. doi:10.21873/invivo.12226
Vaidya A, Aydin A, Ridgley J, Raison N, Dasgupta P, Ahmed K. Current Status of Technical Skills Assessment Tools in Surgery: A Systematic Review. J Surg Res. 2020;246:342-378. doi:10.1016/j.jss.2019.09.006
Vassiliou MC, Feldman LS, Andrew CG, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190(1):107-113. doi:10.1016/j.amjsurg.2005.04.004
Almeland SK, Lindford A, Sundhagen HP, et al. The effect of microsurgical training on novice medical students’ basic surgical skills—a randomized controlled trial. Eur J Plast Surg. 2020;43(4):459-466. doi:10.1007/s00238-019-01615-w
Dormegny L, Neumann N, Lejay A, et al. Multiple metrics assessment method for a reliable evaluation of corneal suturing skills. Sci Rep. 2023;13(1):2920. doi:10.1038/s41598-023-29555-3
Nugent E, Joyce C, Perez-Abadia G, et al. Factors influencing microsurgical skill acquisition during a dedicated training course. Microsurgery. 2012;32(8):649-656. doi:10.1002/micr.22047
Chipman JG, Schmitz CC. Using Objective Structured Assessment of Technical Skills to Evaluate a Basic Skills Simulation Curriculum for First-Year Surgical Residents. J Am Coll Surg. 2009;209(3):364. doi:10.1016/j.jamcollsurg.2009.05.005
Goova MT, Hollett LA, Tesfay ST, et al. Implementation, construct validity, and benefit of a proficiency-based knot-tying and suturing curriculum. J Surg Educ. 2008;65(4):309-315. doi:10.1016/j.jsurg.2008.04.004
Scott DJ, Goova MT, Tesfay ST. A cost-effective proficiency-based knot-tying and suturing curriculum for residency programs. J Surg Res. 2007;141(1):7-15. doi:10.1016/j.jss.2007.02.043
Buckley CE, Kavanagh DO, Gallagher TK, Conroy RM, Traynor OJ, Neary PC. Does Aptitude Influence the Rate at which Proficiency Is Achieved for Laparoscopic Appendectomy? J Am Coll Surg. 2013;217(6):1020-1027. doi:10.1016/j.jamcollsurg.2013.07.405
Nickel F, Brzoska JA, Gondan M, et al. Virtual reality training versus blended learning of laparoscopic cholecystectomy: a randomized controlled trial with laparoscopic novices. Medicine (Baltimore). 2015;94(20):e764. doi:10.1097/MD.0000000000000764
Pape-Koehler C, Immenroth M, Sauerland S, et al. Multimedia-based training on Internet platforms improves surgical performance: a randomized controlled trial. Surg Endosc. 2013;27(5):1737-1747. doi:10.1007/s00464-012-2672-y
Sundhagen HP, Almeland SK, Hansson E. Development and validation of a new assessment tool for suturing skills in medical students. Eur J Plast Surg. 2018;41(2):207-216. doi:10.1007/s00238-017-1378-8
De Champlain AF. A primer on classical test theory and item response theory for assessments in medical education. Med Educ. 2010;44(1):109-117. doi:10.1111/j.1365-2923.2009.03425.x
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. American Educational Research Association; 2014.
Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Front Public Health. 2018;6. Accessed August 24, 2023. https://www.frontiersin.org/articles/10.3389/fpubh.2018.00149
Goodhue DL, Lewis W, Thompson R. Does PLS Have Advantages for Small Sample Size or Non-Normal Data? MIS Q. 2012;36(3):981-1001. doi:10.2307/41703490
Downing SM. Item response theory: applications of modern test theory in medical education. Med Educ. 2003;37(8):739-745. doi:10.1046/j.1365-2923.2003.01587.x
YANG FM, KAO ST. Item response theory for measurement validity. Shanghai Arch Psychiatry. 2014;26(3):171-177. doi:10.3969/j.issn.1002-0829.2014.03.010
Zanon C, Hutz CS, Yoo H (Henry), Hambleton RK. An application of item response theory to psychological test development. Psicol Reflex E Crítica. 2016;29(1):18. doi:10.1186/s41155-016-0040-x
Baker FB. The Basics of Item Response Theory. Second Edition. For full text: http://ericae; 2001. Accessed October 23, 2023. https://eric.ed.gov/?id=ED458219
Nima AA, Cloninger KM, Persson BN, Sikström S, Garcia D. Validation of Subjective Well-Being Measures Using Item Response Theory. Front Psychol. 2020;10:3036. doi:10.3389/fpsyg.2019.03036
Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273-278. doi:10.1046/j.1365-2168.1997.02502.x
Walsh A, Cao R, Wong D, et al. Using item response theory (IRT) to improve the efficiency of the Simple Clinical Colitis Activity Index (SCCAI) for patients with ulcerative colitis. BMC Gastroenterol. 2021;21(1):132. doi:10.1186/s12876-021-01621-y
What do Infit and Outfit, Mean-square and Standardized mean? Accessed November 16, 2023. https://www.rasch.org/rmt/rmt162f.htm
Cecilio-Fernandes D, Medema H, Collares CF, Schuwirth L, Cohen-Schotanus J, Tio RA. Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis. BMC Med Educ. 2017;17:192. doi:10.1186/s12909-017-1051-8
Dichotomous Mean-Square Fit Statistics. Accessed November 16, 2023. https://www.rasch.org/rmt/rmt82a.htm
Wood TJ, Pugh D. Are rating scales really better than checklists for measuring increasing levels of expertise? Med Teach. 2020;42(1):46-51. doi:10.1080/0142159X.2019.1652260
Ilgen JS, Ma IWY, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 2015;49(2):161-173. doi:10.1111/medu.12621

No competing interests reported.

Download PDF

Editorial decision: Revision requested
08 Jul, 2024
Editor assigned by journal
08 Jul, 2024
Submission checks completed at journal
08 Jul, 2024
First submitted to journal
02 Jul, 2024

You are reading this latest preprint version

Minho Suture Assessment Scale – Development and Validation of a scale using Item Response Theory

Status:

Version 1

Abstract

Purpose:

Methods:

Results:

Conclusion:

Figures

Introduction

Material and Methods

Step 1: Identification of the purpose and domains of the assessment

Step 2: Minho Suture Assessment Scale Creation

Step 3: Scale Testing and Sample Size

Step 4: Statistical Analysis – Item Response Theory

Results

Discussion

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1