Team-Based Learning Student Assessment Instrument (TBL-SAI) for measuring the effectiveness of TBL in a Saudi Medical School: Psychometric Analysis and differences by academic years

Background Increasing the use of Team Based Learning (TBL) in health profession education reinforce the need to develop a proper instrument for measuring the applicability of this method. This study aimed to examine the psychometric properties of TBL-SAI and the mean score of instrument subscales by the different academic year of the students. Methods Across-sectional study was conducted at the University of Bisha, College of Medicine (UBCOM), Saudi Arabia. Medical students from second to fourth were included in the study. Participants were completed the TBL-SAI items to measure three subscales of accountability, preference for a lecture or TBL and satisfaction. Cronbach’s alpha, factor analysis, were checked the reliability and validity of the instrument. A principal component analysis (PCA) with varimax rotation was conducted on each subscale. ANOVA analyzed the TBL effectiveness related to the different years of medical school. Results Cronbach’s alpha was 0.798 and factor loading was greater than 0.40 for all the items, indicating the reliability and validity of the scale. In a PCA, accountability items generated two factors with loading >0.40, except items one and four. All preference and satisfaction items have factors loading > 0.40. Fourth-year students’ obtained signicant highest mean scores for accountability (p=0.0.49), preferences (p=0.001) and satisfaction (p<0.001) compared to third and second years students. Conclusions TBL-SAI is a sound tool to measure the favor of TBL among medical students. Longitudinal studies are recommended to bring a clear picture of the effectiveness of TBL in UBCOM. (year ve into TBL, problem-based students contents of modules


Background
Team-based learning (TBL) has become an increasingly important educational method of a medical curriculum in the last few years [1,2]. It was rst introduced by Larry Michaelsen in 1970 within the eld of business education [3,4]. Later, TBL found to be suitable for health profession education because of its relevancy as a student-centered learning philosophy [4]. The primary principles of TBL are to improve students' skills on problem-solving, critical thinking, communication and teamwork by employing multiple groups in a large class setting [5,6]. Well-known that TBL is an ideal education strategy stress higher-order thinking and help in applying scienti c contents in further clinical environments [7,8]. Performing TBL strategy needs three speci c steps, including advance homework before their class, readiness assurance tests to assess understanding of the pre-class materials followed by application of learning materials in a clinicalcontext scenario [2,9].
Considerable research has assessed the value and effectiveness of TBL as a pedagogical approach [8,10,11]. According to Taglieri et al. although TBL is thought to improve student engagement and mastery of material, after initial implementation of TBL, knowledge retention in the long-term appears to be lower than lecture-based learning [12]. In a randomized crossover comparison study of TBL and lecture format, Bleske et al. found that learning outcomes and student con dence in performing higher-order tasks were signi cantly higher with TBL [11]. On the other hand, a study found that blending of TBL and lecture-based learning strategy lead to better learning outcomes than either method alone in neurology clerkships [9]. Increasing the numbers of research on TBL pedagogy with different opinions reinforce the need for developing a sound instrument for measuring the effectiveness of this learning approach [13]. So far, there are limited validated instruments to assess students opinion about major areas of TBL approach [13,14] The team-based learning student assessment instrument (TBL-SAI) has been developed for measuring accountability, learning preference and students' satisfaction with TBL [15]. The psychometric properties of TBL-SAI have been con rmed in several health profession educations across different countries [4,[13][14][15][16].
The medical program at the University of Bisha, College of Medicine (UBCOM) is a hybrid PBL approach blended with a minimum number of traditional lectures and different types of student-centered activities as well as TBL. Both of TBL and lecture are part of the teaching method across all modules of basic sciences, preclinical and clinical phases [17]. Some medical schools in Saudi Arabia use TBL strategy in their undergraduate curricula [17][18][19]. However, the applicability of TBLI-SAI has not yet been examined among medical school in the country. Currently, studies about the effectiveness of TBL in Saudi Arabia remains limited and many dimensions of TBL did not study. For instances, a study conducted at the College of Medicine at Alfaisal University to assess rst-year medical students' perceptions about TBL on three major domains of reaction, learning, and behavior based on Kirkpatrick's theory [20]. Another study found that TBL strategy increased students responsibility for self-learning, and improved their academic performance in the neurosciences block [18].
The present study was designed to examine the validity and reliability of TBL-SAI in a sample of medical students at UBCOM. The study also aimed to assess TBL attitudes in terms of accountability, preference, and satisfaction among second to third years medical students.

Study design and setting
Across-sectional study was conducted among medical students at UBCOM, Bisha, Saudi Arabia, which admits about 40 ± 5 medical students each year to a six-year MBBS program. Medical curriculum in UBCOM arranged into modules that extend through the three phases of basic medical science (year two and three), pre-clerkship (year four) and clerkship (year ve and six). Each academic year consists of two semesters that divided into course modules. TBL, problem-based learning, lectures and other types of students centered activities offered to deliver the contents of modules [21].

Study subjects
Medical students at second, third and fourth years who matriculated for the academic year 2017-2018 were included in the study.
Participation in the study was voluntary and approved by the ethics and research committee at UBCOM.

Conduction of TBL in UBCOM
UBCOM offers the TBL method during the three phases of the curriculum that arranged into modules that extend from two to ten weeks.
One TBL session introduced every week, and the number of the sessions depends on the module duration. The session carried out through three essential steps of pre-class preparation, redness and assurance and application [17]. First, before the class session, the students completed a self-directed study of a selected learning material guided by speci c learning objectives of the topic. For this rst step, a two-to-four hour scheduled in a weekly course timetable and assigned for self-directed-learning for the TBL preparation. Second, 100 minutes class session conducted in the classroom where the students assigned to a team comprised of 7 to 10 members. Students remain in their assigned teams during the course to become more effective in helping members master and apply course content. A content expert joined as a facilitator to directs the discussions and provides educational expertise for the students. Ten multiple-choice questions (MCQs) for 15 minutes taken by each student as an individual readiness assurance test (iRAT) at the beginning of the session. Then, immediately the same ten MCQs repeated by students as a group (group readiness assurance test; gRAT) for 30 minutes, with the intent of promoting discussion to prove group consensus on one answer for each question. MCQs and expected answers were reviewed in the classroom so the facilitator can correct any wrong assumptions and con rm the important learning concepts. The primary purpose of iRAT and gRAT were to measure the level of students readiness for the class session. Third, students work into their groups to apply the gained concepts through solves a challenging clinical scenario or case problems created in MCQs format. The scores existed from iRAT, gRAT and application exercise contributed in students course grade. Such scoring has been applied to increase students attendance and participation in the class.

Instrument description
The permission to use TBL-SAI was granted by Heidi Mennenga. The TBL-SAI scale originally developed for assessing TBL in baccalaureate nursing students [15] and was found to be reliable among medical students [4,16] and other health profession students [13,14]. The psychometric properties of the TBL-SAI scale have been reported as satisfactory, and the construct validity of the scale has been examined previously elsewhere [13,15]. The instrument is a self-administrated contains 33 items (with total scores ranging from 33-165) designed to measure students' attitude toward TBL. It composed of three subscales to measure TBL accountability (items 1 to 8, with scores ranging from 8-40), preference for lecture or TBL (items 9 to 24 with scores ranging from 16-80) and students satisfaction (items 25 to 33 with scores ranging 9-45). Positive perception about the use of TBL pointed out by score of > 99 for the total scale, > 24 for accountability subscale, > 48 for preference for TBL over lecture subscale and > 27 for students satisfaction subscale [15].

Sampling procedure
The purpose of the study was explained to the participants before the data sampling. Participating in the study was anonymous and voluntary, with no in uence on the progress of their educations was clearly stated for the students. The TBL-SAI items translated into Arabic for a better understanding of di cult items and printed copies containing both English and Arabic delivered to the students. The questionnaire forms fully completed by the target medical students during class time and in the presence of the principal investigator to clarify any queries or doubts. Whereas students absent during the survey were excluded from the study. The students asked to answer on a ve-point Likert scale which scored from 1, 2, 3, 4, 5 as disagree, disagree, neutral, agree, or strongly agree, respectively. Among the 33 items, 10 negatively worded items in the scale were reverse scored by assigning 5, 4, 3, 2, or 1 [15].

Statistical analysis
Data entry and analysis were subjected to ISBN SPSS version 20 with 0.05 as a signi cant value. Simple descriptive statistics were used to generate the result data in percentages, mean, and standard deviations. Signi cant correlations between variables were measured using Bartlett's test of sphericity. Kaiser-Meyer-Olkin test (KMO) was performed for the whole instrument items for measuring sampling adequacy. Values higher than 0.60 for the total items indicate that factor analysis can be conducted [13]. A principal component analysis with varimax rotation was conducted on each subscale of TBL-SAI to identify the underlying factor structure. Extraction of factors was determined by consideration of eigenvalues greater than 1 and examination of the scree plot. Items with factor-loading values greater than 0.40 were valid within the instrument and required for interpreting the factor structure [13,15]. Internal consistency and reliability of the instrument were analyzed using Cronbach's alpha, with a greater value of 0.70 and higher is acceptable [15]. Oneway analyses of variance (ANOVA) was analyzed the differences in TBL effectiveness related to the different years of medical school.

Demographic characteristics
One hundred twenty-three male Saudi medical students from year two, three and four were involved in the study. Of the total number of students, 90.1% (109) (Table   1).

Preference subscale
The Kaiser-Meyer-Olkin Measure of Sampling Adequacy yielded an index of 0.719, and Bartlett's test of sphericity gave Chi-Square (120) = 511.711; p<0.001). Four factors were generated from the subscales TBL preference that accounted for 57.9% of the total variances. Factor 1, and 3 consisted of 8 items labeled as TBL. Factor 2 and 4 consisted of six items labeled as a lecture (Table 2).

Satisfaction subscale
The Kaiser-Meyer-Olkin Measure of Sampling Adequacy yielded an index of 0.832, and Bartlett's test of sphericity gave Chi-Square (36) = 379.891; p<0.001). Two factors were generated the subscales satisfaction that accounted for 59.5% of the total variances ( Table 3). Table 4 summarizes the total instrument score and three subscales scores for the students from different academic levels. The mean total score for TBL-SAI was 111.6 (SD=15.4). The mean score for the entire subscale was 28.4 (SD=5.5) for accountability, 51.7 (SD=7.3) for TBL preference and 31.5 (SD=6.3) for satisfaction. Fourth-year students' obtained signi cant highest mean scores for accountability (p=0.0.49), preferences (p=0.001) and satisfaction (p<0.001) compared with the students in year three and year two.

Discussion
The purpose of this study was to describe the psychometric properties of TBL-SAI and its implementation in a medical school. To best of our knowledge, this was the rst study examining attitudes of medical students toward TBL in Saudi Arabia using TBL-SAI. The internal consistency of the total scales was 0.798, and all items showed a factor loading of greater than 0.4. These results support the validity and reliability of the instrument to measure the effectiveness of TBL pedagogy in medical students in Saudi Arabian medical schools context. Other authors have examined the appropriateness of the TBL-SAI scale and found to be valid [13,16,22]. Overall, the mean score for TBL-SAI in this study was favorable (111.6), when compared with the reference values proved by Mennenga [15]. However, scores obtained from this study was lower than 159.68 rated by medical students [4], 117.9 rated by pharmacy students [13] and 115.6 to 125.3 rated by Physical therapy students [14]. A possible reason for our ndings could be attributed to the short experience of TBL since UBCOM was established in 2014. Thus, long terms of experience with TBL may perhaps increase students respect to such educational method.
Principal analysis of accountability items in this study resulted in two underlying factors concern students' contribution to the team and well preparation of learning topic before TBL session. Students' accountability displayed by the commitment to the attendance of TBL sessions, class preparation, increased motivation to class preparation and effective participation in their groups. In this study, medical students had high scores toward accountability subscale. A previous study supporting the power of TBL-SAI instrument to measure students accountability in Iran [4] and the United State [14]. It has widely accepted the keys aspects for effective TBL accountability should include an emphasis on improving students' pre-reading, self-direct learning and teamwork skill [16,23,24]. Interestingly, our results showed that items 1 ("I spend time studying before class in order to be more prepared") and 4 ("My contribution to the team is not important") yielded low factor loading (less than 0.4). However, a decreased factor loading value for item 4 has been found in a study carried out among Pharmacy students in the United Kingdom [13].
The preference for TBL or lecture methods indicates the student's ability to recall material and their degree of attention with one learning methods [14]. Our result showed that all the items of subscale assessing students' preference for TBL or lecture attained loadings of higher than 0.40. However, four factors were extracted and labeled as "TBL; factor 2 and 4" and "Lecture; factor 1 and 3". Several studies compared the learning outcomes of TBL to the lecture, and most of them found that TBL yielded better academic results [4,[25][26][27]. In a study comparing learning outcomes and student con dence between TBL and lecture, Bleske et al. found that student con dence in performing higher-order tasks was signi cantly higher with TBL, and a majority of students preferred TBL pedagogy to lecture format [11]. However, other authors reported a neutral score of students regarding preference of TBL against traditional lecture [22]. Several factors could in uence the students' opinion between TBL or lecture, such as the timing of TBL introduction, the subject matter to be taught by TBL, experience of facilitators and module contents [22]. Therefore, proper organization of course contents, ground conditions in the classroom and appropriate selection of the learning materials for the TBL sessions can lead to a positive attitude toward this instructional strategy.
Since UBCOM is using TBL as instructional method during the three phases of the medical curriculum, more research should be carried out to understand the long-term impact of TBL across the different academic years. In our samples, two factors were extracted on the student satisfaction subscale, indicating that our students have positive feelings toward TBL activities. Moreover, students satisfaction with TBL strategy have been documented in the literature [16,28] It has been noted that the quality and effectiveness of the learning process were associated with TBL satisfaction. According to Roh et al. (2014), ve instructional design factors including, learning process, pre-assignment, course content, peer evaluation, and team activity -signi cantly in uenced learner satisfaction, with learning process exerting the largest in uence [23].
Comparing TBL attitudes among medical students, there were signi cant differences in students mean responses according to the academic level. Generally, fourth-year students obtained a higher mean score for the three subscales compared with second and thirdyear students. This t with the fact that long-term use of TBL in the education system improves the impact of this assay [16]. Contrary to expectations, third-year students achieved lower insigni cant mean score compared to the second year students. This does not re ect the theory of that TBL becomes an enjoyable and valuable learning tool, and students respond positively after experienced it. TBL favor increased after students learning experience with TBL [3]. This phenomenon could be a result of motivational variations between year two and year three students. As mentioned in the literature, students have unfavorable opinions of TBL because of its extensive needs of out-of-class preparation, workload and class participation [12]. Furthermore, evidence suggests that learning styles and personality types of students are signi cant factors in uencing learning that takes place in a small-group TBL setting [29]. Another possible explanation is that learning materials selected for TBL activities might not be well prepared or inappropriate to introduce via such pedagogy. The reasons for this could be identi ed by conducting a focused group discussion among students those are not like TBL method.

Limitations of study
The study addresses the following limitations that should keep under consideration. The main limitation of the study was sample size although the study targeted all medical students enrolled in UBCOM. Several recommendations reported in the literature about the proper sample size to use for conducting a factor analysis. It was widely accepted that larger sample size is better for factor analysis that provides expected measures [30]. The study did not involve medical students at year ve and six since UBCOM did not matriculate students in these years when the study was conducted. In future investigations, it might be possible to include all medical students from year two to six to get a wide picture of TBL in our medical school. Skillful facilitator of TBL has ability creates a right learning environment and engaging reluctant students. In general, this might succeed by the ability to manage the session time, actively listen, paraphrase, summarize and draw out the answers from the students, support students accountability [2]. However, many of the facilitators in UBCOM were a novice in the implementation of TBL that might cause a negative attitude among students toward TBL.

Conclusions
In conclusion, the ndings of the present study provide evidence for the reliability of TBL-SAI and supporting the validity of this instrument for medical students. Our students rated a positive score for TBL favor despite was lowest when compared with those reported in other studies. Overall, our experience with TBL is promising in terms of accountability, preference for learning mode, and students' satisfaction. Medical students in year four have more admiration for the TBL method than second and third-year students. This con rms that long-term use of TBL in the education system improves the impact of this method on students. Moreover, further longitudinal studies are recommended to bring a clear picture of the effectiveness of TBL in UBCOM.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. STROBEchecklistcrosssectionalCopy.doc