Abstract

Background Course quality assessment contributes to evaluate teaching effectiveness and to improve student learning. Several course quality assessment questionnaires have been carried out associated with teaching quality evaluation of medical education. However, little is known about views of medical students regarding the quality of preclinical courses. To fill this gap, we aimed to develop a novel multi-dimension instrument for assessing teaching quality with the perception of medical students in preclinical courses.

Methods The original Teaching Quality Evaluation Questionnaire (TQEQ) containing seven dimensions: course contents, teaching abilities, teaching methods, teaching attitudes, learning outcomes, teacher characteristics, student subjection, consisting of 47 items was formed according to literature reviews and group design. We sent the original questionnaires with items in a random order to medical students of Shanghai Jiao Tong University School of Medicine. After collecting the valid questionnaires, the exploratory factor analysis was conducted to assess construct validity and Cronbach’s alpha coefficient was used for evaluating internal consistency reliability of the questionnaires.

Results In total, 590/646 (91.3%) of participants completed the questionnaire regarding preclinical course evaluation. The exploratory factor analysis yielded seven common factors, learning outcomes, teaching attitudes, student subjection, teaching abilities, teaching methods, teacher characteristics, teaching interactions, consisting of 39 items explained 58.449% of total variance and the factor loading value was above 0.4. In addition, Cronbach’s alpha coefficients ranged from 0.669 to 0.914.

Conclusion This study provides a new, validated and useful instrument for measuring teaching effectiveness of preclinical courses based on the views of medical students. It is feasible for use in medical schools.

Background

Medical education must meet high standards as medical students carry more responsibility in the future. In order to ensure the effectiveness of the medical courses, course evaluation is an integral of medical education [1–2].

Rating scale is emerging as an important determinant in the course evaluation of medical education. For instance, the Course Experience Questionnaire (CEQ), containing five main factors: good teaching, clear goals and standards, appropriate workload, appropriate assessment and generic skills [3], has been demonstrated validity in the medical education setting during preclinical courses [4].The Student Evaluation of Educational Quality (SEEQ) was widely used in higher education institution in United States [5]. Thus, a good quality of preclinical course assessment could also help teachers to improve teaching effectiveness and outcomes.

Traditional lecture-based learning (LBL) still remains the mainly method in the process of preclinical teaching in medical schools in China, though, a growing number of medical schools have adopted the popularity teaching methods in medical courses, such as problem-based learning (PBL) [6], team-based learning (TBL) [7]. So far, although many studies have carried out different questionnaires associated with teaching quality evaluation of medical education, such as clinical teaching [8], outpatient teaching [9], bedsides teaching [10], PBL [11], there lacks views of medical students regarding the quality of preclinical courses.

There is evidence that student perceptions are more predictive of student learning outcomes than other methods such as external observations or teachers’ subjective perceptions of their own teaching behavior [12]. The reliability and validity of student ratings is, to a certain degree, compromised because student evaluations usually reflect their expectations about the teacher, despite of a bigger expenditure [12]. Therefore, in this study, we develop and validate a novel questionnaire covering seven dimensions from the view of medical students, hoping to have an accurate and profound understanding of our students and help teachers and faculties to improve the preclinical courses.

Methods

Item generation and scale development

The items were yield through literature reviews and group design. It was arranged to contain seven dimensions and each dimension was designed to cover 5- to 7-items. Each item was measured by a five-point Likert-type scale, with 1 represents strongly disagree, 2 represents disagree, 3 represents neutral, 4 represents agree, 5 represents strongly agree. Ultimately, an original version of The Teaching Quality Evaluation Questionnaire, consisting of 47 items, was generated. On the top of the questionnaire, it included the investigators information, such as class, grade, and major. Each item of the questionnaire was in a random order when sent to the participants (Table 1).

Participants

We investigated students whose major were nutrition, clinical medicine, pediatrician and public health in Shanghai Jiao Tong University School of Medicine. The inclusion criteria for participants were having the compulsory curriculum at grade 2 to grade 4.

Data analysis

We used SPSS 22.0 to analyze the data. Item scores was calculated as Mean ± standard deviations (SD). Exploratory factor analysis (EFA) was performed to assess the construct validity, while Cronbach’s alpha coefficient was applied to examine the internal consistency.

Structural Validity

Exploratory Factor Analysis

The number of respondents between five to ten participants per item is essential for EFA [13]. The Kaiser-Meyer-Olkin (KMO) measure and Bartlett’s test of sphericity were applied to explore the sampling adequacy of factor analysis. Then, EFA using principal component analysis with varimax rotation was conducted to determine the factor structure of the items. Factor with an eigenvalue higher than one was considered for extraction [13]. Factor loadings, equal to or greater than 0.40, were selected [14].

Reliability

Internal consistency

Cronbach’s alpha coefficient was conducted to measure the consistency of each item. The satisfied Cronbach’s alpha coefficient was more than or equal to 0.70.

Ethics approval

This study was approved by the Ethical Committee of Shanghai Jiao Tong University School of Medicine. Study participants han signed informed consent and all data were collected anonymously.

Results

Missing values and descriptive statistics

In total, 646 medical students participated in this study, and 616 surveys were returned. We screened out and eliminated the samples having missing, outliner or invalid values by descriptive statistics. Totally, there were 590 valid and completely questionnaires. The response rate was 91.3%.

Structural validity

The KMO measure of sampling adequacy was 0.946 and the Bartlett’s test of sphericity was significant (χ2 = 1081.43, p < 0.001), indicating the data were satisfied for factor analysis [15]. Seven common factors, eigenvalues greater than 1, were extracted using principal component analysis with varimax rotation (Fig.1). The seven-factor solution accounted for 58.449% of the variance (Table 2). The items were removed if the factor loading was lower than 0.40. The first factor (F1) was named learning outcomes, which involved thirteen items; the second factor (F2) was named teaching attitudes, which involved five items; the third factor (F3) was named student subjection, which involved four items; and the fourth factor (F4) was named teaching abilities, which involved five items; the fifth factor (F5) was named teaching methods, which involved five items; the sixth factor (F6) was named teacher characteristics, which involved four items; the seventh factor (F7) was named teaching interactions, which involved three items.

Internal consistency reliability

The Cronbach’s alpha coefficient of the new 39-item scale was 0.941, and the Cronbach’s alpha coefficients of the subscales raged from 0.669 to 0.914 (Table 3), indicating excellent internal consistency and stability of the instrument.

Discussion

In this study, we described the development of a new teaching quality evaluation questionnaire (TQEQ) of preclinical courses from the perceptive of medical students with the purpose of improving the effectiveness of preclinical courses in medical schools.

Reliability, validity efficiency and feasibility assessment of the evaluation program is necessary in order to provide an accurate, fair and reliable assessment of teaching quality [16]. We send the original questionnaire whose 47 items are in random to the participants, in order to reduce the impact of effects of subjective consciousness. The EFA of the TQEQ demonstrated seven factors consisting of 39 items: learning outcomes, teaching attitudes, student subjection, teaching abilities, teaching methods, teacher characteristics and teaching interactions, which included the four main aspects of teaching quality program: structure, process, teacher characteristics, and outcomes [17]. Besides, the value of Cronbach’s alpha was exceeded 0.70, indicating an acceptable and good internal consistency reliability of the instrument [18]. Thus, this new instrument may well be used to assess teaching quality for medical schools.

A good questionnaire not only contributes to measure the effectiveness of teaching, but also helps to provide a well informal feedback for the teachers and faculties. In this study, interestingly, we found that the common factor “learning outcomes’ weighted most heavily among the seven factors. Moreover, five of ten highest value items, “Teachers use heuristic teaching method to inspire students to think” (q3, 4.35 ± 0.82), which 86.8% agreed and only 2.9% disagreed; “Teachers are good at cultivating students’ ability of subject thinking rather than indoctrinating knowledge blindly " (q1, 4.49 ± 0.72), which 90.3% agreed and only 1.2% disagreed; “This course will train students to solve the problems” (q20, 4.29 ± 0.82), which 84.2% agreed and only 2.7% disagreed;“Students will have a good command of key points after the course” (q6, 4.4 ± 0.78), which 86.8% agreed and only 2.0% disagreed;“Teachers have excellent theoretical knowledge of their specialty, and appropriate application in teaching " (q21, 4.44 ± 0.71), which 89.7% agreed and only 1.4% disagreed, are in this common factor. In the other words, whether the course will improve students’ abilities, such as heuristic thinking, solving problems, is important for medical students. This is consistent with the previous studies that three types of outcomes need to take in consideration when determining the efficacy of medical education programs: education, clinical career, and environmental outcomes [19]. So, it is critical for teachers to focus on how to cultivate and elevate students’ comprehensive abilities rather than teaching literally.

In addition, the highest value items “Teachers are familiar with the teaching content and able to sort out the key points” (q42, 4.66 ± 0.63), 93.5% agreed and only 1.0% disagreed; “Teachers are well prepared for the course " (q4, 4.60 ± 0.65), 93.8% agreed and only 1.4% disagreed; “Teachers are good at organizing the process of class " (q32, 4.48 ± 0.69), 90.0% agreed and 10.0% disagreed, are in the common factor, “teaching abilities”, revealing that teacher’s performance is the most important for medical students. Thus, in the LBL model, it is essential for teachers to make good preparation and have a good charge of the course.

Moreover, the other two high value items “Teachers encourage students to discuss and ask questions " (q41, 4.57 ± 0.69), 91.7% agreed and only 1.0% disagreed; “Teachers are patient with students’ questions " (q40, 4.32 ± 0.84), 83.4% agreed and only 2.7% disagreed, are in the common factor, “teaching interactions”, indicating that students rely on teachers and hope teachers could give positive feedback to them.

Among the seven common factors, the common factor “teacher characteristics”, containing the four items “Teachers are young and nearly have no generation gap with students” (q43, 3.2 ± 1.07), 38.4% agreed and 22.0% disagreed; “Teachers are humorous” (q45, 3.39 ± 1.16), 47.9% agreed and 21.0% disagreed; “Teachers are good-looking and attractive” (q16, 3.35 ± 1.19), 46.4% agreed and 22.5% disagreed; “The evaluation of teacher related to the course difficulty” (q15, 3.64 ± 1.20), 60% agreed and 18.1% disagreed, seems not so essential to the medical student. This was further validated that teachers should exert their strong efforts to prepare their courses and improve their abilities, while they still need to pay attention to their words and actions.

However, this study still has some limitations. First, the course evaluation was administered at the middle of a curriculum; however, some studies suggested that continuous assessment during the courses brings up higher reliability ratings rather than the retrospective evaluation [1, 20]. Second, the questionnaire is collected from the view of students, but it should be also accessible to teachers. The teaching planning, teaching methods, and teaching skills may more suitable for teachers themselves to evaluate. Thus, both students and teachers are critical in the process of the assessment of preclinical courses. In addition, the data were collected in a single medical school; therefore, application of the questionnaire in other medical schools is needed to validate the effectiveness of the evaluation system.

In conclusion, TQEQ was demonstrated to be a valid and credible instrument for assessing teaching quality of medical school. Future studies needs to explore whether feedback acquired from this questionnaire contributes to facilitate teaching quality of preclinical courses and student learning outcome.

Conclusion

This study provides a new, validated and useful instrument for measuring teaching effectiveness of preclinical courses based on the views of medical students. The Exploratory factor analysis of the new teaching quality evaluation scale demonstrated seven factors consisting of 39 items: learning outcomes, teaching attitudes, student subjection, teaching abilities, teaching methods, teacher characteristics and teaching interactions. This new scale is based on the views of medical students. It is feasible for use in medical schools to evaluate the preclinical courses.

Abbreviations

CEQ: Course Experience Questionnaire; EFA: Exploratory factor analysis; KMO: Kaiser-Meyer-Olkin; LBL: Lecture-based learning; PBL: Problem-based learning; SD: standard deviations; SEEQ: Student Evaluation of Educational Quality; TBL: Team-based learning; TQEQ: Teaching Quality Evaluation Questionnaire

Declarations

Ethics approval and consent to participate

This study was approved by the Ethical Committee of Shanghai Jiao Tong University School of Medicine. Study participants han signed informed consent and all data were collected anonymously.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

There was no funding for this study.

Authors’ contributions

YLQ, NNF designed the study. PH, XFY collected the data, did the statistical analysis and wrote the first draft of the manuscript. HYH contributed to the data collection and statistics. YLQ, NNF contributed to the review of the manuscript. All authors read and approved the final manuscript.

Acknowledgements

We are grateful to all the participants for their contributions to this study.

References

1. Kogan JR, Shea JA. Course evaluation in medical education. Teach Teach Educ. 2007;23(3):251–64.
2. Schiekirka S, Reinhardt D, Heim S, Fabry G, Pukrop T, Anders S, et al. Student perceptions of evaluation in undergraduate medical education: A qualitative study from one medical school. BMC Med Educ. 2012;12:45.
3. Espeland V, Indrehus O. Evaluation of students’ satisfaction with nursing education in Norway. Journal of Advanced Nursing. 2003;42(3):226–36.
4. Broomfield D, Bligh J. An evaluation of the ‘short form’ course experience questionnaire with medical students. Med Educ. 1998;32(4):367–9.
5. Sarah SMAF, Christoph HLTR. Evaluation in medical education: A topical review of target parameters, data collection tools and confounding factors. Med Edu. 2015;13:1–9.
6. Fan APC, Kosik RO, Tsai TCC, Cai QL, Xu GT, Guo L, et al. A snapshot of the status of problem-based learning (PBL) in Chinese medical schools. Med Teach. 2014;36(7):615–20.
7. Burgess AW, McGregor DM, Mellis CM. Applying established guidelines to team-based learning programs in medical schools: a systematic review. Acad Med. 2014;89:4.
8. Stalmeijer RE, Dolmans DHJM, Wolfhagen IHAP, Muijtjens AMM, Scherpbier AJJA. The Maastricht Clinical Teaching Questionnaire (MCTQ) as a Valid and Reliable Instrument for the Evaluation of Clinical Teachers. Acad Med. 2010;85(11):1732–8.
9. Zuberi RW, Bordage G, Norman GR. Validation of the SETOC instrument—Student evaluation of teaching in outpatient clinics. Adv Health Sci Educ Theory Pract. 2007;12(1):55–69.
10. Dreiling K, Montano D, Poinstingl H, Muller T, Schiekirka-Schwake S, Anders S, von Steinbuchel N, Raupach T. Evaluation in undergraduate medical education: Conceptualizing and validating a novel questionnaire for assessing the quality of bedside teaching. Med Teach. 2017;39:820–827.
11. Ravens U, Nitsche I, Haag C, Dobrev D. What is a good tutorial from the student’s point of view? Evaluation of tutorials in a newly established PBL block course “Basics of Drug Therapy”. Naunyn-Schmiedeberg’s Archives of Pharmacology. 2002;366(1):69–76.
12. Maulana R, Helms-Lorenz M, Wim V D G. Development and evaluation of a questionnaire measuring pre-service teachers’ teaching behaviour: a Rasch modelling approach. School Effectiveness and School Improvement. 2015; 26(2):169–94.
13. Williams B, McKenna L, French J, Dousek S. The clinical teaching preference questionnaire (CTPQ): an exploratory factor analysis. Nurse Educ Today. 2013;33(8):814–7.
14. Nikkhah M, Heravi-Karimooi M, Montazeri A, Rejeh N, Sharif Nia H. Psychometric properties the Iranian version of Older People’s Quality Of Life questionnaire (OPQOL). Health Qual Life Outcomes. 2018;16(1):174.
15. Lee EH, Lee YW, Lee KW, Nam M, Kim SH. A new comprehensive diabetes health literacy scale: Development and psychometric evaluation. International Journal of Nursing Studies. 2018;88:1–8.
16. Snell L, Tallett S, Haist S, Hays R, Norcini J, Prince K. A review of the evaluation of clinical teaching: new perspectives and challenges. Med Educ. 2000;34(10):862–70.
17. Gibson KA, Boyle P, Black DA, Cunningham M, Grimm MC, McNeil HP. Enhancing evaluation in an undergraduate medical education program. Acad Med. 2008;83(8):787–93.
18. Jiang H, Zhang S, Ding Y, Li Y, Zhang T, Liu W, Fan Y, Li, Zhang R, Ma X. Development and validation of college students’ tuberculosis knowledge, attitudes and practices questionnaire (CS-TBKAPQ). BMC Public Health. 2017;17(1):949.
19. Blumberg P. Multidimensional outcome considerations in assessing the efficacy of medical educational programs. Teach Learn Med. 2003;15(3):210–4.
20. Peluso MA, Tavares H, D’Elia G. Assessment of medical courses in Brazil using student-completed questionnaires. Is it reliable? Rev Hosp Clin Fac Med Sao Paulo. 2000;55(2):55–60.

Tables

Table 1. The Teaching Quality Evaluation Scale

Table 2. Factor loadings by exploratory factory analysis

Table 3. Internal consistency reliability