Objective structured practical examination in experimental physiology increased satisfaction of medical students

DOI: https://doi.org/10.21203/rs.2.13570/v1

Abstract

Background Medical education is a dynamic process and needs to be improved to meet the new expectation from medical practitioners, health worker and community. An important part of medical student’s education is selection of an appropriate assessment method. The objective structured practical examination (OSPE) can evaluate practical capabilities in suitable step-wise, scientific, targeted and scheduled manner with direct consideration of student’s execution during programmed test stations. The purpose of this study was to investigate the outcomes of the OSPE utilization versus traditional practical examination (TPE) for evaluating students in experimental physiology.

Material and Methods A total 120 second year medical students for experimental physiology exam were entered in this study: 1. TPE group (TPE used as a final exam; n=40); 2) TPE + OSPE group (TPE applied for half of topics and OSPE for another half; n=41); and 3. OSPE group (OSPE performed as a final exam; n=39). In order to evaluate the effect of OSPE, the average of final grade of studied groups were compared. In addition, a 5 points Likert scale questionnaire, consisting of 10 questions was used to qualitatively and quantitatively evaluated the student’ s attitude on the used method.

Results Obtained results showed that the total grade in TPE group was significantly higher in comparison to TPE+OSPE and OSPE groups (respectively P<0.01 and P<0.05), while according to student’s expression, the average score for all of items in feedback questionnaire was increased significantly in TPE+OSPE and OSPE groups compared with TPE group (P<0.001).

Conclusion In summary, feedback from students showed, they were in favor of OSPE compared with the TPE and regarding their expressions in feedback questionnaire OSPE can improve learning in physiology and increase students’ satisfaction.

Background

Assessment encompasses the learning and two of its main indicators are the interpretation and use of the information that is categorized for intended purpose (1). In addition, assessment can affect students learning, their motivation to learn, how teachers teach and has powerful impact on student performance (2). Teachers can help students to be more focused and motivated if assessed them for their ability of integration, usage and linkage of knowledge and especially if students skills are observed and graded (2, 3). Therefore, it seems selection of assessment method by teachers is an important part of the medical student’s education (4).

Although, many options are available to achieve this purpose, the Objective Structured Practical Examination (OSPE) is a highly recommended method (2, 5–9). This assessment tool is derived from Objective Structured Clinical Examination (OSCE) by Harden and Gleeson and used for evaluation of basic sciences, preclinical and paraclinical students (6, 10). Many medical schools in all over the world have been used OSPE for assessing students’ performance in laboratory exercises (11, 12).

OSPE is a specified set of task which assesses what students can do in a structured pattern objectively under direct observation and is able to assign the above-mentioned capabilities (13). The Miller’s framework of development of clinical qualifications, concentrate on four levels of assessment: knows, knows how, shows how and does. It was reported OSPE evaluates the third level “shows how” of this framework by focusing on the assessment of performance of specific skills in a controlled setting (14). It is believed that, OSPE can reduce the examiners’ variability in marking the students and it is a standardized tool which has shown advantages over TPE (12, 15).

In department of physiology of Ahvaz Judishapur University of Medical Sciences (AJUMS) which comes under the medical council of Iran, the human physiology is thought as basic sciences subject in first 4 semesters of medical student’s courses. In this department students are evaluating for experimental physiology using TPE. This method involves performing a particular experiment randomly and the assessment is based on the global performance rather than individual skills (12). According to the Miller’s framework, TPE is mainly focuses on the base of this pyramid, i.e. the “knows” and “knows how” aspects (4). Therefore, It seems that, application of a method which is more objective and structured and provides giving feedback to the students for understanding their weakness and improve their skills is necessary (12).

It was reported that, OSPE in most of cases improve students’ performance, especially in experimental physiology (16). In addition, in some studies feedback from students revealed that, using OSPE increased the satisfaction in medical students (17) and validity and reliability of this method is more than TPE (18).

Department of physiology of AJUMS is trying to impellent a series of reforms in teaching methods and assessment strategies to improve the quality of medical education. Therefore, considering the OSPE advantages comparison to TPE, this study was undertaken to determine student satisfaction regarding the OSPE as a method of assessment of experimental physiology before import it in the forthcoming university examination.

Methods

The study was conducted with 120 second year (3rd semester) students in department of physiology of AJUMS, Ahvaz, Iran, after the approval from the Institutional Ethics Committee for research on human subjects (IR.AJUMS.REC.1398.234).

The lecturers were well trained to achieve an optimal organization of the TPE and OSPE. In addition, these two assessment methods were introduced to the participants by short lecture and in this study verbal consent was obtained and it approved by ethics committee.

TPE session

In the department of physiology of AJUMS, TPE consists of random selection of an experiment which its usage situation and procedure completely taught to the students within a session of that semester. It is noteworthy, through the execution of the experiment, performance of students was evaluated.

OSPE session

OSPE sessions were held according to a blueprint of the syllabus and structured checklist which prepared as Bloom’s taxonomy. In addition, all response and skills stations were validated by a senior faculty member. The OSPE consisted of stations of 3 minutes, including: 1. Response station; questions depend on theory aspect of topics, like blood, muscle nerve physiology, the respiratory and cardiovascular system. In this regard, we used different tools: a) Flash cards; including some pictures related to a disorder and asking student to realize the disease. b) Videos; which are showing a correct or incorrect procedure related to one experiment and the examiner expect the student realize what is the experiment? What is its goal? And how we can use this experiment in clinic? c) Microscope; especially in the case of white blood cell differentiation. 2. Skill stations; questions were regarding different topics, mainly blood cells experiments, for example white and red blood cells (RBC) counting, fragility test for RBC and hematocrit measurement. In these stations, examiners directly graded student’s performance as; 1 (weak), 2 (moderate), 3 (great). It should be noted, examiners frequently discussed about the required arrangements for conduct of the OSPE and preparation checklist/key to reach a consensus without any ambiguity.

Experimental design

Students were assigned to 3 groups: 1. TPE group (n = 40), the assessment method was just based on the gained score in the TPE session; 2. TPE+OSPE group (n = 41), TPE was used for half of topics which taught during the semester and OSPE utilized for another half topics and 3.OSPE group (n = 39), students were evaluated according to total earned scores within 8 stations. Therefore, in this group the total score was the sum of obtained results from TPE and OSPE. It should be noted, total grade in Iran is evaluated out of 20.

Feedback questionnaire

In this study, immediately after TPE and at the 8th station of OSPE a reliable, validated and structured questionnaire was used (7). In the TPE group the students were asked to express their attitude about the TPE and in TPE+OSPE and OSPE groups the students were surveyed regarding the OSPE. The questionnaire consists of 10 items and each item had 5 options. It should be noted, anonymity and confidentiality were held by asking students not to write their names. In addition, 10 items were based on Likert’s five-point scale to assess the impact of TPE and OSPE on student’s satisfaction and their efficiency in experimental physiology.

Likert scale items

Likert scale items analyzed and compared the effect of the TPE and OSPE on various aspects of student’s satisfaction and efficiency: 1) The questions asked were relevant, 2) Sufficient time was given to students, 3) TPE/OSPE is more fair, 4) TPE/OSPE is easier to pass, 5) TPE/OSPE is better method of assessment, 6) TPE/OSPE improves learning in physiology, 7) TPE/OSPE provides chance to score better, 8) TPE/OSPE is less stressful, 9) TPE/OSPE makes students to think more, 10) TPE/OSPE eliminates bias. These 10 items had the following options: strongly agree (option A; score 5), agree (option B; score 4), neutral (option C; score 3), disagree (option D; score 2) and strongly disagree (option E; score 1). A heightened score 5 and a score 1 was given for the options with respectively strong and having completely no impact of TPE and OSPE on student’s satisfaction and efficiency.

Statistical analysis

The student’s total grades and total score for each item in questionnaires were averaged and expressed as mean ± S. E.M using GraphPad Prism version 6.01 for Windows (GraphPad Software, Ca, USA). To evaluate the effect of OSPE and TPE on average of total grade and average scores of each item in the questionnaire one way ANOVA followed by post-hoc Tukey’s test was used. P-value of less than 0.05 was considered to represent a significant difference. In addition, the results of different options in Likert questionnaire were presented as frequency percentage.

Results

The total grade of experimental physiology in TPE group (18.09±0.22) was significantly higher than in compare to TPE+OSPE (16.68±0.26, P<0.01) and OSPE (17.15±0.32, P<0.05) groups (Fig 1). Student’s perspective regarding the TPE and OSPE for each option of 10 questions was presented as frequency percentage (Table 1).

One hundred percent of TPE+OSPE group and 85% of OSPE group felt that the OSPE questions were relevant, comprehensive and covered a wider area of knowledge in comparison to TPE group (72.5%). In addition, the average score of TPE+OSPE (4.63±0.07) and OSPE (4.46±0.17) groups for 1st question showed a significant increase compared with TPE group (3.62±0.2) (P<0.001); (Fig 2A).

Our results for second item of questionnaire showed that majority of students in TPE+OSPE (88%) and OSPE (77%) groups expressed that the sufficient time was given to them during OSPE, while 52.5% of TPE group agreed with this statement about TPE. Analyzing the second item in questionnaires revealed a significant increase of average score in TPE+OSPE (4.63±0.13) and OSPE (4.07±0.23) groups compared with TPE group (2.97±0.24) (P<0.001); (Fig 2B).

Although, the average of total grade in TPE group was significantly higher than in two other groups, minority of students in response to items three and four believed that TPE is a fair method (12.5%) and easier to pass (37.5%). By contrast, most of students in TPE+OSPE (respectively 95% and 63%) and OSPE (respectively 74% and 70%) groups declared this opinion regarding the OSPE. Furthermore, average score for items three and four showed a significant increment in TPE+OSPE group (respectively 4.51±0.09 and 3.87±0.15) as well as OSPE group (respectively 4.07±0.22 and 3.94±0.22) compared with TPE group (respectively 2.25±0.17 and 2.62±0.21) (P<0.001); (Figs 2C and 2D).

In response to item five, unlike the TPE group (27.5%), majority of students in TPE+OSPE (88%) and OSPE (74%) groups expressed OSPE is a better method for assessment. One way ANOVA analysis for item five revealed a significant increase of average score in TPE+OSPE (4.29±0.12) and OSPE (4.02±0.23) groups compared with TPE group (2.55±0.21) (P<0.001); (Fig 2E).

The results showed, just 22.5% of students in TPE group believed that TPE improve learning in physiology, while most of students in TPE+OSPE (76%) and OSPE (77%) groups express this issue regarding OSPE. Analysis of average score for this item indicated a significant increase in TPE+OSPE (4.09±0.14) and OSPE (4.07±0.22) groups in comparison to TPE group (3.32±0.22) (P<0.001); (Fig 2F).

Interestingly, data analysis for items 7 and 8 revealed that less than fifty percent of students in TPE group (respectively 37.5% and 27.5%) agree the statement that TPE provides chance score better and is less stressful, while most of students in TPE+OSPE (respectively 66% and 71%) and OSPE (respectively 58% and 69%) verify this issue regarding OSPE. One way ANOVA analysis for average score of items 7 and 8 showed a significant increase in TPE+OSPE (3.85±0.12 and 3.95±0.15) and OSPE (respectively 3.71±0.23 and 3.84±0.23) groups compare to TPE group (respectively 2.67±0.21 and 2.30±0.2)(P<0.001); (Figs 2G and 2H).

Considering item 9 in three questionnaire indicate that, just 15% of students in TPE group declared that TPE makes students think more, otherwise 83% of students in both TPE+OSPE and OSPE groups stated this matter about OSPE. In addition, average score analysis revealed a significant increase in TPE+OSPE (4.19±0.12) and OSPE (4±0.20) in comparison to TPE group (2.05±0.20) (P<0.001); (Fig 2I).

Finally, analysis of different options of item ten showed a few students in TPE group (12.5%) expressed TPE eliminates bias, while more than fifty percent of students in TPE+OSPE (61%) and OSPE (64%) groups declared this issue regarding OSPE. Moreover, one way ANOVA analysis indicate significant increment of average score in TPE+OSPE (3.78±0.14) and OSPE (3.74±0.22) groups compared with TPE group (1.95±0.17)(P<0.001); (Fig 2J).

Discussion

This study was aimed to receive the acceptability of the OSPE for the first time between students as a relatively new assessment method for experimental physiology in AJUMS. The obtained results showed that the average of total grade in TPE group was significantly higher in comparison to TPE+OSPE and OSPE groups. In spite of better grade in TPE group, students’ satisfaction and learning in experimental physiology increased in TPE+OSPE and OSPE groups compared with TPE group.

Although in our study the average of total grade in TPE group was significantly better than two other groups, but Dissanayake et al. reported a marked improvement in the mean scores for the experimental physiology at King Faisal University Medical School (11). We supposed that, our finding is presumably as a result of students’ first experience with OSPE in TPE+OSPE and OSPE groups. Moreover, they were not familiar enough with this assessment method. On the other hand, it was reported that getting feedback has profound effect on students’ examination performance (19), as in TPE+OSPE and OSPE groups, students did not receive any feedback about OSPE before exam day, while students in TPE group were completely familiar with this assessment method.

In the present study, feedback from students immediately after exam indicated that, they were in favor of the OSPE compared with the TPE (Table 1). In line with our results, in many studies OSPE was rated by the students as a creditable, effective, useful, interesting and challenging assessment method (2, 5, 9, 20).

Unlike TPE group, majority of students in TPE+OSPE and OSPE groups similar to other studies (5, 7, 16, 21) felt that OSPE questions were more relevant, sufficient time was given to them, this assessment tool is more fair, helps to score better, less stressful and eliminates bias. These items were satisfactions criteria in feedback questionnaire. As students’ declaration, exposure to similar types of question with the same difficulty was an important advantage for validation of OSPE by students. In line with our findings it was reported that, OSPE is accepted between students because, it includes a large number of questions and skills with wide variety of physiological concepts, in which students can test for much more abilities compared to TPE (2, 9) and they will be more satisfied if their talents considered for assessment (3, 22). In addition, Lakshmipathy pointed that students’ concentration increased when their skills were demonstrated during OSPE (2). Respect to OSPE advantages, few students in this study expressed opposite attitude regarding OSPE and this could be attributed to habituation to the TPE. In addition, presence of examiners in some of stations with the checklist in their hands may be makes OSPE more stressful and anxiety driven for some of students, thus they were not happy with this assessment method. Moreover, Pramod Kumar, et al. pointed that, although most of the participants believed that OSPE is a better method of examination and covered wide range of knowledge than TPE, but a large number of students in their study (63.5%) expressed that the OSPE may be exhausting and stressful if numbers of stations are increased (23). Therefore, the number of the stations and difficulty of questions are important points in acceptance of OSPE among students and in this study this issue was considered, therefore nobody complained about it.

An interesting achievement in this study was, despite significant reduction in average of total grade in TPE+OSPE and OSPE groups compared to TPE group, the most of students declared that OSPE improved their learning in physiology and made them to think more. Although the feedback questionnaire was presented to students before score announcement, but this result illustrated that students agreed with an assessment method which increased the level of learning. It is suggested that, assessment as learning is where it helps teachers gain information about what students understand and how they can use their knowledge (2). OSPE skill stations assesse the “shows how” level of Miller’ framework, while response stations demonstrate “knows” (recall of facts, principles, and theories) and “knows how” (problem solving, application and interpretation) (2). Therefore, it seems OSPE can be useful to improve the medical education and prepare medical students to achieve the better clinical competencies.

There were few limitations in the present study, such as the first counter of the students with OSPE, and the number of student in each group and it will be the basis of the future development to reform and refine the OSPE as an assessment tool. In addition, this study emphasized the need of continuous faculty development in the field of medical education for its betterment.

Conclusion

The present study showed that the OSPE was well accepted by the medical students of AJUMS compared with the TPE. Moreover, it provides atmosphere for refining the method before OSPE implement in the forthcoming university examination.

Abbreviations

OSPE: Objective Structured Practical Examination

TPE: Traditional Practical Examination

Declarations

Availability of data and materials

All data generated or analyzed during this study are included in this manuscript.

Acknowledgement

Not applicable.

Funding

This study was supported by a grant (IR.AJUMS.REC.1398.234) from Avaz Jundishapur University of Medical Sciences.

Competing Interest

The authors declare that there are no competing interests.

References

1.McLachlan JC. Problem-based examining: a different approach to assessment. Med Educ. 1997;31(4):299–301.

2.Lakshmipathy K. MBBS student perceptions about physiology subject teaching and objective structured practical examination based formative assessment for improving competencies. Adv Physiol Educ. 2015;39(3):198–204.

3.Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226–35.

4.Gupta P, Dewan P, Singh T. Objective Structured Clinical Examination (OSCE) Revisited. Indian Pediatr. 2010;47(11):911–20.

5.Abraham RR, Raghavendra R, Surekha K, Asha K. A trial of the objective structured practical examination in physiology at Melaka Manipal Medical College, India. Adv Physiol Educ. 2009;33(1):21–3.

6.Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 1979;13(1):41–54.

7.Kundu D, Das HN, Sen G, Osta M, Mandal T, Gautam D. Objective structured practical examination in biochemistry: An experience in Medical College, Kolkata. J Nat Sci Biol Med. 2013;4(1):103–7.

8.Lunenfeld E, Weinreb B, Lavi Y, Amiel GE, Friedman M. Assessment of emergency medicine: a comparison of an experimental objective structured clinical examination with a practical examination. Med Educ. 1991;25(1):38–44.

9.Sandila MP, Ahad A, Khani ZK. An objective structured practical examination to test students in experimental physiology. J Pak Med Assoc. 2001;51(6):207–10.

10.Harden RM. What is an OSCE? Med Teach. 1988;10(1):19–22.

11.Dissanayake AS, Ali BA, Nayar U. The influence of the introduction of objective structured practical examinations in physiology on student performance at King Faisal University Medical School. Med Teach. 1990;12(3–4):297–304.

12.Rahman N, Ferdousi S, Hoq N, Amin R, Kabir J. Evaluation of objective structured practical examination and traditional practical examination. Mymensingh Med J. 2007;16(1):7–11.

13.Nayar U, Malik SL, Bijlani RL. Objective structured practical examination: a new concept in assessment of laboratory exercises in preclinical sciences. Med Educ. 1986;20(3):204–9.

14.Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63–7.

15.Matsell DG, Wolfish NM, Hsu E. Reliability and validity of the objective structured clinical examination in paediatrics. Med Educ. 1991;25(4):293–9.

16.Abraham RR, Upadhya S, Torke S, Ramnarayan K. Student perspectives of assessment by TEMM model in physiology. Adv Physiol Educ. 2005;29(2):94–7.

17.Menezes RG, Nayak VC, Binu VS, Kanchan T, Rao PP, Baral P, et al. Objective structured practical examination (OSPE) in Forensic Medicine: students’ point of view. J Forensic Leg Med. 2011;18(8):347–9.

18.Ananthakrishnan N. Objective structured clinical/practical examination (OSCE/OSPE). J Postgrad Med. 1993;39(2):82–4.

19.Lipnevich AA, Smith JK. Effects of differential feedback on students’ examination performance. J Exp Psychol Appl. 2009;15(4):319–33.

20.Malik SL, Manchanda SK, Deepak KK, Sunderam KR. The attitudes of medical students to the objective structured practical examination. Med Educ. 1988;22(1):40–6.

21.Feroze M, Jacob AJ. OSPE in pathology. Indian J Pathol Microbiol. 2002;45(1):53–7.

22.Kemahli S. Clinical Teaching and OSCE in Pediatrics. Med Educ Online. 2001;6(1):4531.

23.Pramod Kumar GN, Sentitoshi, Nath D, Menezes RG, Kanchan T. Student’s perspectives on objective structured practical examination (OSPE) in Forensic Medicine - a report from India. J Forensic Leg Med. 2015;32:39–41.

Tables