Comparison of the statistics and item analysis of multiple-choice questions between free and tuition-paying admission

doi:10.21203/rs.3.rs-650668/v2

Download PDF

Research Article

Comparison of the statistics and item analysis of multiple-choice questions between free and tuition-paying admission

https://doi.org/10.21203/rs.3.rs-650668/v2

This work is licensed under a CC BY 4.0 License

Version 2

posted

You are reading this older preprint version

Read the latest preprint version →

Background: The purpose of this pilot was to compare the multiple-choice test statistics of medical and dental students' exams between free and tuition-paying.

Methods: This descriptive-analytical study was conducted at Golestan University of Medical Sciences in Iran in 2020. The study population included students of medicine and dentistry. A total of 56 exams were selected in two student groups of free and tuition-paying admission in the physiology course. The results of quantitative evaluation of tests were used as the data of this study. The variables included difficulty index, discrimination index, the degree of difficulty, score variance, and Kuder-Richardson correlation coefficient.

Results: There were 32 medical and 24 dentistry exams. The cumulative total number of questions in these exams was 437 and 330 multiple choice questions, respectively. The number of medical students participating in the free-tuition and paying-tuition admissions was 1336 and 1076, and for dental students, these numbers were 395 and 235, respectively. There were no significant differences in normalized adjusted exams scores between two admission groups in both medical and dentistry tests. The mean of discrimination index in the free-tuition group was higher than in the paying-tuition group. The interaction between the type of admission and the field of study was significant for the discrimination index. This difference was more in tuition-free dental students than tuition-free medical students and tuition-paying dental students.

Conclusion: The type of student admission has no significant effect on student assessments in multiple-choice exams in matched educational conditions.

Internal Medicine

Educational Philosophy and Theory

Test

Student

Tuition

Free admission

Medical

Dentistry

Question analysis

Multiple Choice Question (MCQ)

Educational researches have indicated the impact of student admission policies on educational performance, both academic and non-academic^1-3. In terms of paying tuition, student admission policies apply in medical schools for various reasons. Some of these reasons include providing services in rural regions³, the number of students admittances⁴, the provision of primary care physicians, and educational justice⁵. The impact of tuition on educational performance is one of the main concerns of educational policymakers in most universities.

In Iran, universities are divided into two major groups in terms of field of activity. The first groups are universities that work in the field of medicine and its related fields and are known as medical universities. These universities are managed according to the rules and regulations of the Ministry of Health and Medical Education. The second groups are universities in the fields of engineering, art, literature, and other non-medical fields. These universities are managed according to the rules and regulations of the Ministry of Science, Research and Technology. In Iran, the universities can be categorized into two major groups, governmental and non-governmental, in terms of ownership and funding. Accordingly, student admission also includes free tuition and paying tuition. Free tuition is only at public universities and admission with tuition is at public and non-public universities. Admission without paying tuition and using free education has many applicants, and the competition between the candidates is fierce and requires a high score in the entrance exam. If the applicant is not accepted in the free tuition government admission, the next option is to apply for admission by paying tuition.

In the case of medicine and dentistry, the limited capacity of universities and a large number of applicants for them cause intense competition even in admission with paying tuition. Of course, the average score level in the entrance exam of those admitted without tuition is always higher than those admitted with tuition. This fact in some cases has led to bias and misjudgment about the ability and educational performance of paying-tuition students.

Various studies have been conducted about the educational performance of paying-tuition and free-tuition students. Some studies state that admission without tuition does not affect educational performance^{6, 7} and others point to the effect of tuition on low academic performance^{8, 9}. In most of these studies, the comparison criteria in the two groups of paying-tuition and free-tuition students were not fully matched or could not be implemented. Comparing the grade point average, scores of different courses, scores of a simple course in different semesters, and other similar methods without sufficient matching of the two groups can be accompanied by bias and error. The present study was conducted in terms of a maximum matching in 56 exams in terms of the course title, instructor, sources of the test, educational content, the volume of course content, time of the exam, number of questions, and holding the test timing at the same time in two independent groups of paying-tuition and free-tuition students. The purpose of this study was to compare the item analysis of multiple-choice questions between paying-tuition and free-tuition students in the basic science stage.

The present study was a descriptive-analytical study that was conducted cross-sectional in 2020 at Golestan University of Medical Sciences in Iran. The study population included students of medicine and dentistry. Medical and dental students have nearly identical curriculums and educational programs in their basic science stage. This stage includes the first four semesters. A total of fifty-six physiology course exams were selected in two groups of free-tuition and paying-tuition students in basic sciences stages of medicine or dentistry. After matching the conditions including the matching of the exam features (number of questions, date of exam, exam time, volume of content, topics, and resources) and the instructors matching, the results of quantitative evaluation of exams were used as the data of this study. Matching was not considered in terms of the location and order of the correct answers of multiple-choice questions, the number of students participating in each exam, and other student-related conditions such as admission quota, gender, or grade point average due to limitations and impossibility. It should be noted that at the time of the exams, none of the teachers, learners, facilitators, and colleagues had any information about the presentation of this study, and the idea of this research was developed long after the exams and their quantitative evaluation report.

For increasing the power of the study and provide a proper sample size, we included both medical and dental students’ exams. During the basic science stage, which consists of the first four semesters, the medical and dental students have almost the same educational programs and very similar curriculums. The physiology course of medical and dental students is also very comparable to each other. Therefore, we assumed considering both medical and dental exams’ data had no bias on results.

The data were collected based on the official test evaluation results of the Education Development Office and the official reports of the education office of the medical and dental school and the International Campus, which are the exams evaluators and exams organizers of medical and dental students for both free-tuition and paying-tuition, respectively. The items in these reports for each exams included twenty-one parameters, which are the title, date, correct answer score, incorrect answer score, no answer score, passing score, number of options, number of questions, number of candidates, difficulty index, discrimination index, number of the item with the degree of difficulty, number of the item with the ability to discriminate, mean, Score variance, standard deviation, highest scores, lowest scores, Kuder-Richardson correlation coefficient, and standard error of measurement. The number of questions in different exams was not equal. Therefore, we used adjusted exams scores for comparison between two admission groups in both medical and dentistry exams. The number of questions and the maximum score of the exam were different among these fifty-six exams. Therefore, they must be normalized for statistical analysis. Because the twenty is routinely considered as the maximum exam score in Iran’s educational system, this value was considered as the maximum exam score and all adjustments were normalized to it.

In this study, we classified the difficulty index of each question into five categories: simple, moderate, hard, very hard, and extremely hard. We also classified the discrimination index of each question in for categories: negative, poor, fair, and good.

A Chi-square test was used to compare qualitative variables. The Kolmogorov-Smirnov test was used to test the normality of quantitative data. Tables and graphs were used to describe the variables. Independent t-test and Mann-Whitney test were used to compare quantitative variables between the two independent groups. Two-way univariate analysis of variance was used to investigate the interaction of independent variables on quantitative variables. Statistical package for the social sciences (SPSS) software version 22 was used for statistical analysis. This study was approved by the ethical committee and the institutional review boards of the Golestan University of Medical Sciences (code: IR.GOUMS.REC.1397.159).

In this study, the item analysis of multiple-choice questions for students of medicine and dentistry between free-tuition and paying-tuition courses was compared. There were thirty-two medical and twenty-for dentistry exams. The number of exams in each of free tuition and paying tuition courses in the students of medicine were sixteen exams and in the students of dentistry were twelve exams. The total number of exams’ questions in the free-tuition and paying-tuition courses of the medical students was 437 questions and in the dental students was 330 questions. The number of medical students participating in the free-tuition and paying-tuition courses was 1336 and 1076 students, respectively, and the number of dental students was 395 and 235 students, respectively.

The number of questions based on the difficulty index and discrimination index and their relation to difficulty and discrimination level is shown in Table 1. According to the findings, the number of questions with simple and hard difficulty in the free-tuition group was more than the paying-tuition group. Also, the number of questions with poor and good discrimination index in the free-tuition group was more than the paying-tuition group.

Table 1- Frequency of number of questions based on the degree of difficulty and the ability to differentiate according to the field of study and the admission type

		*Admission typefield of study**						Total (%)
		Free-tuition			Paying-tuition
		Medical (%)	Dental (%)	Total (%)	Medical (%)	Dental (%)	Total (%)
Difficulty index	Simple	100 (22.9)	53(16.1)	153(19.9)	85(19.4)	46(13.9)	131(17.1)	284(18.6)
	Moderate	136(31.1)	103(31.2)	239(31.2)	150(34.3)	111(33.6)	261(34)	500(32.6)
	Hard	98(22.4)	84(25.4)	182(23.7)	93(21.3)	77(23.3)	170(22.2)	352(22.9)
	Very hard	76(17.4)	61(18.5)	137(17.9)	81(18.5)	63(19.1)	144(18.8)	281(18.3)
	Extremely hard	27(6.2)	29(8.8)	56(7.3)	28(6.5)	33(10.1)	61(7.9)	117(7.6)
	Total	437(100)	330(100)	767(100)	437(100)	330(100)	767(100)	1534(100)
Discrimination index	Negative	11(2.5)	24(7.3)	35(4.6)	16(3.8)	40(12.2)	56(7.5)	91(6)
	Poor	55(12.6)	22(6.7)	77(10.1)	37(8.9)	36(10.9)	73(9.8)	150(9.9)
	Fair	138(31.6)	75(22.7)	213(27.8)	129(30.9)	116(35.3)	245(32.8)	458(30.3)
	Good	232(53.2)	209(63.3)	441(57.6)	235(36.3)	137(41.6)	372(49.9)	813(53.8)
	Total	436(100)	330(100)	766(100)	417(100)	329(100)	746(100)	1512(100)

Regarding the difficulty index, there were no exams in the simple and extremely hard level. There were nineteen and thirty-five exams in the moderate and hard class, respectively. The findings show that the discrimination index of the exams was fair (thirteen for both free and paying-tuition admission) and good (fifteen for free and fourteen for paying-tuition admission). There were no negative or poor exams for differentiating ability.

Table 2 shows the results of the independent t-test to compare the difficulty index, discrimination index, Kuder-Richardson correlation coefficient, and standard error of measurement between two groups of free-tuition and paying-tuition students.

Table 2 - Independent t-test results to compare exam variables in two groups of free-tuition and paying-tuition students

	Group	Mean	Std. deviation	t value	Degree of freedom	P value	Lower	upper
Difficulty index	Free-tuition	0.624	0.105	0.185	53	0.854	-0.049716	0.059796
	Paying-tuition	0.619	0.097
Discrimination index	Free-tuition	0.327	0.060	2.023	53	0.048	0.000307	0.072524
	Paying-tuition	0.290	0.073
Kuder-Richardson correlation coefficient	Free-tuition	0.610	0.129	0.601	54	0.550	0.039013	-0.054751
	Paying-tuition	0.586	0.161
standard error of measurement	Free-tuition	0.610	0.129	0.602	54	0.550	0.039143	-0.054905
	Paying-tuition	0.587	0.162

According to the findings, the mean discrimination index of questions in free-tuition students was higher than paying-tuition students, and this difference is statistically significant (p = 0.048). The mean difficulty index, Kuder-Richardson correlation coefficient, and the standard error of measurement of the questions were not different in the two groups of free-tuition and paying-tuition students.

Table 3 shows the results of the independent t-test to compare the difficulty index, discrimination index, Kuder-Richardson correlation coefficient, and standard error of measurement of the questions between two independent medical and dental students’ groups.

Table 3- Independent t-test to compare test variables in two groups of medical and dental students

	Group	Mean	Std. deviation	t value	Degree of freedom	P value	Lower	upper
Difficulty index	Medical	0.641	0.095	1.701	53	0.095	-0.008215	0.099899
	Dental	0.595	0.103
Discrimination index	Medical	0.324	0.052	1.802	34.184	0.080	-0.004516	0.075494
	Dental	0.288	0.083
Kuder-Richardson correlation coefficient	Medical	0.593	0.133	0.601	54	0.550	-0.054751	0.101680
	Dental	0.605	0.163
standard error of measurement	Medical	0.593	0.133	0.602	54	0.550	-0.054905	0.102048
	Dental	0.605	0.163

The finding showed that the difficulty index had not differed between the two groups of medical and dental students. According to the results of the Levin test (F = 6.738), the significance level was lower than 0.05 (p = 0.012), so for the difficulty index, the variance of medical and dental groups was not equal. Discrimination index, Kuder-Richardson correlation coefficient, and standard error of measurement of the questions were not different in the two groups of medical and dental students.

Two-way univariate analysis of variance was used to show the interaction between the type of student admission (free and paying-tuition) and the field (medicine and dentistry) on the two variables (the degree of difficulty and discrimination index of the questions). (Table 4)

Table 4- Univariate two-way analysis of variance to examine the interaction between the type of admission and the field in the difficulty index and discrimination index of questions

		df	Mean square	f	P value
Difficulty index	Admission type	1	0.000	0.040	0.843
	Field	1	0.028	2.790	0.101
	Admission type*field	1	0.000	0.014	0.906
	error	51	0.010
Discrimination index	Admission type	1	0.031	9.724	0.003
	Field	1	0.019	6.092	0.017
	Admission type*field	1	0.057	18.117	0.0001
	error	51	0.003

The interaction between the type of admission and the field was not significant on the difficulty index of questions (p = 0.906) but was significant on the discrimination index of the questions (p = 0.0001). Therefore, the mean discrimination index of questions in four groups including the tuition-free medical, tuition-free dental, tuition-paying medical and tuition-paying dental groups was not the same. This was shown in figure 1. The type of admission in the medical and dental groups was different so that the mean of the discrimination index of questions in tuition-free dental students was more than in tuition-paying students in that field.

This study was conducted to compare the item analysis of multiple-choice questions among medical and dental students in two groups of tuition-free and tuition-paying students in a maximum matched condition. The results of the present study generally showed that the item analysis of multiple-choice questions among the free-tuition and paying-tuition students was the same. Also, the results showed that the item analysis of multiple-choice questions was the same among students in medicine and dentistry.

The results showed that only the differentiation index of questions between free-tuition and paying tuition students was significant. As Taib and Yusoff (2014) pointed, multiple-choice questions can distinguish between good and poor students¹⁰. According to the results of this study, the mean differentiation index of questions in free-tuition students was higher than paying-tuition students, which means that multiple-choice questions in free-tuition students were more differentiated between good and poor students than paying-tuition students. Although the number of exams’ questions was the same for the free and paying tuition students, the variable of student numbers was not. The number of free-tuition and paying tuition students was 1731 and 1311, respectively.

In this study, nearly 50% of the questions were considered hard in terms of difficulty. This case indicates that the questions were designed appropriately in the sense of difficulty. Given that the quantitative evaluation of the questions was done under the same conditions, the quality of questions has no relation with the admission type. Nematbakhsh (2015) points out that students' admission type for free or tuition-paying is not an important issue, but how the student graduates are necessary¹¹. Therefore, we conclude that the students' evaluation methods, whether free or with tuition, should be such that it leads to the promotion of education quality.

According to the results, the mean, standard deviation, standard error of measurement, and Kuder-Richardson correlation coefficient of the exams’ showed no statistical significance in the two groups of free and paying tuition students in medicine and dentistry. It seems that the type of student admission and their field do not affect the exam’s results.

The statistical results to examine the interaction between the type of admission and the field in the difficulty index showed that the mean difficulty index was the same in both admission type and field groups. Despite examining the interaction between the discrimination indexes of questions with these variables, the results showed that the mean of discrimination index was different between the groups of admission type and field. This index was higher in tuition-free dental students than tuition-free medical students and tuition-paying dental students.

Two limitations need to be considered for interpretation and future research. First, in this study, the educational objectives in different disciplines were not examined. Although the type of content of the educational program is considered in both medical and dental groups, it is suggested that in another study, other variables such as educational goals in different fields be examined. Second, this study was conducted in one course, which limited the generalizability of the results. Thus, this limitation might be considered during the interpretation of the result.

The type of student admission has no significant effect on students’ assessments in multiple choice exam in matched educational conditions.

Ethics approval and consent to participate

This observational study was confirmed by the institutional review board standards at the [ANONYMOUSED] University of Medical Sciences, [ANONYMOUSED], Iran. No: IR. [ANONYMOUSED]. REC.1397.159. No informed consent was obtained from participants because the publication of the results of this study was anonymous and the need for informed consent was deemed unnecessary according to enactment No: 1397.159 of the review board of [ANONYMOUSED] University of Medical Sciences.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was financially supported by the grant No.269687 from [ANONYMOUSED] University of Medical Sciences. The funding agency had no role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

Authors’ contributions

All authors designed the study. SMH, HS, and RR collected data. SMH, MHK, and VT analyzed data. MHK, SMH, and VT prepared the manuscript. All authors read and approved the final manuscript.

Acknowledgments

This work was done at the [ANONYMOUSED] University of Medical Sciences. Sincere thanks are given to all students who participated in the study.

1. Schripsema NR, van Trigt AM, Borleffs JCC, Cohen-Schotanus J. Selection and study performance: comparing three admission processes within one medical school. Medical Education. 2014;48(12):1201-10.

2. Yang EB. Comparison of competency level of medical, non-medical students and its relevance for admission policy. Korean J Med Educ. 2018;30(3):219-27.

3. Matsumoto M, Inoue K, Kajii E. Characteristics of medical students with rural origin: Implications for selective admission policies. Health Policy. 2008;87(2):194-202.

4. Gallagher J, Calvert A, Niven V, Cabot L. Do high tuition fees make a difference? Characteristics of applicants to UK medical and dental schools before and after the introduction of high tuition fees in 2012. British dental journal. 2017;222(3):181-90.

5. Thomas B. Free Medical School Tuition: Will It Accomplish Its Goals? JAMA. 2019;321(2):143-4.

6. CMJ S. Paying tuition and academic performance of students at the Zagreb University School of Medicine. Croat Med J. 2001; 42:74-8.

7. Janković SM, Tufegdzić BI, Vuković MV, Folić M. Paying tuition and academic performance of students at Kragujevac University School of Medicine. Croatian medical journal. 2002;43(1):63-6.

8. Atuahene F. The impact of “tuition-paying” policy on retention and graduation rates at the University of Ghana. Journal of College Student Retention: Research, Theory & Practice. 2012;14(1):37-65.

9. Bunce L, Baird A, Jones SE. The student-as-consumer approach in higher education and its effects on academic performance. Studies in Higher Education. 2017;42(11):1958-78.

10. Taib F, Yusoff MSB. Difficulty index, discrimination index, sensitivity and specificity of long case and multiple choice questions to predict medical students’ examination performance. Journal of Taibah University Medical Sciences. 2014;9(2):110-4.

11. Nematbakhsh M. Assess a lesson in both public and tuition admission. Iranian Journal of Medical Education. 2015;15(0):189-91.

No competing interests reported.

Download PDF

Version 2

posted

You are reading this older preprint version

Read the latest preprint version →

Comparison of the statistics and item analysis of multiple-choice questions between free and tuition-paying admission

Status:

Version 2

Abstract

Figures

Introduction

Methods

Results

Discussion

Conclusion

Declarations

References

Additional Declarations

Status:

Version 2