Relationship between performance on the progress test and selection for medical residency

The Progress Test (PT) is a formative evaluation used to analyze student progress throughout medical undergraduate studies. Further, classificatory summative assessments are carried out by medical graduates worldwide, either to acquire a practicing license or to obtain a vacancy in the Medical Residency (MR) selection processes. We aimed to identify whether performance in the PT is related to the performance in the MR selection process. The physicians who attended and graduated from a public university in the state of São Paulo, Brazil, between 2011 and 2014 were assessed on the basis of voluntary participation in the PT throughout their undergraduate program, performance in the PT in their last year, performance in the multiple-choice test of the MR and convocation in the first classificatory list, and approval for the MR selection. Descriptive statistics and comparison of scores were performed using the Student t-test for independent and paired samples and the Pearson correlation coefficient.

3 approval in MR among graduates whose mean score on the multiple-choice test was above the mean score in PT (p < 0.0001).

Conclusions
PT performance was related to performance in the MR selection process for the study period; there was a strong correlation between student performance in the PT and multiple-choice test in MR. Students with multiple-choice test results for MR that were below the average performance level in the PT were less frequently approved in the MR selection process.

Background
The learning assessment process should permit students to identify their difficulties in order to help students overcome their shortcomings and allow teachers to use student performance as a tool to improve instruction [1].
In this context, there are three ways of evaluating the teaching and learning process [2].
The first is the diagnosis, or pre-test, performed to obtain the cognitive dimensions of learning that students possess before starting an educational process, and this defines whether students have the basic knowledge and cognitive ability that are necessary to pursue new learning [2].
The second form of assessment is summative, and is applied at the end of the subject or course of study to identify whether students have acquired the skills necessary for the new stages of learning [3]. Summative assessment, traditionally performed by means of tests or discursive questions, is applied at the end of this period of the process [3].
The third kind of evaluation is formative, carried out during a teaching period, and includes the teacher's feedback on the student's performance [4]. It aims at verifying whether students progressively understand each stage of the educational process by identifying the difficulties and learning possibilities in the cognitive construction of 4 knowledge throughout the undergraduate program [5].
The Progress Test (PT), for example, is a formative assessment tool in the medical course of study focused on clinical reasoning, which seeks to identify the student's progress throughout the undergraduate program [6,7]. This assessment allows students to have the necessary feedback about their progress in the teaching and learning processes, while identifying the difficulties and providing the educational institution with input for the evaluation of the curricular model used [8][9][10].
The PT, as a mechanism for improving student performance, establishes new teaching and self-evaluation strategies. It is widely recognized and utilized by medical schools internationally with different teaching and learning methodologies [4,11].
This instrument is an important complement for the assessment of students in different teaching and learning methodologies, from the perspective of a longitudinal approach. The PT is performed at fixed intervals, ranging from one to four times per year [6,7]. It focuses on the evaluation of students' knowledge and competencies over the years and is conducted at the same time by students of different schooling levels. Moreover, the PT can also be used for summative purposes [6,7].
In relation to its construction, the PT began to be produced in a collaborative and multicentric approach, including in its management cost reduction, through simultaneous application in different partner schools [6,7]. When PT is performed multicentrically-in a consortium of several medical schools from the same or different countries-besides cost reduction, efficiency is increased and an extensive database of evaluations is created [6][7][8]. Database consortia such as these provide students with important feedback regarding the construction of knowledge [8,9].
A partnership among schools has contributed to the development of high quality questions, because question writers, reviewers, and test administrators can be shared 5 [12].
Regarding the number of items used in the PT, there are consortia, such as the Dutch and German consortia, that include 200 questions. Others, such as the Canadian consortia, use 180; 120 questions are included on the PT in the United Kingdom. The number of items varies according to the frequency with which the tests are administered; and, since they are very short, they may underrepresent the curriculum contents, reducing the content validity of the PT [6,7]. In Brazil, these tests generally consist of 120 multiple-choice questions, each with four alternative answers of which only one is correct. The total resolution time is four hours [13]. The questions are drafted in accordance with and in the context of medical practice, and their purpose is to evaluate whether students meet the expected levels of knowledge for the practice of medicine and to assess their abilities to apply complex principles of reasoning, reflection, and judgment in the resolution of the questions, thus avoiding the sole measurement of memorized information [12].
However, after completing the course of study, graduates worldwide are faced with summative assessments of a classificatory nature, that is, assessments that enable the successful students to acquire the right to practice the profession or to fill a vacancy in the selection processes, such as Medical Residency (MR) [14].
MR, as a required postgraduate program for physicians, unfolds with in-service training, and is the most important model for the training of specialists [15]. It is considered the gold standard of medical specialization, directed towards the development of medical skills and professional abilities in a qualified work environment [16].
In Brazil, the program accredited by the National Commission on Medical Residency provides the doctor who performs the training with the title of specialist in the area in which he has worked [17]. The average annual workload is 2,880 hours, for a number of years that varies according to the resident's specialty. There are more than one hundred 6 accredited programs, with some allowing direct access, while others require previous training [15,16]. As a large number of recent graduates seek to join MR, there are relatively few vacancies in reputable institutions in comparison to the number of trainees [18,19]. In certain fields, such as in the specialties of neurosurgery, dermatology, ophthalmology, and imaging diagnosis [18], openings for the position are highly coveted by the candidates. In other areas, there are more vacancies offered than are filled by demand from graduates, such as family and community medicine [19].
Thus, when considering MR as the primary training program for specialized medical practice, one must recognize the fact that, over the years, there has been an increase in competition in the selection processes in general. Therefore, it is crucial to verify during the medical course of study whether there is any indication of a particular student performance that correlates with successful performance in the selection process for MR.
Moreover, considering the current importance of the PT in the pedagogical context of medical schools and the test's characteristics, from the form of construction of this instrument to its application, it is necessary to determine the relationship between these two evaluations. One relationship is formative, as the PTs are performed throughout the medical course of study, and the other is summative, focusing on the selection processes for MR, as both evaluate fundamental concepts of conclusion of the medical course.
Therefore, the objective of this study is to identify whether student performance in the PT of the final year of the medical course of study is related to performance in the MR selection process, while specifically focusing on the possible correlation between students' PT performance and their performance on the multiple-choice test (MC) of MR selection; in addition, the study also considers the relationship of PT performance to convocation in the first classification list, which is considered to be approval for MR.

Methods 7
The present study's quantitative research is characterized as an analytical and crosssectional study, in which the total of students from the undergraduate medical school of a public university in the state of São Paulo, Brazil, during the years 2011 to 2014, were analyzed.
The students' performance data for PT and the selection test for MR were requested from those responsible for the information, and stored ethically and confidentially with prior authorization of the participating institution.
The data regarding students' performance in the PT were obtained from the coordinating body of the interinstitutional consortium, which was constituted at that time by nine medical courses of study. The PT of this consortium is applied once a year, and the For the present study, the general performance of students in PT during the last year of the course of study, and the performance on the questions of each of the five major areas -GS, IM, GO, PE, and PH -was considered. The performance in the BS was not considered, since the study correlated the data of the PT with the MR selection test, which did not have specific questions on BS content.
Data on the performance of the MR test were obtained from the coordinating body of the institution's Medical Residency committee.
These selection processes were prepared by professors at the institution itself, and consisted of a test composed of multiple-choice questions (MC), similar to the PT, as well as an essay test and a practical test on the five major areas. The multiple-choice test consisted of 80 questions, and the essay test of 20 questions, and both sets were divided 8 among the common areas chosen for study. The practical test consisted of one station from each area, totaling five stations of practical simulations [20].
The first phase consisted of two tests (MC and essay), with a total weight of 50, distributed as 25 for each. The second phase (practical) test had a weight of 40. The third phase, with a weight of 10, consisted of analysis and argumentation of the curriculum vitae [20]. To participate in the essay test and all other phases, the candidates who occupied the first positions in the MC were called, in the proportion of five candidates per vacancy in each of the programs [20].
In the period studied, the summative tests of the selection processes for direct access to MR selected candidates for the available vacancies among the specialties: anesthesiology, general surgery, medical clinic, dermatology, medical genetics, infectious disease, family and community medicine, occupational medicine, nuclear medicine, preventive and social medicine, neurosurgery, neurology, obstetrics and gynecology, ophthalmology, orthopedics and traumatology, otorhinolaryngology, pathology, clinical pathology/medicine laboratory, pediatrics, psychiatry, radiology and diagnostic imaging, and radiotherapy [20].
In this study, the students' performance on the MC was predominantly selected, since it has characteristics similar to those of the PT. The data were analyzed from the viewpoint of overall performance, as well as the specific performance in each of the major areas.
We identified the numbers of correct answers in the questions of the five major areas of the PT in the last year of medical training, and the MC of the MR selection; and then these numbers were converted into scores from zero to ten, to facilitate the understanding and comparison of performances.
The data of this study were analyzed by means of descriptive statistics, comparison of scores by means of Student's t-tests for independent samples and paired samples, and the

Results
From 2011 to 2014, 424 physicians graduated from the medical school of the institution where this study was conducted. Of these, 374 students (88.2%) participated every year in both the PT course and the institution's MR selection process. Twenty-two students (5.2%) participated in the entire PT undergraduate program, but did not participate in the institution's MR selection process. Twenty-two students (5.2%) did not participate in the six PTs, but did participate in the MR selection process. Six students (1.4%) did not participate in all six of the PTs, and also did not participate in the selection process for MR upon graduation (Fig. 1).
Initially, the performance means on the PT were compared between the students who participated in the MR test when graduating (mean = 6.65) and those who did not participate (mean = 6.20). It was observed that the students who participated in the MR test performed better on the PT than those who did not participate in the MR (p < 0.05) (Fig. 2).
The same comparison was made regarding student performance on the multiple-choice test for MR, between the students who participated in six PTs (mean = 7.36) and those who did not participate regularly in the PT during the course of study (mean = 6.35). The students who regularly participated in the PT presented superior performances on the multiple-choice test for MR, compared to those who did not participate in the six years of the PT (p < 0.0001) (Fig. 3).
The descriptive analysis of the sample is presented in Table 1, considering the general 10 performance in the five major areas as a basis.
As Fig. 4 illustrates, the average performance on the PT of all students in the outgoing classes from 2011 to 2014 was significantly lower than the average performance on the MR multiple-choice test (p < 0.05).
It is also possible to observe that the mean PT performance of all students in this sample was significantly lower than the mean performance on the MC test of MR (p < 0.05), when analyzing the data on outgoing classes from 2011 to 2014 separately (Fig. 5).
Next, the Pearson correlation coefficient was used to identify whether there was a correlation between the students' performance on the PT and the multiple-choice test for medical residency. A regression analysis of the correlation of these data was also performed.
Considering the general data from all students in the sample, it is possible to state that there was a statistically significant correlation (p < 0.001) between these variables (Fig.   6).
Subsequently, the same data correlation analysis was performed separately for each outgoing class from 2011 to 2014, as shown in the correlation matrix in Table 2.
It may be observed that there was a correlation between the overall performance on the PT and on the multiple-choice test for MR in all groups from 2011 to 2014.
We also conducted an analysis of the performance of students in the outgoing classes from 2011 to 2014, in each of the five major areas of medical knowledge-GS, IM, GO, PE and PH-in order to identify whether the correlation between the assessments also occurred among the questions specific to the areas of knowledge.  Table 2. In all areas of knowledge, the Pearson correlation coefficient (r) is weak.
The performance of the students in the PT sample was then analyzed in relation to those performances gathered in the first classification list that were considered approved (having passed the test), and the sample was also analyzed in relation to those who failed in the MR selection process for graduation. As can be seen in Fig. 8, most of the failures demonstrated below-average performance on the PT.
It is observed that, in general, the PT sample students who performed well on the PT also performed well on the MC questions of the MR test upon graduation.
Of the 374 physicians who were selected for MR and who had taken the six PT tests, 253 who obtained a mean on the multiple-choice test greater than or equal to the mean of the PT were approved, and 38 failed. Of those who obtained the lowest average, 30 were approved, and 53 were unsuccessful. There was a statistically significant difference among the performances on the PT, on the multiple-choice test at MR, and on the selection test for MR (p < 0.0001).

Discussion
Similar to other parts of the world, the selection process for accessing available posts for MR is very competitive in Brazil, especially in the renowned institutions. Finding an evaluative parameter for the medical course of study that correlates with the performance on the MR selection test is relevant for training institutions.
This study correlates the performance on the PT with the selection test for MR. The analysis was performed primarily by correlating the scores obtained in the five major areas of the PT with multiple-choice test for MR, because both evaluate knowledge in the specialty fields of GC, IM, PE, GO, and PH through objective questions.
Although the PT and the MR selection tests have different purposes (the first is primarily formative, and the second is summative and classificatory), the two tests aim to evaluate the general training of the physician.
A large majority of graduates in the classes included in this study (93.4%) participated in the process for access to MR vacancies at the institution itself, which is considered one of the best in the country. Of these, moreover, only 5.2% did not take the PT every year. This significant level of participation benefitted the analyses presented in the study.
In any institution where the PT is administered, the aim is to assess the effectiveness of the structure and content of medical curricula, as it is important to identify how students' knowledge progresses during their medical training. However, there is a German study [21] that longitudinally and transversally observed the various impacts on the acquisition of cognitive knowledge among the curricula that use PT, and correlated the evolution of the tests with the serial exams necessary to obtain a professional license. The study showed that medical students who demonstrated ascending growth through administrations of the PT during the undergraduate course of study also had better performance on national exams at the end of the licensing training cycles for the medical profession. The correlation with PT was established, even though there were oral tests at 13 the end of the first and last of the three evaluation cycles [21].
According to the German study [21], the importance of PT should be valued in the construction and development of students' knowledge. However, the authors highlight other influences on high levels of success in such exams, such as the training school, the influence of teachers in knowledge construction, and the efficiency of the school curriculum.
Among the expectations and practical aspects of the PT is the reduction of student anxiety before the summative evaluations [7]. This element was initially highlighted in the research conducted at McMaster's University. This study evaluated the stimulus that undergraduates presented when studying continuously to prepare for the medical exams held at the end of the course of study, as well as to prepare for the licensing exams for becoming professionals (which are given in several countries) [7]. There is no documentation of the same effect on the performance of screening tests for MR, although our results point to better performance on the MR multiple-choice test among students who took their PT regularly throughout the course of study.
Our study also demonstrates that there was a correlation between the interinstitutional tests prepared by teachers of nine medical schools and the tests for selection into medical residency prepared by teachers of the testing institution.
This study was conducted in a Brazilian medical school and shows that students who participated regularly in the PT, consisting of 88.2% of the graduates of the four groups studied, had superior performances on the multiple-choice test for MR, compared to those who did not participate regularly. Those with regular participation showed higher means on the MR multiple-choice test when compared with the PT tests, even when compared according to specialty areas, corroborating the German study that refers to the influence of the PT on performance for exams taken afterwards. It is likely that other factors, such as greater dedication to selective evaluation and dedication to studies in the period between the PT and the selection process for MR, are also related to better performance on the multiple-choice test for MR.
In addition to these limitations, the influence of preparatory courses for MR selection tests could interfere with the performance on these tests, but this aspect was not analyzed.
Another relevant aspect of this study was that students who annually performed in all PTs Correlating performance between PT and MR tests may also contribute to the improvement of curricular education, as both tests are intended to assess the general training of the physician.

Conclusions
The performance on the PT in the last year of medical training by students who had participated in this evaluation regularly throughout the course of study was related to performance in the MR selection process. There was a strong correlation between overall student performance on the PT and the MR multiple-choice test, with students who demonstrated below average performance on the PT being less successful in the MR selection process.

Availability of data and material
The data that support the findings of this study are available from School of Medical Sciences (FCM), University of Campinas (UNICAMP) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of UNICAMP.

Competing interests
The authors declare that they have no competing interests.

Authors' contributions
All authors contributed equally in the conception of the work, in the interpretation of the data, in the analysis of the data, in the writing of the work and approved the final version submitted for publication.      Progress test and multiple-choice test performances among those who passed or failed residency tests