Analyzing Answering Patterns in Progress Test Data

doi:10.21203/rs.3.rs-1738174/v1

Download PDF

Research Article

Analyzing Answering Patterns in Progress Test Data

https://doi.org/10.21203/rs.3.rs-1738174/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 29 Mar, 2023

Read the published version in BMC Medical Education →

You are reading this latest preprint version

Background: The Progress Test Medicine (PTM) is a 200-question formative test administered to approximately 11,000 students at medical faculties (Germany, Austria, Switzerland) each semester. Participants receive feedback on their knowledge (development) mostly in comparison to their own semester. In this study, we use the data of the assessment to find groups with similar answer patterns.

Methods: We performed k-means clustering with a dataset of 5,444 participations, selected cluster number k=5, and answers as features. Subsequently, the data was passed to XGBoost with the cluster assignment as the target enabling the derivation of cluster-relevant questions for each cluster with SHAP. Clusters were examined by total score, answer patterns, and confidence. Relevant questions were evaluated for difficulty, discriminatory index and confidence levels.

Results: Three of the five clusters can be seen as ’performance clusters’: cluster 0 (n=761) consists predominantly of participants in higher semesters. Relevant questions tend to be difficult, but participants answer confidently and correctly. Participants in cluster 1 (n=1,357) are amidst their study, cluster 3 (n=1,453) consists mainly of early study stage participants. Guessed questions increase successively, relevant questions are rather easy. Two clusters are mostly ‘leaver clusters’: participants in cluster 2 (n=384) drop out of the test about halfway through after initially performing well. Cluster 4 (n=1,489) includes students from the first semesters - but also "non-serious" participants with mostly wrong guesses or no answers.

Conclusion: Clusters dissolve the allocation to one's own semester and place performance in the context of the overall cooperation.

progress test

unsupervised machine learning

supervised machine learning

participant groups

clustering

k-means

classification

ensemble learning

boosting algorithm

explainer

Progress Testing: Conditions and Feedback

Progress Testing is a cross-sectional and longitudinal assessment that provides a unique and demonstrable measurement of the growth and effectiveness of students’ knowledge (1). Students receive tests on a regular basis during their whole course of study. Question sampling follows a fixed content blueprint with questions at graduate level resulting in different but comparable tests (2,3). The longitudinal nature of the test allows to monitor a student’s progress until graduation. The cross-sectional nature allows comparisons between students within the same cohort and between cohorts or institutions since every student receives the same test. As such, progress tests are a rich source of feedback for individuals, cohorts and institutions (4–7).

In Germany, progress tests have been used in medical education since 1999 when the Charité - Universitätsmedizin Berlin (Charité) and the University of Witten/Herdecke, introduced the ‘Progress Test Medizin’ (PTM) jointly. Nowadays, the PTM consortium delivers a progress test to around 11,000 students of 17 faculties in Germany, Austria and Switzerland each semester. The PTM is purely formative and consists of 200 multiple-choice questions. The selection is based on a two-dimensional blueprint where the questions are mapped according to the organ system and medical subject they are most closely related to (8). These questions are then randomly sorted and all participants receive them in the same order. Students have 180 minutes to complete the test. Depending on their knowledge and motivation, participants select the questions they answer and may skip some or even all of them. Afterwards, all participants receive a detailed individual cross-sectional and longitudinal feedback. To counter any effects different curricula might have on detailed analysis of subdomains, the respective feedback is provided based on the own cohort. More global reporting of aggregated results includes the peers among institutions, and a synopsis as global benchmark against all participants of prior tests. This feedback is based on numerical scores expressed as percentages, total numbers and rankings within the comparison groups (1,4,9,10). Figure 1 depicts the percentage of correct answers grouped by semester.

Starting spring of 2018 about half of the participating faculties have added certainty-based (or "confidence-based") marking to their examination environments. Participants report the trust in their own answer as certainty on a 3-level Likert scale (11).

Previous work suggests that starting from semester 2. medical students self-monitor their knowledge well, but not perfectly. This includes the selection of questions to be answered as well as the self-reported confidence of the answer (11,12). However, the PTM feedback does not yet include the individual confidence levels in the joint metrics and parameters. They only serve as additional information in the reports sent to the participants. In any case, further incorporation of this marking into the analysis of results is gradually being implemented. Additionally, surveys have shown that participants in the PTM desire more personalized feedback focusing on their strengths and weaknesses, but also to compare their performance to that of the whole set of participating faculties (13,14).

Challenges of low stake assessment

In low stakes assessment such as formative testing, a low test-taking effort usually does not lead to personal consequences. However, low examinee effort might result in underestimating the actual performance and level of proficiency of individual test-takers, which in turn leads to negatively biased overall scores that might put the validity of test outcomes into question (15,16). To avoid imprecise feedback, it is necessary to differentiate between insufficient knowledge and low test-taking effort and exclude ‘non-serious’ test takers from the traditional faculty and cohort comparisons. The PTM consortium has addressed this problem by adopting a set of criteria based on the work of Schüttpelz-Brauns et al. (17) and Karay et al. (18). However, detection of ‘low test-taking effort, but sufficient knowledge’ is still lacking.

Aims

When looking at Figure 1, clear separations between neighbouring semesters are not present; fluent transitions can be seen instead. This, along with fulfilling the students’ wish of comparing their performance to that of the whole set of participating faculties, led us to the idea of detecting groups based on the knowledge of participants instead of focusing on semesters. Furthermore, confidence-based marking and ‘low test-taking effort’-‘sufficient knowledge’ detection is still lacking. We thus explored an additional analysis method to identify so far unknown underlying patterns. Participants should be linked together without arbitrary restriction such as test-taking effort (‘seriousness’), semester peers, and curricular differences. We aimed to find groups of participants by deriving answer patterns using answer status, correctness and confidence of all questions. To expand our understanding of these groups, we then derived the questions that had the biggest effect on the grouping and examined them more in depth regarding intended competency level, difficulty index and discrimination index.

Educational data mining approaches have been shown to help in finding underlying structures in educational data (19). As our classification was to be independent from previously used cohorts, clustering by unsupervised learning came as an obvious option. The purpose of clustering algorithms is to identify groups of objects, or clusters, that are more similar to each other than to other clusters, seeking to provide more insights about complex data (20). Clustering has been used to group students according to their level of competence, mainly to foster their learning (21,22).

Clustered data can subsequently be classified with the retrieved clusters as targets followed by an explainer algorithm to gain further information about the important features. Classification may be performed by a boosting algorithm, that combines simpler models with limited predictive ability each, producing only weak hypothesis (23,24). In this context, weights are readjusted in such a way that misclassified input data gains (mostly) a higher weight, while correctly classified data loses weight. Thus, future weak learners focus more on the examples that previous weak learners misclassified. In this way boosting algorithm form a relatively more accurate model than each weak learner on its own (25,26). Gradient boosting trees have for example been applied in analysis of massive open online courses to derive the most important features for success or failure of the participants (27). This can be followed by explainers like SHAP (28) allowing the derivation of feature importance per class.

In summary, we aim discover unknown underlying patterns, using the correctness and confidence of the answers, without the restriction of the faculty or the semester, and the presumed seriousness of the participation. For that, we apply unsupervised clustering followed by supervised classification and explanation. In the end, we retrieve new information about differently performing participant groups and relevant questions that distinguish the groups.

In this section we introduce the pipeline we followed to detect groups based on test responses. Figure 2 gives an overview of the analysis workflow.

Data Pipeline

Input Data

For this analysis, faculties of the PTM consortium using confidence marking were asked to participate in this study of which 8 faculties agreed. Therefore, we used PTM data from winter semester 2020 containing 5,852 PTM participations associated with eleven semesters. Participants were anonymised, faculties pseudonymised. From the general analysis of the PTM of the winter semester 2020, we obtained the seriousness of every participation.

PTM data include a matrix where for each test participation (rows) the marked answer to each of the 200 questions ordered by their appearance in the test (columns) is present. Marked answers are composed of correctness and confidence. Firstly, questions could either be answered correctly or incorrectly, or not answered at all. Secondly, the participants indicated their confidence level in each answer as: I am very sure – ‘sure’, I am fairly sure – ‘likely’, or I am guessing – ‘guessed’ (11).

Additional question information are test specific difficulty and discrimination index of each question. The difficulty index (correct [%]) is the percentage of participants that answered the question correctly (29). For clarification, the higher the difficulty index, the easier was the question. The discrimination index shows how well a question separates the high scoring participants in the test from the lower scoring. The discrimination index ranges from -1 to 1. An excellent discriminating question usually has a discrimination index above 0.35(30).

In addition to the blueprint mapping categories, each question in the PTM is also categorized with one of two competency levels (‘apply’, ‘recall’). ‘Apply’ questions include a brief clinical or laboratory vignette that needs to be interpreted or analysed (29), while ‘recall’ questions aim to test the knowledge of participants regarding a subject.

Preprocessing

In the final analysis data set, we only included participants from semester one to ten, who answered at least one question. Participants of the eleventh semester were excluded since their participation is exclusively voluntary and most of the participating faculties do not offer the PTM beyond the 10th semester.

For clustering, confidence and correctness had to be translated to metric values. We assigned initial scoring as 3 points for certain answers (‘sure’), 2 for probable (‘likely’), 1 for guessed and scored positively for correct and negatively for incorrect in each case. We then adjusted the scoring to keep the relative distances of the mean percentage correct answers (difficulty) for each certainty (Appendix Table 3). Accordingly, ‘sure’ answers are scored with + (correct) or -3 points, ‘likely’ ones with + or - 1.8, and ‘guessed’ ones with + or - 0.6 respectively; unanswered questions are scored with 0. The total score of a participation is calculated by summing up the corresponding scores for all 200 questions. Hence, total scores can lie between -600 and 600. These scores will, however, only be used for comparison in the context of this study.

Analysis Pipeline

Algorithms

In our analysis setting, each participation is an observation, with the metric values of the translated confidence and correctness of each question as features. The applied pipeline was implemented in Python (version 3.8.3; Van Rossum & Drake, 2009) and consisted of three main algorithms: First, we used an unsupervised method for detection of underlying patterns. We then trained a classifier with the clustered data and applied an explainer-algorithm on these results to extract the relevant features that distinguish each cluster form the others. The resulting accuracy of the classifier was additionally used as evaluation parameter for the clustering algorithm.

Clustering. We used the unsupervised learning algorithm k-means for clustering of our data. Observations (participations) were clustered on the basis of the combined confidence and correctness values. A centroid point represents each cluster and every observation will belong to the cluster with minimum Euclidean distance to the centroid (32). As the first step, the optimal number of clusters k with respect to the data needs to be determined. The number of clusters should yield stable, meaningful, interpretable clusters. Interpretability and meaningfulness might decline with too few, but also with too many clusters (33). We determined the number of clusters mathematically using the elbow-method with the Euclidian distance as the distortion score. The function KElbowVisualizer from the Yellowbrick package (34) returns the optimal cluster size k for an examined range of potential cluster sizes. We evaluated the clusters based on returned k for interpretability before deciding on the final k. As k-means is known to converge to local minimum (35), we run k-means 200 times with our final k. We then chose the run with the best Calinski-Harabasz score (CHS). CHS represents the within versus between cluster dispersion ratio. The higher the score, the better the clusters are separated (36). Additionally, the accuracy of the classifier further down the pipeline served as an additional performance measurement (see below).

Classifier. Same as in the clustering step, we used the metric values of the translated confidence and correctness of each question as features for the classifier. We performed multiclass classification (37) in which the class to predict is the cluster resulting from the previous algorithm. For that, the dataset was spilt into a training dataset with 75 % of the data and testing dataset with 25% using sklearn (38).

For classification, we used the GradientBoostingClassifier XGBoost (39). The default gbtree served as booster and multi:softprob served as learning objective for the prediction of each data point belonging to each class. The learning rate was set to 0.2 and early stopping selected to prevent overfitting. Remaining parameters were left as default. As evaluation metric we used mlogloss, which returns the logistic loss in a multiclass dataset (37,38). With the testing dataset, the performance of the trained model was evaluated using the overall accuracy as performance metrics. This classifier step was repeated 100 times with random training-test-splits. The run that yielded the highest accuracy was served to the explainer.

Explainer. We used the TreeSHAP method of the SHAP library to estimate the relevance of the features (40). The average of the absolute SHAP-values returns the global importance of each feature for each class (41). We selected the 20 most relevant questions according to their SHAP-values based on the answer-patterns of the participation belonging to the respective class. We related these findings to the position in the test, difficulty and discrimination index, and the mapping to the intended competency of each question.

Preprocessed Dataset

408 participants did not meet our above mentioned inclusion criteria and were removed during preprocessing. Hence, our preprocessed final dataset included participations from 5,444 students from eight faculties. The number of students per semester ranged between N=383 in semester 6 to N=942 semester 3 (Appendix Table 1). Most participations came from one university (N=3,077), while participations from the other seven faculties ranged from N=152 to N =526 (Appendix Table 2).

From the 200 questions, 90 were of competency ‘apply’ and 110 of ‘recall’. The mean ± standard deviation (std) of the difficulty index and discrimination index were 34.08 ± 17.12 and 0.42 ± 0.12, respectively.

Clustering

Based on the elbow method, the optimal cluster number was k=5 (Appendix Figure 1). With this cluster size, k-means was run 200 times (descriptive statistics: Appendix Table 4) and the best run with a CHS of 215.077 selected. The corresponding cluster assignments were chosen for further analysis.

With this final cluster assignment, all clusters contain participants from all ten semesters. However, each cluster shows a tendency to peak at certain semesters. Three of the clusters are characterized by different levels of performance, while two clusters show a mixture of performance and ‘early termination’ of the exam. In the following, we present descriptive information of the clusters, first for the ‘performance’ clusters and second for the ‘early termination’ clusters. In-depth visualisation of the clusters can be seen in Figure 3 (semester distribution, total score) and Figure 4 (heat maps, boxplots) with exact values in Appendix Table 5. Appendix Figure 2 and Appendix Figure 3 show cluster-semester associations.

The performance cluster are clusters 0, 1 and 3 as seen in Figure 3. Cluster 0 (n=761) includes participants who are primarily in their final semesters. These participations have the highest mean total score (± std) of 243.45 (± 48.95); they also show a high proportion of correct answers labelled as ‘sure’ (Figure 4, Appendix Table 7). Cluster 1 (n =1,357) are participants who are primarily in the middle of their studies, and cluster 3 (n=1,453) are those who are in their first semesters, with a peak at third semester – which is the semester with most participants in this test (Appendix Table 1). These clusters have a mean score of 135.23 (± 33.87) and 42.60 (± 33.85), respectively.

Cluster 2 (n=384), the smallest cluster, contains primarily participants of later semesters who drop out after about halfway through the test, although the questions were randomly allocated. The mean total score of this cluster is 75.68 (± 41.85). Cluster 4 (n=1,489) mainly contains the lowest scoring participants, that is, participants who fail to answer a sufficient number of questions with the accuracy and certainty needed to achieve a higher score. They either leave many questions unanswered or provide guessed responses to most questions in case they fully participate in the test. The peak is at the first semester (Appendix Figure 2), which serves in most cases as a baseline test. This cluster also includes 590 of the 596 test takers classified as non-serious. The mean total score is -18.09 (± 43.25).

Considering their very high number of unanswered questions compared to that of clusters 0, 1, and 3, cluster 2 and 4 may be seen as ‘leaver’ clusters. Additionally, cluster 4 also includes a high amount of mostly guessers. However, for easier distinction we refer to these two clusters as ‘leaver’-clusters for the rest of the paper.

The main difference between the ‘performance’-clusters and the ‘leaver’-clusters is the number of unanswered questions. The ‘performance’ clusters average 5.94 unanswered questions for cluster 0, 12.67 for cluster 1 and 13.49 for cluster 3, while the two ‘leaver’ clusters average 103.71 unanswered questions for cluster 2 and 97.08 for cluster 4.

Aggregated results per cluster and confidence level show that participants of the first three clusters (cluster 0, 1, 2) show an above-average accuracy for all confidence-levels (Figure 5, Appendix Table 6). Cluster 3 shows an almost-average level of accuracy. Cluster 4 shows lower-than-average mean of accuracy with a comparably high variance.

Classification

The splits consisted of 4,083 observations for training and 1,361 for testing. Splitting was performed 100 times with a consecutive training of the boosting algorithm. The model accuracies ranged between 0.855 and 0.899 with a median of 0.876. We kept the model with the highest accuracy for further analysis (0.899). We also acquired the corresponding weighted F1-scores (42), representing the harmonic mean between weighted precision and weighted recall, which ranged from 0.856 to 0.899. The selected models’ weighted F1-score was also 0.899. Appendix Table 8 and Appendix Table 9, and Appendix Figure 4 show precision and recall in addition to before mentioned metrics for all 100 runs and the selected model.

Explainer

Figure 6 and Figure 7 show the difficulty and competence level of the relevant questions in detail. For clusters 0 and 1, the relevant questions are mainly correctly and surely answered questions. They differ in difficulty – and competency level. Of the 20 most relevant questions that distinguish cluster 0 from the other clusters, 65% address the competence level ’apply’, only 35% ‘recall’. This is noteworthy because in this PTM, 45% of the questions are ‘apply’ questions and 55% are ’recall’ questions. The 20 most relevant questions show above-average discriminatory power and are slightly more difficult than the mean difficulty of the overall test. For the relevant questions in cluster 1, ‘apply’:‘recall’ is reversed (35%:65%). These questions are on average easier than medium difficulty of the overall test, but above average in discriminative power. The relevant questions of cluster 3 are above average in ease and discrimination index. However, the answer pattern is different: Participants answered these questions partially correctly, partially incorrectly, and predominantly guessed. Here the ratio ‘apply’:’recall’ corresponds to cluster 1 (35%:65%).

The relevant questions for the two ‘leaver’ clusters show different characteristics. Again, these questions span the entire test. However, for cluster 2, 35% of the relevant questions are in the second half of the test and their common distinguishing feature is that participants of cluster 2 did not answer them. The remaining 65% of the relevant questions are in the first half of the test and were mostly answered correctly. Since in this cluster mainly the position in the test is relevant, an exact consideration of the degree of difficulty, the discriminating power and the competence level is irrelevant at least for the last questions. The detection pattern for the relevant questions of the last cluster (cluster 4) consists mainly of the fact that participants did not answer or guessed wrongly above-average easy questions. The respective SHAP-values for all questions per cluster can be seen in Appendix Table 10.

The importance of low stakes tests is established in education systems worldwide (43). It offers feedback for students and faculties on current knowledge and curricula, respectively. However, a common challenge of low stakes test is a potentially reduced test-taking effort as it not only influences the validity of the test data (44). but also, directly effects the feedback. In combination with the wish to include PTM-wide comparison in our analysis, we therefore wanted to explore patterns in the answer behavior of PTM participants in order to find grouping of participants beyond test-taking effort, semester peers, and possible curricular differences.

For that, we used a cluster algorithm for underlying pattern detection followed by a boosting classifier. At the end, we pass the obtained model from the previous step to an explainer that calculates the relevance of each feature for each class.

Parameters and Performance Measurements

We mainly used preset parameters and did not adjust the hyperparameters. Instead, we took the best result from 200 (cluster) and 100 (classifier) runs for further analysis. We chose this approach with the ulterior motive of adopting the pipeline into the standard analysis of the test. There, tuning the hyperparameters for each test is not feasible. The performance measurements for the multiple runs of the cluster- and classifier-algorithms gave values in close and appropriate ranges. The CHS for the k-means runs differed by only 2.5 with a median of 215.04. The classifier accuracy ranged from 85.5% to 89.9% and the F1-score ranged from 85.6% to 89.9%, with 89.9% for both scores in the final model. Given an accuracy of about 90% suggests that the classifier is able to learn well the assignment of the answer patterns to the corresponding clusters. We therefore concluded that the analysis pipeline gives meaningful and interpretable results without hyperparameter tuning.

Cluster

Our clustering yielded three ‘performance’ clusters and two ’leaver’ clusters; performance clusters are loosely structured around groups of study semesters, each one corresponding to a certain segment of medical studies. These groups offer an alternative to traditional cohort-based feedback, where participants are compared exclusively with peers of the same faculty and semester. Specifically, cluster 0 contains mainly participations from the end of the studies (semesters 9 and 10), while cluster 1 focuses on semesters 5 to 7, and cluster 3 on semesters 3 to 5. All relevant questions of the ‘performance’ clusters had a discrimination index above 0.35 (excellence cut-off) and almost all exceeded the test average (0.42).

The main difference between the relevant questions in cluster 0 and 1 was the question difficulty. Relevant questions for cluster 0 were slightly more difficult than the average difficulty of the entire test (cluster 0: 33.69% versus PTM: 34.64%), while the relevant questions for cluster 1 were slightly easier than the overall average. The participants in both clusters answered the respective questions with a high confidence predominantly correct.

However, at this point, it is important to note, that ‘relevant questions’ should always be considered together with the corresponding answer patterns. For example, relevant questions in cluster 3 showed a slightly higher average difficulty than the relevant questions in cluster 1, however Figure 6 shows that participants of cluster 3 answered the easier questions with high confidence correct and guessed the difficult ones wrong.

In summary, the combination of high discrimination index, question difficulty and respective answer pattern supports our claim that these questions serve as good group separators and support our ‘performance’ cluster groupings further. Although the separation between clusters might be somewhat fluent, these indicators describe the respective main group per cluster well.

Cluster 2 consists primarily of participants in the later stages of their studies. From their partial answer patterns, we can infer that most of them would have probably achieved scores comparable to those of the two upper performance clusters if they had completed the 200 questions. The graphs in Figure 7 additionally support this claim, as they show similar accuracy in confidence for participants in cluster 0, 1 and 2 for the questions that the respective participants answered. This means that the participants in cluster 2 showed similar answer characteristics to cluster 0 and 1 for their answered questions.

The answer patterns of the most relevant questions of cluster 2, namely 12 mostly correctly answered within the first 35 questions and 7 questions not answered at the second half of this PTM, presumably, could be responsible for the fact that the majority relevant questions for clusters 0, 1 and 3 are behind the first quarter of the test. Questions may be answered differently when they are positioned earlier in the test than at a later time. These positions for the relevant questions and certain answer patterns leads us therefore to believe, that the position a question has, should be seen as a feature in future analysis.

Cluster 4 must ultimately be considered in a differentiated way. The mean score is the lowest of all clusters with -18.09 and relevant questions have the unique characteristic of not being answered or wrongly guessed. However, labeling the whole cluster as ‘leavers’, is unfair to a high amount of allocated participations, because it contains in addition to all non-serious test takers, except six (N=590 of 596 overall), 61.09% of first semester students who mainly guessed. This cluster therefore serves partially as a lower ebb of cluster 3. For the participants of the first semester, the respective progress test can be considered the ‘baseline’, as it takes part at the beginning of the semester, and thus of their study. They do respond to many questions, but in a cautious way. Hence, the rather low scores in this cluster are not surprising. When looking at the line plots in Figure 7, self-assessment accuracy of cluster 4 stands out. The mean value is lower than average for each confidence level, but the variance is much higher than for the other cluster, again leading to the assumption that this cluster contains two different groups of participants. The separation of cluster 4 into two separate clusters, one with non-serious participants and one with ‘beginners’ would most likely require a different scoring. However, our goal was not to find non-serious test takers, but to gain a new view on our participations and group together similarly performing participants concerning the PTM-wide picture. A more in-depth look at this cluster might be of interest for further research in the detection of non-serious test taker by gaining insights into the differences between highly motivated, yet little medically knowledgeable early semesters versus low motivated higher semesters.

Inclusion in PTM Feedback

The integration of the pipeline and the resulting clusters into the PTM analysis can complement the current feedback with a global estimate, for both – students and faculties. Students ask for additional detailed comparisons including all participating faculties, beyond their own comparison cohort (their own semester at their own university) (14). Placing students in clusters based on their answer patterns is the first step in this direction. Great care has to be taken to include all information necessary to explain the results in a comprehensible way. With the help of the heat maps (Figure 4), participants can visually compare their answer patterns with the answer patterns of all clusters and thus get a better overview of their current level of knowledge. For example, participants in cluster 2 can estimate their performance on the first half of the test better than they can solely with standard numerical feedback and averaged scores. Additionally, by classifying all participants by clusters, faculties can get feedback on participants’ performance that is independent of the semester-dependent curriculum. Grouping similar performing participants can help to construct a profile focusing on their strengths and weakness.

Limitations

The influence of the imbalance in participation numbers of the faculties and semesters needs further analysis. The largest participating institution administers the test every semester and admits new students twice a year, as opposed to other faculties where enrollment only takes place at the beginning of the winter semester. Smaller faculties show a much more uneven distribution of test takers among semesters and not all faculties offer the PTM every semester. It would not have been possible to address the semester-related and the faculty-related imbalance, and still have the answer structure of the respective PTM. As the goal was to group similar participations within one PTM run, we thus believe it is necessary to keep the original structure of the dataset in this respect.

In the original data, each participants’ answer is represented by an identifier composed of the confidence of the given answer and its correctness. For clustering, we translated these categories to numerical values knowing that this transition has possibly an impact on our clustering results. Hence, our scoring assignment was based on a mathematical background, and as argued above, we find the resulting clusters reasonable and interpretable for our aim.

Future Research

In this study, we only included the participants’ scores in our analysis. A future research goal would be to include other variables to adjust for certain distractions. One feature to include could be for example the time a participant spends on a question. These inclusions could help to find new cluster indicators and possibly give us even deeper knowledge on the differences of certain participant groups.

Longitudinal analysis of both the participants and relevant features is of high interest as well.

The length of stay of participants in one cluster and the transition to another during their study could give information on knowledge growth of participants and derive developmental patterns. One usage example would be identification of those in need of support, possibly by early indicators.

Performing an analysis on relevant features of past PTMs could offer further insights into specific characteristics and identify relevant question groups.

The inclusion of confidence-based marking in the PTM and the students’ wish to compare themselves on a PTM-wide base, motivated us to develop an additional workflow incorporating these new additions to our already existing PTM analysis pipeline and feedback. We applied k-means, followed by XGBoost and TreeSHAP. We found at least 3 different performance groups and groups of participants leaving the respective PTM early. The participants in the groups differed among others in study stages, knowledge on easier and more difficult questions, and confidence in their answers in combination with their self-assessment accuracy.

Conclusively, we plan on incorporating these cluster findings into our currently existing feedback pipeline enabling participants and faculties to receive feedback on the respective or all participants’ performance in contrast to that of the confidence assessing faculties of the PTM cooperation.

PTM – Progress Test Medizin

Ethics approval and consent to participate

All methods were performed according to relevant guidelines and regulations. Regarding the usage of data about student performance in Progress Tests, we also refer to the local university law (BerlHG; §6) and the local examination regulations. The use of the anonymised participation data was approved by the Ethics Committee of Charité - Universitätsmedizin Berlin (14.04.2020, EA1/030/20). The need for written informed consent was waived by the before mentioned Ethics Committee of Charité - Universitätsmedizin Berlin (14.04.2020, EA1/030/20). because this test is in accordance with the examination regulations under local university laws.

Consent for Publication

Not applicable

Availability of data and materials

The datasets generated during and/or analyzed during the current study are not publicly available for data security reasons but are available from the corresponding author on reasonable request and after approval of the Progress Test cooperation partners and an extended ethical approval.

Competing interests

The authors declare that they have no competing interests.

Funding

The research project was funded by the German Ministry of Education and Research (Grant Numbers: 16DHB4008, 16DHB4009).

Acknowledgements

We would like to thank Dr. Jochen Kruppa for his feedback and the support in this project.

Authors’ contributions

MM & MS outlined the concept and design of the article. MM & MS developed the study design, prepared and analyzed the data. MM, MS, VS, PM, MTT & IRA discussed and evaluated the results. MM, MS, VS, PM, MTT & IRA drafted the introduction section. MM, MS, MTT & IRA drafted the methods and results. MM, MS, VS, PM, MTT & IRA drafted the discussion and conclusion section. MM, MS, VS, PM, MTT & IRA contributed to the manuscripts’ revision. MM, MS, VS, PM, MTT & IRA read and approved the final manuscript.

Author details

¹Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, AG Progress Test Medizin, Charitéplatz 1, 10117 Berlin, Germany.

²Technische Universität Ilmenau, Fakultät für Informatik und Automatisierung,

Data-intensive Systems and Visualization Group (dAI.SY),

Helmholtzplatz 5 (Zusebau), 98693 Ilmenau, Germany

Wrigley W, Van Der Vleuten CP, Freeman A, Muijtjens A. A systemic framework for the progress test: Strengths, constraints and issues: AMEE Guide No. 71. Med Teach. 2012;34(9):683–97.
Freeman A, Van Der Vleuten C, Nouns Z, Ricketts C. Progress testing internationally. Med Teach. 2010;32(6):451–5.
Schuwirth LWT, van der Vleuten CPM. The use of progress testing. Perspect Med Educ. 2012;1(1):24–30.
Coombes L, Ricketts C, Freeman A, Stratford J. Beyond assessment: Feedback for individuals and institutions based on the progress test. Med Teach. 2010;32(6):486–90.
Muijtjens AMM, Schuwirth LWT, Cohen-Schotanus J, Van Der Vleuten CPM. Differences in knowledge development exposed by multi-curricular progress test data. Adv Heal Sci Educ. 2008;13(5):593–605.
Schmidmaier R, Holzer M, Angstwurm M, Nouns Z, Reincke M, Fischer MR. Using the Progress Test Medizin (PTM) for evaluation of the Medical Curriculum Munich (MeCuM). GMS Z Med Ausbild. 2010;27(5):Doc70.
Tontus Omer H., Ozlem Midik. Evaluation of Curriculum by Progress Test. J US-China Med Sci. 2017;14(6):232–40.
Nouns ZM, Georg W. Progress testing in german speaking countries. Med Teach. 2010;32(6):467–70.
Schauber S, Nouns ZM. Using the cumulative deviation method for cross-institutional benchmarking in the Berlin progress test. Med Teach. 2010;32(6):471–5.
Muijtjens AMM, Timmermans I, Donkers J, Peperkamp R, Medema H, Cohen-Schotanus J, et al. Flexible electronic feedback using the virtues of progress testing. Med Teach. 2010;32(6):491–5.
Kämmer JE, Hautz WE, März M. Self-monitoring accuracy does not increase throughout undergraduate medical education. Med Educ. 2020;1–8.
Smrkolj Š, Banˇ E. The Reliability and Medical Students ’ Appreciation of Certainty-Based Marking. 2022;
Langenbeck S, Stroben F, März M, Verba M, Werner S. 19 Jahre Progress Test Medizin (PTM) in Berlin: Welchen Nutzen ziehen Medizinstudierende aus dem PTM? – Suche nach motivationalen Anreizen für Medizinstudierende, den PTM ernsthaft mitzuschreiben. In: Jahrestagung der Gesellschaft für Medizinische Ausbildung. 2018.
Sehy V, Struzena J, März M. Wie wünschst du dir dein Feedback? Das neue Wissensprofil des Progress Test Medizin. In: Jahrestagung der Gesellschaft für Medizinische Ausbildung (GMA). 2020.
Wise SL, DeMars CE. Low examinee effort in low-stakes assessment: Problems and potential solutions. Educ Assess. 2005;10(1):1–17.
Wise SL, DeMars CE. Examinee noneffort and the validity of program assessment results. Educ Assess. 2010;15(1):27–41.
Schüttpelz-Brauns K, Hecht M, Hardt K, Karay Y. Institutional strategies related to test – taking behavior in low stakes assessment. Adv Heal Sci Educ [Internet]. 2019;(0123456789). Available from: https://doi.org/10.1007/s10459-019-09928-y
Karay Y, Schauber SK, Stosch C, Schüttpelz-Brauns K. Computer Versus Paper—Does It Make Any Difference in Test Performance? Teach Learn Med. 2015;27(1):57–62.
Romero C, Ventura S. Educational data mining: A review of the state of the art. IEEE Trans Syst Man Cybern Part C Appl Rev. 2010;40(6):601–18.
Rodriguez MZ, Comin CH, Casanova D, Bruno OM, Amancio DR, Costa L da F, et al. Clustering algorithms: A comparative approach. Vol. 14, PLoS ONE. 2019. 1–34 p.
Harimurti R, Buditjahjanto IGPA. Integrating k-means clustering into automatic programming assessment tool for student performance analysis. 2021;22(3):1389–95.
Pang Y, Xiao F, Wang H, Xue X. A Clustering-Based Grouping Model for Enhancing Collaborative Learning. In: 2014 13th International Conference on Machine Learning and Applications [Internet]. IEEE; 2014. p. 562–7. Available from: http://ieeexplore.ieee.org/document/7033177/
Kearns M. Thoughts on hypothesis boosting. Machine Learning class project. 1988.
Schapire RE. The Strength of Weak Learnability. Mach Learn. 1990;5(2):197–227.
Schapire RE, Freund Y. Boosting: Foundations and Algorithms. The MIT Press. 2012.
Nielsen D. Tree Boosting With XGBoost Why Does XGBoost Win “Every” Machine Learning Competition? Norweigian University of Science and Technology; 2016.
Vie J-J, Popineau F, Bruillard É, Bourda Y. A Review of Recent Advances in Adaptive Assessment. In: Learning Analytics: Fundaments, Applications, and Trends. 2017. p. 113–42.
Lundberg SM, Erion GG, Lee S-I. Consistent Individualized Feature Attribution for Tree Ensembles. [csLG] [Internet]. 2018 Feb 11; Available from: http://arxiv.org/abs/1802.03888
Case SM, Swanson DB. Constructing Written Test Questions For the Basic and Clinical Sciences. Philadelphia: National Board of Medical Examiners; 2003.
Attali Y, Fraenkel T. The Point-Biserial as a Discrimination Index for Distractors in Multiple-Choice Items: Deficiencies in Usage and an Alternative. J Educ Meas [Internet]. 2000 Apr 20;37(1):77–86. Available from: http://www.jstor.org/stable/1435063
Van Rossum G, Drake FL. Python 3 Reference Manual. Scotts Valley, CA: CreateSpace; 2009.
MacQueen J. Some methods for classification and analysis of multivariate observations. Proc fifth Berkeley Symp Math Stat Probab. 1967;1(14):281–97.
Thorndike RL. Who Belongs in the Familiy? Psychometrika. 1953;18(4):267–76.
Bengfort B, Danielsen N, Bilbro R, Gray L, McIntyre K, Richardson G, et al. Yellowbrick V0.6 [Internet]. Zenodo; 2018. Available from: https://zenodo.org/record/1206264
Odashima S, Ueki M, Sawasaki N. A Split-Merge DP-means Algorithm to Avoid Local Minima BT - Machine Learning and Knowledge Discovery in Databases. In: Frasconi P, Landwehr N, Manco G, Vreeken J, editors. Cham: Springer International Publishing; 2016. p. 63–78.
Caliński T, Harabasz J. A dendrite method for cluster analysis. Commun Stat [Internet]. 1974 Jan 1;3(1):1–27. Available from: https://www.tandfonline.com/doi/abs/10.1080/03610927408827101
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine Learning in Python. J Mach Learn Res. 2011;12:2825–30.
XGBoost Documentation [Internet]. [cited 2020 Jul 30]. Available from: https://xgboost.readthedocs.io/en/latest/#
Tiangi C, Guestrin C. XGBoost: A Scalable Tree Boosting System. In: KDD ’16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016. p. 786–94.
Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, et al. Explainable AI for trees: From local explanations to global understanding. arXiv [Internet]. 2019;2(January). Available from: http://dx.doi.org/10.1038/s42256-019-0138-9
Molnar C. Interpretable machine learning. A Guide for Making Black Box Models Explainable [Internet]. [cited 2020 Sep 30]. Available from: https://christophm.github.io/interpretable-ml-book/
Grandini M, Bagli E, Visani G. Metrics for Multi-Class Classification: an Overview. 2020;1–17. Available from: http://arxiv.org/abs/2008.05756
Schüttpelz-Brauns K, Kadmon M, Kiessling C, Karay Y, Gestmann M, Kämmer JE. Identifying low test-taking effort during low-stakes tests with the new Test-taking Effort Short Scale (TESS) – development and psychometrics. BMC Med Educ [Internet]. 2018;1–10. Available from: https://doi.org/10.1186/s12909-018-1196-0
Wise SL, Owens KM, Yang S, Weiss B, Kissel HL, Kong X, et al. An Investigation of the Effects of Self-Adapted Testing on Examinee Effort and Performance in a Low-Stakes Achievement Test. 2005;1–16.

No competing interests reported.

SiegetalClusteringSupplement.pdf

Download PDF

Journal Publication

published 29 Mar, 2023

Read the published version in BMC Medical Education →

Editorial decision: Major revision
15 Jan, 2023
Reviews received at journal
07 Jan, 2023
Reviewers agreed at journal
06 Jan, 2023
Reviewers agreed at journal
04 Jan, 2023
Reviewers invited by journal
04 Jan, 2023
Editor assigned by journal
04 Jan, 2023
Editor invited by journal
10 Jun, 2022
Submission checks completed at journal
10 Jun, 2022
First submitted to journal
08 Jun, 2022

You are reading this latest preprint version

Analyzing Answering Patterns in Progress Test Data

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Materials And Methods

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1