Impact of guided self-study on learning success in undergraduate physiotherapy students in Switzerland – a feasibility study of a higher education intervention

Abstract

Background: Guided self-study (G-SS) can be used as a self-directed learning method (andragogy) or self-determined learning (heutagogy) that fosters changes in knowledge and skills in a higher physiotherapy education setting. Until now, there has been no empirical evidence for the use of G-SS in higher physiotherapy education. Furthermore, and to the best of our knowledge, no practical guidelines for G-SS exist for physiotherapy undergraduate students to foster changes in knowledge and skills. This higher education feasibility study aimed to investigate the feasibility to establish a G-SS program in a fulltime undergraduate physiotherapy degree course at the Bern University of Applied Sciences (BFH), Switzerland. In addition, the effectiveness of the G-SS was assessed on changes in knowledge and skills.

Method: Fifty-one first-semester physiotherapy students volunteered from BFH. They were randomly divided into a G-SS group or control group (CG). The G-SS group received six cases. Each case was processed in an eight-day cycle. One week in advance, the clinical case (i.e. description of a patient and the symptoms) and learning goals were provided to the students electronically (day 1). The students prepared the cases in groups and were guided (e.g. via Skype) by the tutor during this preparation time (day 2 to 7). The results of the group work were presented and reflected on during a moderated plenum session (90 minutes) at day 8. The primary outcome parameters were i) exposure as the number of G-SS sessions performed over 90 minutes, ii) the responsiveness of students with a previously determined 83% willingness to participate in the G-SS program iii) program differentiation, to illustrate the difference between G-SS content and the curriculum and the total score in the objective structured clinical examination (OSCE) and written exams. Statistical analyses were conducted using an Intention-To-Treat (ITT) approach.

Results: All six G-SS sessions on day 8 were performed over 90 minutes. The responsiveness of students as willing to participate in the G-SS program was 23%, clearly below the set 83%, and in terms of program differentiation, no differences were found between G-SS program and normal curriculum content. G-SS as compared to the CG scored significantly better on OSCE (p < 0.01) and on the written exam (p < 0.04).

Conclusion: The results showed that this higher education G-SS program in its current form was not feasible. Slight modification of the study protocol (e.g. better time planning in the academic calendar) is needed to improve the student’s responsiveness. The adjustments to the timetable must allow the physiotherapy students to prepare the clinical cases. G-SS have the potential to promote change in knowledge and skills in undergraduate physiotherapy students when students prepare the case, present the case solutions and reflect upon their actions.

Background

The shift away from traditional concepts of work requires individuals who can adapt quickly to new situations and know how to acquire knowledge. Within this changing world, humans should learn more independently, in preparation for higher education, work and life [1]. To prepare humans for changes in the world of work, pedagogical methods are no longer fully sufficient [2]. Previously, learning was related to childhood, but lifelong independent learning is becoming increasingly essential.

The higher education landscape must satisfy this requirement with the implementation of adult education science. This includes andragogy which focusses on self-directed learning (SDL) or heutagogy which focusses on self-determined learning (SDtL).

Knowles [3] defined andragogy as the art and science of helping adults learning, in contrast to pedagogy which has been defined as the art and science of teaching children. The focus of attention is on the wish to learn through active and independent work [4]. This leads to a redefinition of the higher education teacher's role. The higher education teacher becomes a facilitator, who supports the adult learners and diagnoses individual learning needs. As a facilitator, the higher education teacher plans active sequences, choosing the most effective teaching and evaluation methods.

Heutagogy depicts self-determined learning as a holistic, and lifelong process [5]. The learner is in the center and knows his best learning needs [6]. The aim is to go further than immediate problem solving, to extend the learning process itself, and thus giving learners the opportunity to develop self-efficiency and capability [5]. The learning process takes place by means of self-chosen and self-directed actions. The higher education teacher takes a minor role, teacher-centered instruction (TCI) is avoided. The method of heutagogy has not yet established in the German-speaking countries.

Luckin et al [7] described a Pedagogy-Andragogy-Heutagogy (PAH) continuum as a construct within which students move cognitively, metacognitively and epistemically within the PAH to enable learning, arguing that students need to be supported in managing their own learning.

Based on the Bologna Process, the Rektorenkonferenz der Fachhochschulen der Schweiz [8] defined that academic programs in Switzerland are divided into required time of attendance session (e.g. TCI, workshops, seminars) and self-study (free self-study (F-SS) and guided self-study (G-SS)). In the context of higher education self-study could be used to foster SDL and SDtL.

Theoretical background and the formulation of research question

Self-study is an essential component of scientific higher education [9, 10]. Two forms of self-study have been proposed: F-SS and G-SS [10]. While students are self-reliant in F-SS, they will be supported by lecturers or tutors during G-SS [11]. Both self-study forms are implemented in the higher physiotherapy education curricula at the Bern University Applied of Science (BFH). In the context of learning practical physiotherapy skills in higher education, understanding and applying the mechanisms of G-SS is of particular importance. Up to now, there has been no standardized procedure for G-SS at the BFH. Based on information provided by Landwehr and Mueller [10] and Rogan [11], a standardized procedure to implement G-SS was established.

Landwehr and Mueller postulated five phases for G-SS [10]. During a preparatory phase (phase one), students receive a learning assignment (clinical case description) from the tutor with clear-cut learning objectives. Phase two consists of the realization stages one and two in which the students work independently on the learning assignment. The tutor provides coaching (stage one) and checking (stage two) to the students. During phase three the students present an insight into their learning outcomes to the tutor and their fellow students (e.g. during a plenum session). Both students and tutor will reflect on the learning process during phase four. Finally, in phase five, the students give each other peer feedback on their presentations and learning processes. Rogan [11] proposed a theoretical model on the structure of G-SS and on how physiotherapy students gain knowledge in practical physiotherapy skills. However, empirical evidence of this theoretical model according to Rogan [11] is still lacking. Therefore, an empirical study was carried out to gain preliminary evidence. As a first step in a line of research, the feasibility and acceptability of a study should be investigated as a pilot/feasibility study, because pilot studies are research methods that may help to improve the feasibility of larger studies [12]. A pilot/feasibility study also includes the examination of treatment effects, but this is a secondary goal [12, 13].

It is known that in medical education evidence was generated on prejudices, hunches, opinions, and guesses [14]. As in other disciplines, deficiencies in medical education have been identified for reporting quality of published studies [15] and the lack of robust study designs [16]. This study used the Kirkpatrick’s four level evaluation model including (1) learners’ satisfaction, (2) changes in knowledge and skills, (3) changes in behavior and (4) changes to the patient outcome behavior and results to evaluate the learning effectiveness of the guided self-study intervention model of the BFH. The Kirkpatrick’s four level evaluation model is employed widely in education research.

This research project will investigate the feasibility and the potential impact of G-SS on changes in knowledge and skills based of Rogan’s theoretical work [11], into one cohort of an undergraduate physiotherapy higher education fulltime program at BFH, Switzerland. The primary objective of this feasibility study was to report the fidelity of implementation of the G-SS how the implementation was and what are the possible adjustments for further studies and why G-SS succeed and fail. A further goal was to identify changes in knowledge and skills of the learners. The research question was: It is possible to develop a study design that will evaluate knowledge changes in undergraduate physiotherapy students?

Methods

Study design, setting, quality reporting, ethics

This cohort randomized feasibility higher education trial was designed as a prospective, single-center, two-arm design. This study was conducted at the BFH Department of Health, Division of Physiotherapy. Undergraduate physiotherapy students (n = 51) from the first semester were invited to participate.

At the BFH, the workload is divided into required time of attendance session (e.g. TCI, tutorials, workshops) and self-study sessions (F-SS and G-SS), whereby the workload of 1 European Credit Transfer System (ECTS) is defined as 30 learning hours. ECTS credits described the student’s workload needed to complete all learning activities (e.g. TCI, tutorials, workshops, self-study, etc.) and to achieve the specified learning objectives or competencies. The BFH guidelines recommended a workload ratio of 40% required time of attendance (e.g. TCI) to 60% of time for self-study. The workload at the BFH for the undergraduate physiotherapy degree course is < 40 hours per week.

To represent the quality of reporting of our higher education study, the CONSORT 2010 checklist was used [17]. This checklist could be used also for education studies. The ethics committee approved this study and a study registration was carried out on Registry of Efficacy and Effectiveness Studies (REES: ID: #1726.1). The sample size of n = 30 participants was based on the minimum recommended for a pilot randomized trial [18].

Participants and recruitment

Inclusion criteria for the current higher education feasibility study were young healthy physiotherapy students of the undergraduate physiotherapy degree course 2018 from the first semester (Fig. 1) of the BFH in the German speaking part of Switzerland. Exclusion criteria were: physiotherapy students of the undergraduate physiotherapy course 2018 from the first semester who did not want to take part in this study.

In Switzerland, there is a numerus clausus for the bachelor’s degree in physiotherapy. The suitability of applicants for the undergraduate physiotherapy program is examined in a two-stage professional individual suitable-test. To be admitted to the suitable-tests, candidates must have an abitur or a vocational school-leaving certificate. The best 51 candidates will be selected for the fulltime study degree course. All students fulfil the same requirements in terms of school-leaving qualifications and previous knowledge.

Recruitment of physiotherapy students of the undergraduate physiotherapy course 18 was done by means of an oral information session. After the information session, the declarations of consent were distributed. The potential study participants were given sufficient time to read the document at home in peace and quiet in order to decide for or against voluntary study participation. All participants provided written informed consent.

Randomization

As a rule, the undergraduate physiotherapy degree course regularly assigns physiotherapy students to groups A, B, C or D. This procedure is necessary to keep the group size for practical lessons small and to promote group learning. The randomized group allocation was continued before the start of the study. This randomization was not study-specific. Groups A and B and groups C and D are together in the practical sessions. The groups A and B and groups C and D were after recruitment procedure randomly assigned to either a tutor-G-SS group (n = 26) or a control group (CG; n = 25). Randomization was computer generated by an independent researcher.

Intervention

The process was based on the previous published Rogan's recommendations [11]: a total of six G-SS periods were scheduled for the G-SS group between the start of November 2018 and mid of January 2019. The duration of a G-SS period was an eight-day cycle in which a case was processed. In total six weeks were scheduled, whereby one case peer week was processed.

Cases were used for the G-SS sessions that aligned to the module contents of the undergraduate physiotherapy degree program and which were not targeted to the semester exam. The module content included areas of examination and investigation, such as basic movement (including gait), testing coordination, manual muscle strength tests, etc. Table 1 gives an overview of the contents of the clinical cases in the tutor-G-SS sessions.

The eight-day cycle of a G-SS period started from mailing the case with their tasks, the processing time and the presentation of the case was carried out as follows. On day 1, one week prior to the G-SS session, the students of the G-SS group were informed about the learning goals and received the clinical case description via email (phase one). From day 2 to 7, students could choose between an SDL or SDtL approach (phase two). With regards to SDL each group had the possibility to make an appointment with the tutor to clarify questions and processes, both on-site at the university campus and via Skype (phase two) or, alternatively, could follow the SDtL.

On day 8, the G-SS groups presented results of their work to the tutor as well as to their peers (phase three). Students carried out a reflection on their work orally (phase four). At the end of day 8, the tutor moderated an in-class plenary session and gave feedback about the presentations of the G-SS groups (phase five). The duration time of each G-SS session was 90 minutes. The tutor was a higher education lecturer from BFH with 19 years of teaching experiences.

Control group

Students are self-reliant in F-SS and performed F-SS as planned in the traditional curriculum of the bachelor’s degree course.

Outcomes

Control of extraneous variables

To establish comparability, a randomization was carried out. Due to the numerus clausus, the school-leaving certificate, the credit points in the two-stage professional individual suitable-test and the age of the students, a homogeneous group can be assumed. These factors can be excluded as confounders. However, other confounding variables such as communication, motivation or self-regulatory skills were not assessed. Theses variables may have reduced the validity of the study results. However, the underlying assumption was that these variables were likely to be evenly distributed within both groups.

Primary outcome: feasibility

The primary outcome measure of feasibility was the fidelity of implementation which was reported as i) exposure, if the total number six G-SS sessions were covered over a period of 90 minutes, ii) students’ responsiveness to the G-SS program and iii) program differentiation to illustrate the difference between the content of the G-SS sessions and the curriculum.

Mowbray et al. [19] postulated fidelity of implementation as the extent to which the delivery of an intervention adheres to the program model originally developed and confirms that the implementation of the independent variable in outcome research occurred as planned. The Lastica and O’Donnell [20] recommendation was followed. They propose that fidelity of implementation in education intervention studies should illustrate the following aspects. What and how often was measured and how was the critical features and the degree of acceptable variation explicitly documented.

The fidelity was measured as follows: i) Exposure was measured as 1. the total number of the conducted G-SS sessions and 2. the duration of each G-SS session in minutes, ii) Students responsiveness was documented by the tutors in the attendance list after each G-SS session (Phase 3–5) and by a post-study oral group interview survey. An adequate responsiveness to the protocol was defined that every student of the G-SS group should have attended five of six G-SS session, with 83% of students consenting. iii) Program differentiation was evaluated in the course of the program conception, we examined which program-related and competing contents could be observed in the G-SS cases and the curriculum.

Secondary outcome measures:

The Kirkpatrick’s model [21] was used to evaluate students’ learning outcomes. This model has been used to evaluate training programs and for evaluation of a variety of educational setting [22, 23]. Kirkpatrick’s model is based on four levels, 1) reaction, 2) learning, 3) behavioral changes and 4) organizational performance. It is usually not possible to measure level one to level four [24]. Therefore, Level 2 has been implemented in this study. Related to learning outcomes of the intervention, this level considers whether the students acquired the intended knowledge and skills based on their participation in the intervention G-SS. Students were assessed twice per semester for written (multiple choice; MC) and practical (objective structured clinical examination; OSCE) competences. These two exams were part of the undergraduate study of physiotherapy. The OSCE consisted of eight stations. The total score for the MC exam was 94 points, the total score for the OSCE was 48 points, and the total score of each OSCE station (6 points) have been used as dependent outcome variables. The OSCE exam grade depended on their total score and the pass will be set at 60% for MC and OSCE.

Statistics

To analyze the effect of G-SS on the final grades of the written exam and OSCE (dependent variables) the number of attendances of each student on the presentation day of the G-SS was used as an independent variable.

For the secondary outcome measures, descriptive statistics were conducted and presented as means with corresponding standard deviations (SD). To guarantee that the randomization remains unbroken, an intention-to-treat analysis (ITT) was performed, where missing data were replaced by mean values of the group to which subjects were originally allocated [25]. The student’s t-test was applied to determine differences in the exam results between the two groups after the first semester. All calculations were performed using Statistical Package for Social Sciences (SPSS) version 25.0 (IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp.)

Results

This manuscript reports data for 51 undergraduate physiotherapy students from the BFH. All students volunteered in this higher education feasibility study. Students from group A and B were randomized into the G-SS group (n = 26) while students from group C and D (n = 25) were randomized into the CG.

Discussion

According the recommendation of the Rektorenkonferenz der Fachhochschulen der Schweiz [8], the Department of Health at the BFH has the responsibility to design a student-centered curriculum with self-study periods. Implementing G-SS in the physiotherapy undergraduate curriculum could improve teaching and could foster students-centered learning approach. This feasibility study evaluated the fidelity of implementation (whether G-SS was implemented at all) to explain succeed and fail of G-SS implemented in comparison with the original program design. In addition, the changes in knowledge and skills in undergraduate physiotherapy students at the BFH were assessed.

The research question “it is possible to develop a definitive study design that will evaluate knowledge changes in undergraduate physiotherapy students?” could be answered as follows: this present study design is not appropriate to assess knowledge changes in undergraduate physiotherapy students. We were able to demonstrate that G-SS as originally planned and scheduled in the academic timetable is not feasible. A modification of the study design must be carried out.

Regarding the fidelity of implementation of exposure, it could be demonstrated that all six G-SS sessions at day 8 were performed in the planned time of 90 minutes. In terms of exposure, it can be assumed that program are less effective if the target group does not receive the number of intended interventions [26].

Only six students (23.1%) instead of the expected 83% participated in five G-SS sessions at day 8. The first session was very well frequented, because this session was scheduled in a time period with normal workload of < 40 hours. The fifth and sixth GSS sessions (day 8) was used by the students as exam preparation units > 45 hours. The learning workload increased in December of the circumstances that students must additional write a project thesis. Another reason was that the timetable in November and December 2018 were designed with a higher learning workload to allow sufficient learning preparation time in January 2019 for the exam in February 2019. The scheduling of the G-SS in the timetable was mentioned by the students as the reason for the low acceptance. This is in line with the findings of the literature.

Newble and Entwistle [27] postulated that student learning is influenced by individual characteristics and by external circumstances. External learning circumstances are exam and learning workload of the curriculum. Exams can be a strong stimulus for learning [28]. The module exams occurred at the beginning of February 2019. Curricular circumstances are a further determinant of learning [27]. The learning workload was very high between November and December 2019 (> 45 hours per week). It is known that curricula influence undergraduate students’ preference for learning environments [29]. The analysis showed that the quality of the G-SS cases was rated as good. The contents of the cases and goals of the cases were aligned with the curriculum. This promotes the willingness to accept the cases and not to reject them in advance. The cases (Table 1) can be transferred in this form to the next study.

Based on the findings, the following modifications must be considered for the upcoming study design. G-SS study should be scheduled in time periods where workload is in normal range (< 40 hours). It must also be taken into account that the learning workload during the module is evenly distributed,

Preliminary efficacy results

An ITT-analysis was conducted, the results were discussed very conservative. TCI and seminars are the most suitable to promote the transfer of factual knowledge [30]. In order to develop practical skills, other learning methods must be offered [30]. This present study determined that all students in the G-SS group passed both exams, while students in the CG (n = 4) have been failed in both exams. Moulaert et al. [31] explained that repeated practice of a skill is particularly important for the learning process. The ability to independently acquire knowledge and skills is an important prerequisite for clinical work [28]. G-SS meets this requirement because the preparation of the case, the presentation and the student’s feedback as reflection process were included. If students attend the G-SS sessions six times, it indicates beneficial change of knowledge and skills.

However, less than six G-SS did not change knowledge and skills. Murat et al. [32] conducted a meta-analysis to illustrate the effectiveness of the SDL versus traditional teaching methods on knowledge domain. The authors included 50 articles with 8000 medical students and health professionals’ students. SDL showed significant gain in factual knowledge. No significant difference between SDL and traditional teaching groups could be determined for skills. However, SDL was effective if the participants were involved in the selection of learning aids. It is recommended to support learners in identifying suitable learning materials in the context of self-study (e.g. video, books). Furthermore, they recommend that SDL should be used primarily as a teaching method for advanced learners. The participants of the current study were undergraduate students in the first semester. In general, the students proceeded in this feasibility study according to the SDtL learning method. The principle of heutagogy seems not applicable to undergraduate physiotherapy students in the first semester.

The future study should be structured according the SDL principle. The university lecturer should guide the students in which meetings are offered. Furthermore, students should be involved in their learning process. In the preliminary stage, the learning methods must be clarified. The first three cycles of the G-SS program, the scheduled meetings with the university lecturers should be declared as mandatory meeting. The remaining meetings in last three cycles of the G-SS program can be declared as voluntary meetings.

Limitation

A possible weakness of this feasibility study was that confounding variables such as communication, motivation or self-regulatory skills were not assessed. Despite the randomization, this may have influenced the results. Further studies should measure these variables prior the baseline.

Another limitation was the measurement of the number of fidelities of implementation criteria. Dusenbury et al. [33] postulated five fidelity of implementation criteria. 1. Adherence: were the components of the intervention being delivered as designed. 2. Exposure: number, length, or frequency of sessions implemented. 3. Quality of delivery: the way in which the program was delivered using the techniques, processes, or methods prescribed. 4. Participant responsiveness: participants engagement by and involved in the activities and content of the program. 5. Program differentiation: if there are critical characteristics that distinguish the program from the comparison condition, are they present or not during implementation. In this present study, qualitative methods were used. However, the power improves if statistical quantitative measure of fidelity will be carried out. Further studies should use qualitative and quantitative methods.

Additionally, three of the five criteria have been measured. The next study should take into account to evaluate all five criteria of fidelity of implementation.

Conclusion

This study presents the feasibility of a G-SS program based on the learning method SDL (andragogy) and SDtL (heutagogy) that has been developed for first semester undergraduate physiotherapy students at the BFH in Switzerland. Findings from the results suggest that the developed study design was not feasible, and the study design must modify for future studies. We demonstrated that all six G-SS session were carried out as scheduled, student’s attendance to G-SS sessions was low with 23.02% instead of the expected 83%. The scheduling of the G-SS sessions into the period of a high workload was considered as the main reason. Future studies must consider the workload of the timetable when planning the G-SS.
G-SS showed beneficial effects on changes in knowledge and skills when students prepare and present all cases and students give feedback as reflection process at the end of each of six presentation sessions.

Declarations

• Ethics approval and consent to participate: Ethical approval was obtained from the Kantonalen Ethikkommission Bern, Switzerland (2018-01683). 'Retrospectively registered on Registry of Efficacy and Effectiveness Studies (26^th February 2019). https://sreereg.icpsr.umich.edu/sreereg/search/search?
• Consent for publication: Not applicable.
• Availability of data and materials: The datasets used and/or analyzed during the current

study are available from the corresponding author on reasonable request

• Competing interests: The authors declare that they have no competing interests.
• Funding: This project has no funding.
• Authors' contributions: This research project was developed by SR and JT. Data collection

was undertaken by SR. SR carried out the statistical analysis. SR wrote and JT, SZ and EZ edited the manuscript. All authors read and approved the manuscript.

• Acknowledgements: We would like to say a special thanks to all the participants who participated in this study and Eugen Mischler, Head of bachelor’s degree program Physiotherapy and Deputy Director of the Department of Health for the support.
• Authors' information:

Affiliations

Division of Physiotherapy, Department of Health, Bern University of Applied Sciences, Bern, Switzerland: Slavko Rogan, Jan Taeymans, Stefan Zuber

Academy for integrative physiotherapy and training education, Grenzach-Wyhlen, Germany: Slavko Rogan

Faculty of Physical Education and Physiotherapy, Vrije Universiteit Brussel, Brussels, Belgium: Slavko Rogan, Jan Taeymans, Evert Zinzen

Abbreviations

BFH: Bern University of Applied Sciences, CG: control group, ECTS: European Credit Transfer System, GSG: guided self-study group, ITT: intention-to-treat-analysis, MC: multiple choice, OSCE: objective structured clinical examination, SD: standard deviation, SPSS: Statistical Package for Social Sciences

References

1. du Toit-Brits C: A focus on self-directed learning: The role that educators’ expectations play in the enhancement of students’ self-directedness. South African Journal of Education 2019, 39(2).
2. Blaschke LM: Heutagogy and lifelong learning: A review of heutagogical practice and self-determined learning. The International Review of Research in Open and Distributed Learning 2012, 13(1):56-71.
3. Knowles MS: The modern practice of adult education. Englewood Cliffs, NJ, USA: Cambridge Adult Education; 1980.
4. Knowles MS: Möglichkeiten und Methoden der Erwachsenenbildung. In: Erwachsene lernen Methodik der Erwachsenenbildung. edn. Edited by Knowles MS, Husén, T. Stuttgart, Deutschland: Ernst Klett Verlag; 1963: 15-146.
5. Hase S, Kenyon C: From andragogy to heutagogy. Ulti-BASE In-Site 2000.
6. Hase S, Kenyon C: Heutagogy: A child of complexity theory. Complicity: An international journal of complexity and education 2007, 4(1).
7. Luckin R, Clark W, Garnett F, Whitworth A, Akass J, Cook J, Day P, Ecclesfield N, Hamilton T, Robertson J: Learner-generated contexts: a framework to support the effective use of technology for learning. In: Web 20-based e-learning: applying social informatics for tertiary teaching. edn. Edited by Lee MJW, MCLouglin, C. . Herhey, New York, USA: IGI Global; 2011: 70-84.
8. KFH RdFdS: Konzeption modularisierter Bachelor und Masterstudiengänge. In. Bern, Switzerland: Rektorenkonferenz der Fachhochschulen der Schweiz KFH; 2011: 28.
9. Herren D: Das Selbststudium begleiten. Bern, Schweiz: Hep-Verlag; 2018.
10. Landwehr N, Müller E: Begleitetes Selbststudium: didaktische Grundlagen und Umsetzungshilfen, 2 edn. Bern, Schweiz: hep Verlag; 2008.
11. Rogan S: Begleitendes Selbststudium in der Hochschulausbildung Physiotherapie . Lerntheorien und Lernmethoden. München, Deutschland: GRIN Verlag; 2015.
12. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH: A tutorial on pilot studies: the what, why and how. BMC medical research methodology 2010, 10(1):1.
13. Rogan S, Karstens S: Verwendung der Begriffe Machbarkeits-bzw. Pilotstudien. physioscience 2018, 14(01):1-2.
14. M. Harden PMLR: Best evidence medical education: the simple truth. Medical Teacher 2000, 22(2):117-119.
15. Meinema JG, Buwalda N, van Etten-Jamaludin FS, Visser MRM, van Dijk N: Intervention Descriptions in Medical Education: What Can Be Improved? A Systematic Review and Checklist. Academic Medicine 2019, 94(2):281.
16. Lim WS, Tham KM, Adzahar FB, Neo HY, Wong WC, Lim I, Ringsted C, Samarasekera DD: Trends of clarification purpose of medical education research in the Asia-Pacific region: A scoping review. Asian Pacific Scholar 2019, 4(1):24-33.
17. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, Schulz KF: CONSORT for reporting randomised trials in journal and conference abstracts. The Lancet 2008, 371(9609):281-283.
18. Torgerson CJ, Torgerson DJ, Director YTU: Randomised trials in education: An introductory handbook. London: Education Endowment Foundation 2013.
19. Mowbray CT, Holter MC, Teague GB, Bybee D: Fidelity criteria: Development, measurement, and validation. American journal of evaluation 2003, 24(3):315-340.
20. Lastica J, O’Donnell C: Considering the role of fidelity of implementation in science education research: Fidelity as teacher and student adherence to structure. In: Annual Meeting of the National Association for Research in Science Teaching: 2007 2007; New Orleans, LA, USA: Citeseer; 2007.
21. Kirkpatrick DL: Evaluating Training Programs. San Francisco: Berrett-Koeh. In.: ler Publishers. http://www. centrorecursos. com/mec/ayudas/casaVer. asp; 1998.
22. Chrysafiadi K, Virvou M: PeRSIVA: An empirical evaluation method of a student model of an intelligent e-learning environment for computer programming. Computers & Education 2013, 68:322-333.
23. Diefes‐Dux HA, Samant C, Johnson TE, O'Connor D: Kirkpatrick's Level 1 Evaluation of the Implementation of a Computer‐Aided Process Design Tool in a Senior‐Level Engineering Course. Journal of Engineering Education 2004, 93(4):321-331.
24. Embi ZC, Neo T-K, Neo M: Using Kirkpatrick’s Evaluation Model in a Multimedia-based Blended Learning Environment. Journal of Multimedia Information System 2017, 4(3):115-122.
25. Hollis S, Campbell F: What is meant by intention to treat analysis? Survey of published randomised controlled trials. Bmj 1999, 319(7211):670-674.
26. Kuhn D: Implementationsgrad und Wirksamkeit von Unterrichtsinterventionen zur Förderung der selbstbestimmten Lernmotivation. Ludwigsburg, Deutschland: Pädagogische Hochschule Ludwigsburg; 2018.
27. Newble DI, Entwistle NJ: Learning styles and approaches: implications for medical education. Medical education 1986, 20(3):162-175.
28. Wieland AK: Einfluss verschiedener Lernanreize auf das Lernverhalten und die Prüfungsleistungen von Studierenden der Humanmedizin. Göttingen, Deutschland: Georg-August-Universität Göttingen; 2016.
29. Kell C, Deursen Rv: Student learning preferences reflect curricular change. Medical teacher 2002, 24(1):32-40.
30. Kern DE: A six-step approach to curriculum development. P Thomas, D Kern, M, Hughes, & B Chen, Curriculum development for medical education 2016:5-9.
31. Moulaert V, Verwijnen MGM, Rikers R, Scherpbier AJJA: The effects of deliberate practice in undergraduate medical education. Medical education 2004, 38(10):1044-1052.
32. Murad MH, Coto‐Yglesias F, Varkey P, Prokop LJ, Murad AL: The effectiveness of self‐directed learning in health professions education: a systematic review. Medical education 2010, 44(11):1057-1068.
33. Dusenbury L, Brannigan R, Falco M, Hansen WB: A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health education research 2003, 18(2):237-256.