Rationale for using a realist evaluation:
Realist evaluation was chosen as a pragmatic theory-based approach avoiding the “epistemological poles of positivism and relativism” (2) (p. 5). It sits between the positivist stance that there is a mind-independent reality and the constructivist stance that ‘reality’ is unknowable (20) and seeks to “answer the question of what works for whom in what circumstances” (2) (p. 125). Realist evaluation combines a belief in a mind-independent reality (context) with acknowledgement of its effect on human responses (mechanisms) leading to outcomes. Realist analysis of empirical case studies seeks outcomes linked to mechanisms triggered by aspects of external reality (context) leading to the identification of context-mechanism-outcome configurations (CMOCs) (20). The aim is to improve a programme’s theory via a deeper understanding of programme context, mechanism, and outcome patterns (2).
Realist evaluation aims for continual improvement of practice rather than universal truths, however empirical generalisations or demi-regularities can be drawn that help build or refine theory (21). So, while a programme will work differently in different contexts the theory-based understanding of ‘what works for whom, in what context, and how’ is more widely applicable (22). As in this study an initial programme theory is often based on a mid-range theory to develop a programme rather than developing a programme from scratch. Mid-range theories, in turn, are usually a subset of a unified social theory (2). In this study Dornan et al’s experience-based learning theoretical framework (ExBL) has been used as a mid-range theory (18, 19) which is a subset of Lave and Wenger’s ‘community of practice’ social learning theory (23, 24). As Cruess et al (2018) argue the ‘community of practice’ learning theory provides a comprehensive, foundational theoretical framework for medical education as learners acquire the identity of a medical practitioner and accept medical culture’s norms (25).
The ExBL theoretical framework resulted from a review of the literature to describe the conditions, processes and outcomes of undergraduate, medical students’ clerkships (19) and was an elaboration of a previous model (18). The categories that were developed by Dornan et al (2014) for the ExBL theoretical framework were not presented as CMOCs (19).
Taking an iterative approach to data-analysis and subsequent theory refinement initially using an elaborative, or top-down, coding approach (26), based on Dornan et al’s mid-range theory (19) CMOCs were identified specific to medical students’ perspective of the RACF programme while developing a realist experience-based learning programme theory that is potentially relevant to generic clerkship placements.
Viewing ExBL codes from a realist evaluation perspective mostly results in a reconfiguration of ExBL’s conditions, processes and outcomes into context, mechanism, and outcome categories.
Context contains all the environmental aspects in which a learning programme is delivered (27, 28). Context includes prevailing beliefs, social and cultural norms, regulatory and economic forces (20). Realist mechanisms are usually hidden, sensitive to variations in context, and generate outcomes (29). Depending on the range of the intended explanation of a realist evaluation, the mechanisms described may be conceptualised as explaining large-scale social transformations or individual reasoning and responses (30). An inclusive definition is “a mechanism is an element of reasoning and reactions of (an) individual or collective agent(s) in regard of the resources available in a given context to bring about changes through the implementation of the intervention” (31) (p. 8). Outcomes include the intended and unintended consequences of an intervention (32).
The complex intervention of medical education is particularly suited to realist evaluation due to the iterative programme theory-building process that realist methods support (20). As Wong et al (2012) note in medical education the question may be “what kinds of educational interventions will tend to work for what kinds of learners, in what kinds of contexts, to what degree, and what explains such patterns?” (20) (p. 93). Realist programme evaluation also answers the call for better health professions education programme evaluation than to answer the question ‘did it work’? (33).
In this study outcomes include each student’s global evaluation of, claimed learning outcomes from, or reported learning experience of their RACF clerkship. Mechanisms account for outcome variation as each student responds to their perception of their context. All mechanisms in this analysis are embodied in the student i.e. they are intraindividual. The context triggering a mechanism was a student’s experience of the clerkship in the RACF which was nested within the wider context of their medical education, which is nested within their total environmental context.
Environment surrounding the evaluation:
Australian RACFs provide care for 9% of Australians. The medical care provided is complex and challenging with 70% of residents 80 years or older, 90% dying in the facility, and 40% of permanent residents dying within a year of admission. The medical problems managed are mixed with 52% of permanent residents having dementia, 25% a mental illness, 25% cardiovascular disease, 17% musculoskeletal and connective tissue disease (34).
In Australia, general practitioners (GPs) (family or primary care physicians) manage the medical care of RACF residents. The GP caring for a RACF resident usually conducts their main practice off-site, visiting the RACF routinely and/or on an as-needs basis (35).
The RACF system of healthcare delivery is seen as under strain in Australia due to factors such as too few and/or poorly trained nursing and care staff, poor availability of GP-emergency and after-hours care, and poor remuneration of RACF healthcare professionals (36). This resource-poor environment may also have to deal with the unrealistic expectations of family members or residents for curative care when a more palliative approach is warranted, as in the care of people with end-stage dementia. The skills to deliver a more appropriate palliative approach may also be lacking (37–40).
Description of the programme evaluated:
The development of the novel programme was undertaken to expose senior medical students to the unique clinical environment of an RACF while also highlighting explicit learning outcomes related to the care of elderly, frail patients. In particular the placement aimed to give students experience in dementia care, and palliative care for patients including those with dementia.
The clerkship was highly structured to overcome the lack of hour-by-hour on-site supervision by a GP and to ensure the safe care of the residents with whom students’ worked.
No curriculum existed for medical learners delivering care for RACF residents (41) so various curricula or other resources were adapted to guide the initial development of the one-week placement (42–44).
A GP tutor, whose patients the students engaged with, met students for two hours at the beginning of the day in the RACF, and students reported progress at the GP’s practice at the end of the day. The GP directed student activity, delivered tutorial topics, assessed progress, and gave feedback. Nursing staff were employed and trained to mentor health professional students within the RACF, including the medical students (45). The nurses worked closely with the GP tutor and co-supervised the medical students. Opportunities for interprofessional learning were hoped for where medical student placement weeks coincided with nurse and/or paramedic clerkships.
Students were orientated to the placement with a three-hour workshop on the characteristics of people who lived in RACFs and the medical management of the frail elderly who may have dementia and need a palliative approach to their care.
The placement was contextualised to students’ impending role as hospital interns noting that all would be caring for in-patients who had either been transferred from an RACF before admission or be transferred to RACFs after an in-patient stay. Understanding the context of their prospective patients’ care would therefore aid the care they delivered as interns.
RACF residents, or their guardians, prospectively consented residents to participate in the programme. Resident safety was of paramount importance in curriculum design. The highly structured programme required students to cooperate with peers to practise their comprehensive clinical assessment skills with residents. The syllabus for the clerkship was the Royal Australian College of General Practitioners “Silver Book” (42).
The clerkship programme was also scaffolded (46) to enhance the students’ skill acquisition over the course of the week. The residents assessed had no co-morbid dementia at the beginning of the week. The second day involved assessing residents with mild to moderate levels of dementia. By the third- or fourth-day students assessed residents with severe dementia, many of whom resided in a secure unit to optimise the management of behaviour such as wandering.
The lead author(JR) worked with a key GP tutor to devise the initial curriculum, and third author (AT) assisted with the development and supervision of student-engaged clinical audits. All worked closely with RACF staff on the design of the programme.
The initial evaluation design:
The evaluation involved student focus groups undertaken on day four of the five-day placement, and of written feedback collected at the end of the week. The written feedback asked students to provide a global rating for the placement ranging from poor to excellent and asked them to note why they rated the placement as they did, what worked well with the placement and what worked poorly and how the placement could be improved. The written feedback provided weekly advice to the programme developers, whilst the focus-group interview data added another dimension to the written feedback at six-monthly intervals. All data were used to refine the initial programme theory.
Recruitment process:
All students were invited to participate and consented by academics independent of student assessment. All but two students participated in the evaluation.
Data analysis:
Written feedback forms were deidentified and transcribed. Focus group interviews were audio-recorded, transcribed and deidentified. The texts were initially coded using Dornan et al’s experience-based learning (ExBL) theoretical framework for clerkship education (p. 735-6) (19) which was organised into clearer numbered codes. (see supplementary file 1). Excel (47) spreadsheets were used to identify codes and, subsequently, context-mechanism-outcome configurations.
The process of analysis, theorising and developing CMOCs was iterative taking a retroductive approach to theory refinement i.e. both inductive and deductive logic was used as well as insights and hunches (48). The iterative process involved elaborative coding of text-based data based on the ExBL theoretical framework [2014 dornan] with new concepts added to that framework, and the identification of CMOCs.
The coding of both types of data was undertaken initially by JR, then reviewed by MB, and AT with discussion leading to final agreement. Acknowledging reflexivity we are a multidisciplinary team with JR a clinical educator, a GP who cares for RACF residents, and a cognitive behavioural therapist; MB a registered nurse and research project officer; and AT a pharmacist and clinical educator who has conducted medication reviews for RACF residents.
The report meets realist evaluation reporting standards (49) and qualitative research reporting standards (50).