Design
In line with the MRC guidance on process evaluation14,15 this mixed methods process evaluation using qualitative and quantitative approaches will collect data on implementation of the intervention, mechanisms of impact, outcomes and contextual factors. The tasks, aims, data and data source for each of these are summarised in Tables 1-4. All data is collected, from intervention arm participants only, after the 6 month study period is completed for an individual participant.
Implementation
Data will be collected on the effectiveness of the training, and the services delivered by the PIPs (see Table 1 below) to provide an understanding of whether the PIPS were adequately prepared for the role and the fidelity with which it was delivered.
Table 1: Implementation tasks and data collected as part of process evaluation
Task
|
Aim (what is being assessed)
|
Data collected
|
Data source
|
Provide training for PIPs
|
Effectiveness of training
|
PIP views on training
|
Post training feedback forms (at end of two day training session)
|
PIP Interview
PIP questionnaire
|
Competency
|
Competency assessments (feedback from independent assessors)
|
Appropriateness of PCPs (20% sample:appendices 3,4)
|
Views of stakeholders (interviews)
|
PIP delivery of the intervention
|
Fidelity to intervention
|
Services provided and frequency with which provided
|
PIP activity logs
|
Number of pharmaceutical care plans
|
PIP questionnaire
|
Quality of medication review
|
Review of 20% of pharmaceutical care plans
|
Mechanism of impact
Data will be collected to confirm the mechanism of impact of the intervention in achieving the desired aim of improved patient quality of care (see Table 2 below). This section draws particularly on the logic model and hypotheses for addressing the highlighted issues.
Table 2: Mechanism of impact and data collected as part of process evaluation
Impact
|
Mechanism of impact
|
Data collected
|
Data source
|
Medication changes identified
|
PIP medication review
|
Recommendations for change and rationale
|
Pharmaceutical care plans
|
PIP interview
PIP questionnaire
|
Medication changes made
|
PIP prescribing
|
Total no. medications per patient at baseline and 6 months
|
Pharmaceutical care plans
|
GP records
|
No. medications stopped per patient at 6 months
|
Pharmaceutical care plans
|
GP records
|
No. medications started per patient at 6 months
|
Pharmaceutical care plans
|
GP records
|
No. medications amended e.g. dose change, formulation change
|
Pharmaceutical care plans
|
GP records
|
No. antipsychotics/psychotropics prescribed at baseline and 6 months
|
Pharmaceutical care plans
|
GP records
|
Categorised description of drugs changed, stopped, started
|
Resident medical records
|
Biochemical monitoring
|
PIP medication review
|
Recommendations made for biochemical monitoring
|
Pharmaceutical care plans
|
Medication errors
|
PIP medication review
|
Number of prescribing, dispensing and administration errors
|
Pharmaceutical care plans
|
GP records
|
Non-patient facing activities improved eg medication storage advice
|
PIP support for care home
|
Services provided and frequency
|
PIP activity log
|
Views on usefulness of services
|
Care home staff interviews
|
PIP interview
PIP questionnaire
|
Better/tailored training for staff
|
PIP training for care home staff
|
Training provided and frequency
|
PIP activity log
|
Views on usefulness of training
|
Care home staff interviews
|
PIP interview
PIP questionnaire
|
Quality of communication between care home, GP and community pharmacy improved
|
PIP input into improved communication
|
Views of care home staff
|
Care home staff interviews
|
Views of GPs
|
GP interview
|
Views of PIPs
|
PIP interview
PIP questionnaire
|
Outcomes
The outcomes that are collected and which will be used in the process evaluation are described in Table 3. The selected outcomes are those where there is a clear link to the intervention proposed and where they inform the process.
Table 3: Outcomes and data collected as part of process evaluation
Aim
|
Outcome
|
Data collected
|
Data source
|
To improve quality of care for those over 65 years old resident in care homes
|
Falls
|
Fall rate per person at 3 months
|
Care home falls record
|
Fall rate per person at 6 months
|
Care home falls record
|
Quality of life
|
Self-reported quality of life
|
Face to face self-reported EQ-5D-5L (only applicable for participants with capacity) at baseline, 3 months and 6 months
|
Carer assessed quality of life
|
Proxy EQ-5D-5L (quality of life) at baseline, 3 months and 6 months
|
Physical functioning
|
Carer assessed physical functioning
|
Proxy Barthel Index (physical functioning) at baseline and 6 months
|
Health-service utilisation and associated costs
|
Costs of care (medication, health care team contacts, monitoring and tests)
|
GP records at baseline and 6 months
|
Drug Burden Index
|
Calculate DBI based on medications
|
GP records at baseline and 6 months
|
To assess intervention safety
|
Mortality
|
Information on numbers dying
|
Monthly call to care homes
|
Hospitalisations
(NB not always a negative marker of safety)
|
Information on numbers hospitalised
|
Monthly call to care homes
|
*Global view
|
Perceptions of GPs
|
GP interview
|
Perceptions of care home staff
|
Care home staff interviews
|
Perception of residents/consultee/WPOA
|
Resident/consultee/WPOA interviews
|
Perceptions of PIPs
|
PIP interview
|
*Adverse events
|
New drug related symptoms
|
Stakeholder feedback using standard template
|
*Serious adverse events
|
See hospitalisations/deaths
|
Monthly call to care homes
|
*Sudden unexpected serious adverse events
|
See hospitalisations/deaths
|
Feedback from GPs/independent medical assessor on causal link with PIP intervention
|
*Other than those asterisked these are also primary and secondary outcomes for main trial
Contextual factors
Any contextual factors identified which might have affected the delivery and impact of the intervention are described in Table 4 below. This information may include factors related to individual personnel and organisations as well as macro-level issues such as CQC requirements or head office requirements.
Table 4: Contextual factors collected as part of process evaluation
Contextual factor
|
|
Data collected
|
Data source
|
Barriers to delivering the intervention
|
|
Feedback from stakeholders
|
Care home staff interview
|
GP interview
|
PIP interview
|
NoMAD16 survey to GPs/PIPs and care home staff
|
Other anecdotal feedback
|
Facilitators to delivering the intervention
|
|
Feedback from stakeholders
|
Care home staff interviews
|
GP interview
|
PIP interview
|
NoMAD16 survey to GPs/PIPs and care home staff
|
Other anecdotal feedback
|
Site and participant factors
|
Inter PIP variation
|
Competency
|
Variation in outcomes
|
Review of PCPs for both safety and missed opportunity
|
GP interview
|
Care home interviews
|
Employment status
|
Baseline PIP questionnaire
|
Qualifications
|
Baseline PIP questionnaire
|
Inter site variation
|
Care home factors
|
Baseline CH survey
|
Resident factors
|
Baseline resident data
|
Inter location variation
|
Views of researchers
|
Meeting minutes
|
Normalisation of intervention into routine practice
|
Actions taken by participants to ensure the intervention works
|
Coherence
(Making sense of the service)
|
NoMAD survey16 to PIPs, CH staff, GPs
|
GP Interview
CH staff Interviews
|
PIP interview
|
Cognitive participation
(Engaging with the service)
|
NoMAD16 survey to PIP, CH staff, GPs
|
Interviews (GP and CH staff)
|
PIP interview
|
Collective action
(delivering the service/responding to the service)
|
NoMAD16 survey to PIP, CH staff, GPs
|
GP interview
CH staff interviews
|
PIP interview
|
Reflexive monitoring
(appraising and reviewing the service)
|
NoMAD16 survey to PIP, CH staff, GPs
|
GP interview
CH staff interviews
|
PIP interview
|
Data collection methods
The following text refers to the data sources identified in Tables 1-4 above
Quantitative
Data sources related to training and pharmacist competency
- Training feedback: At each PIP training event, PIPs are asked to complete a feedback form at the end of the two-day face-to-face session
- Pre intervention competency: Following the training PIPs submit their competency framework to one of the study competency assessors who discuss these with the PIP and signs them off as ‘fit to practise’ as a CHIPPS PIP, prescribes further training or that they are not competent to deliver the study
- Review of Pharmaceutical Care Plans (PCPs): Following an agreed process (see Appendices 4 and 5) a random 20% of PCPs are reviewed for appropriateness by study team members who are specialists in care of the elderly medicine. Whilst this process is primarily about safety, the assessment templates also capture data on missed opportunities.
Data sources related to activity
- PIP activity log: Intervention PIPs are asked to keep an activity log of their daily activity detailing the time spent on tasks as listed in the service specification (see Appendix 2)
- PIP survey: Following each phase, intervention PIPs will be asked to complete a short questionnaire asking about their experiences and the extent to which they delivered aspects of the intervention focussing especially on non-medication review aspects of the service specification ( see also NoMAD16 survey below).
Data sources related to prescribing
Most of the prescribing associated data is collected as part of the main trial processes to assess effectiveness and efficiency of the intervention (GP records, health care utilisation, falls records, hospitalisations and deaths) and processes are detailed in the main trial protocol (version 5 1.7.18). The following lists additional data collected as part of the process evaluation
- Adverse Events: Adverse events which are not deemed serious are reported using a standard template emailed to the Clinical Trials Unit. All study participants with a professional role (PIP GP,GP staff and CH staff) are made aware of this template and are asked to use this facility if they suspect any adverse event, whether or not there is a perceived causal relationship with the intervention
- Pharmaceutical Care plans: these are completed by the PIP as a clinical record of their actions including the rationale for these. Data extraction from these will inform the details of medication changes that underpin the global measures such as total number of medicines, BNF categories most involved in changes, and overall DBI. They will also include information on homely remedies and medications available from pharmacies (P medicines) and other retail outlets (GSL medicines) which could result in therapeutic duplication
Data sources related to variability
Variability may be due to inherent non-modifiable differences across participating organisations, sites and individuals, or to the way the CHIPPS service has been normalised. The former will be explored using standard statistical approaches and the latter by applying normalisation process theory (NPT) via a NoMAD16 survey to all participating GPs, PIPS and care home staff at the end of each phase.
- Regression modelling: The main study outcome measures, study group and participant variables will be analysed using regression modelling to identify the extent of the contribution of the different PIPs/sites, or locations to variation in study outcomes.
- NoMAD survey16: The NoMAD survey is an implementation measure based on the Normalisation Process Theory (NPT)17,18. The survey form includes preliminary demography and general questions about experiences and satisfaction followed by four sections each relating to one of the NPT domains of coherence, cognitive participation, collective action and reflexive monitoring.
Qualitative
Feedback from all stakeholders on their experiences and views of the intervention is a core part of this process evaluation. This will help contextualise the intervention and increase understanding of the process of implementation and any variation between sites and stakeholders.
Interviews: At the end of each phase of the intervention a purposive sample of up to three of each of GPs, care home managers, staff, residents and relatives (if available) in each of the four geographical areas will be invited to take part in a semi-structured interview. Sampling will be based on a maximum variation sample to reflect differences in site and PIP characteristics e.g. PIP employment status, previous PIP experience, demographic profile of care home residents, rural or urban location. All PIPS will be invited to take part in an interview.
Interviews will be guided by a topic guide (see Appendix 5, developed from the qualitative outputs from the earlier non-randomised feasibility study12. Topics will include participants’ views of the PIP service implementation and delivery, communication between staff, perceived effectiveness of the intervention and the identification of any unintentional consequences. All aspects of the service will be probed and there will be specific probes for unforeseen effects to understand whether anything else about the service impacted positively or negatively on patient care or cost-effectiveness. In addition to the above topics the PIP interview will explore their perception of the training programme and its utility.
Conduct and analysis: All participants invited to interview will be given an information sheet and consent form prior to participation. Ideally, interviews will be held face to face at a location of the interviewee’s choice but virtual modes will be considered for logistical reasons. All proceedings will be audio recorded and transcribed verbatim. Thematic analysis will draw on the NPT framework but an inductive approach will enable recognition of unexpected emergent themes. Data will be managed in NVIVO.
Documentary evidence: Minutes of meetings will provide researcher-reported information on barriers, facilitators and other confounding factors that may have affected delivery of the trial eg (recruitment challenges, reach)
Data integration/synthesis
Once all process and main trial outcomes are reported, all the data sets (qualitative and quantitative) will be integrated19 to identify and clarify causal pathways, potentially explain unexpected outcomes, and identify optimal intervention contexts. Should the main RCT findings suggest the CHIPPS service is effective and efficient the process evaluation will inform recommendations for implementation into routine services. The process evaluation will also be interrogated to understand reasons why the intervention has not been successful, including variable success rates in different sites.