An exploratory mixed methods project incorporating participatory co-design principals was planned to actively involve those who will become ‘users’ of the tool throughout the development process (13). Such user-centric methods included individuals with lived experience of clinical placements (i.e. students, lecturers, supervisors, etc.) engaged as active design partners to generate ideas, prototype, gather feedback and make changes (14). Incorporating these principals, the aim was to develop a deep understanding of clinical placements and relevant high utility assessment approaches. The project was undertaken and supported by a project team of 10 nursing academics in seven Australian tertiary educational institutions across three states. The project included a Phase 1 tool development stage, incorporating six key steps, and Phase 2 pilot testing.
Ethical approval for Phase 2 of the project (pilot testing) was obtained from the lead institution (Federation University Australia B19-070) with reciprocal approval from a further six institutions/pilot sites. Informed consent was required based on the participant information sheet provided at the start of the survey. No incentives, such as gifts, payments, or course credits were offered for participation.
Phase 1: Tool Development
Stage 1: Literature Review. A literature review was conducted to identify existing placement evaluation instruments. Ten original tools published between 1995 and 2015 were identified, incorporating a total of 303 rated items (e.g.1; 9; 15; 16; 17)
Overall there was a lack of contemporaneous language, international and cultural differences, grammatical and translation errors and outdated contexts. Further, from a feasibility perspective, most tools were considered too lengthy with the majority including over 30 items.
At this stage the project team decided not to include negatively worded items based on their tendency to cause confusion. Acquiescence was thought to be unlikely as participants would be rating personal clinical experiences (18). Further, for feasibility, transferability and dissemination the tool was developed as a one page document, with generic questions that are applicable for clinical placements in Australia and with potential for future testing in other health professions and countries.
Stage 2: Review of published items. Two researchers reviewed the identified items, removing duplications and non-applicable statements, leaving 190 items for consideration. An expert panel of six clinical academics (mean years of nurse registration - 32) rated the ‘Relevance’ and ‘Clarity’ of these items to produce an Item Content Validity Index (I-CVI) (19). This enabled the exclusion, after discussion, of items that did not reach an acceptable level, i.e. an I-CVI of < 0.78. Approximately half the items were relevant and clear and were retained for further deliberation. Finally, several items from other broad generic training evaluation tools were selected e.g. Q4T (16) and H-PEPSS (15) with the intent of triangulating items with data generated in the Nominal Group meetings in the selection and adaption stage (described below in Stage 4).
Stage 3: Nominal Group meetings. The Nominal Group Technique (NGT) is designed to generate ideas, explore opinions and determine priorities (20), with previous use in, for example, diabetes education (21) and emergency care (22). The Delphi Technique is an alternative consensus generating approach, however questionnaires are circulated anonymously, as opposed to face-to-face meetings in the Nominal Group Technique, enabling a greater exploration of the field of focus (20).
Two Nominal Group University based meetings were held, one in the State of Victorian and the second in the State of Queensland, Australia. The aim was to generate ‘fresh’ or ‘novel’ additional question items related to clinical placement quality from participants with first-hand experience. In order to comply with the co-design principals of the PET project we recruited a convenience sample from a range of stakeholders in each University region to attend one of the two three-hour meetings. Participants were recruited by a researcher at each site aiming to ensure adequate representation. In the Victorian group two 2nd year students, three 3rd year students, two graduate year nurses, one clinical placement coordinator and one clinical educator attended. In the Queensland group two 2nd year students, five 3rd year students, two clinical placement coordinators and two nursing academics attended. Total attendees for the two groups was therefore 20.
The Nominal Group Technique is described in detail elsewhere (23) but in summary the process included:
- An introduction to the project aim and the NGT process.
- Silent/individual generation of potential survey items on cue cards.
- Round robin listing of items with discussion.
- Group discussion and clarification of items.
- Ranking of items.
- Review and discussion regarding final listings.
By the end of each meeting a set of high priority evaluation statements was identified based on individual participants’ ranking. Ranking was achieved by accepting only high priority items prioritized by at least three participants. Fifty-six items in total were carried over to the next stage.
Stage 4 - Selection and adaption of items. The principal researcher (SC) performed an independent primary analyses of items, followed by a five-hour meeting with three additional clinical researchers. Their clinical experience ranged from 27-37 years (mean 32). Potential items from the above stages were selected, adapted and thematisised using a paper based tabletop approach. The principal researcher’s initial development was then used as a reference point/check aiming for consensus. Individual items were listed under key themes e.g. supervision, the culture of the clinical environment, learning outcomes. A priori specification of items to Kirkpatrick’s evaluation model (24) - Level 1 (Reaction to the experience/clinical environment), Level 2 (Learning outcomes) and Level 3 (Behavioural change/practice impact) was also performed at this point. Items were then selected and wording was adjusted if necessary, generating a 20 item questionnaire.
A five point Likert scale was selected with a scale ranging from (1) ‘strongly disagree’ to (5) ‘strongly agree’. An even numbered scale (forced choice) was not selected as participants were likely to require a mid-point response i.e. ‘neither agree or disagree’. Further, a five point scale enabled a direct concurrent validity comparison with another validated tool - the Clinical Learning Environment and Supervision Scale (17) (described below). A 20th item was included, as an overall satisfaction rating, with a response scale of 1 (very dissatisfied) to 10 (extremely satisfied).
Stage 5 – Tool review (educators and students). The draft tool was then circulated to 10 clinical educators from the Australian states of Queensland, New South Wales, and Victoria and to 12 nursing students from Queensland and Victoria, in order to calculate the I-CVI prior to final selection. The expected I-CVI of >.78 was exceeded for relevance and clarity in all but three educator rated items, which were resolved with minor changes to wording. For example item 6 was originally worded “Patient safety was integral to the work of the unit(s)” and was reworded to “Patient safety was fundamental to the work of the unit(s)”, to ensure greater clarity.
Stage 6 - Deans of Nursing review. A final review was provided by 37 Deans of Nursing and Midwifery (Australia and New Zealand) at a meeting in Queensland (July 2019) where minor wording changes to the demographic section were adopted.
Phase 2: Pilot testing and validation
Stage 1 - Pilot testing. The tool was pilot tested through an on-line survey at six Australian universities and one Technical and Further Education (TAFE) institution where Bachelor of Nursing degree students were enrolled (i.e. excluding Enrolled Nurse trainees). These sites were selected as they were led by a project team member who was also the Dean of School or their representative. One site ran a two year graduate entry Masters program whose students were excluded and a double degree nursing/midwifery four year program, where students were surveyed only after a nursing placement.
Purposive population sampling aimed to include all 1st, 2nd, 3rd and 4th year nursing students who had completed a clinical placement in 2nd Semester (July 2019 to-February 2020). Invitations to complete the PET were distributed by a clinical administrator at each site, who provided the survey access link and distributed e-mail reminders. In this pilot testing phase, participants and their review sites were not identifiable. Participants were asked to rate their ‘most recent’ clinical placement only.
The survey was uploaded to Qualtrics survey software (Qualtrics, Provo, UT, USA) enabling anonymized student responses. Three academics tested the survey for accuracy, flow, and correct response options. Access to the Participant Information Statement was enabled and consent requested via a response tick-box. Seven questions regarding demographics were included e.g. age group, year of study course, placement category. This was followed by the 20-item PET and two open ended questions relating to students’ placement experience and suggestions for improving the PET. Access to the survey was enabled via smart phones and computers. The survey remained open between July 2019 and February 2020 whilst students were completing their placements. Finally, 62 students were approached at one university in order to measure the concurrent validity of the PET against the Clinical Learning Environment and Supervision Scale. The test-retest reliability of the PET, with the same test seven days later, was reported by 22 students from two universities.
Stage 2 - In this final stage the aim was to confirm the validity, reliability and feasibility of the PET using applicable statistical and descriptive analyses. Outcomes are described in the results section below.
Survey data downloaded from the Internet were analysed using IBM SPSS vs 26 (25). Descriptive and summary statistics (means, standard deviations) were used to describe categorical data whilst between group associations were explored using inferential statistics (t tests, ANOVA). Pearson’s product moment correlational analysis of item-to-total ratings and item-to global-scores was conducted. The Intra-class Correlation Co-efficient (2-way random-effects model) (26) was used to examine inter-item correlation. P= <0.05 was regarded as significant. The internal consistency reliability was computed using Cronbach’s alpha.
A Principle Component Analysis was conducted to identify scale items that grouped together in a linear pattern of correlations to form component factors, using the method of Pallant (27). The sample exceeded the recommendation of at least 10 participants for each variable. The factorability of data was confirmed by Bartletts’s test of sphericity (<.0.5) of p= <.001 and the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (range 0-1, .6 minimum) of .97. The high KMO of .97 indicates a compact range of correlations with data appropriate for factor analysis (28 p.877). An eigenvalue >1 was applied to extract the number of factors and a Scree plot showed two components. The correlation matrix was based on correlations above .3. Assisted by the large sample, the variables loaded strongly, as described below.
Prior to analyses the normality of the total scale score was confirmed by the Kolmogorov-Smirnov statistic (0.148, df 1263, p= <0.001) and Shapiro-Wilk Test (0.875, df 1263, p= <0.001). Although positive skewness was noted with scores clustered towards higher values (Skewness: 1.327, Kurtosis: 1.934), these data were within the acceptable normal distribution range (27).