Phase One - Development
Part 1—Identification of the items for inclusion on the draft checklist
A total of 205 items were identified from the literature review of the RAND consensus panel of international experts [11]who considered 37 were of “high priority”. Following comparison with the CQC, CQBT and UK mandatory requirements and the review of the study team for repetition and clarity a total of 13 items were included on the draft checklist.
Part 2—Refinement of the draft checklist to produce the final prototype
A total of ten East Midlands (EM) and six Greater Manchester (GM) practices tested the draft checklist. Within these practices a total of 38 respondents (23 from EM and 15 from GM) completed the pilot version of the checklist and accompanying questionnaire (23 GPs, four nurses, nine practice managers, and two administrative staff). An additional 25 staff from participating practices were interviewed with regard to the applicability of the PST provided feedback on the checklist.
At this stage the checklist was divided into four sections (information flow, safety information about the practice, prescribing, and use of IT systems), each with an introductory statement that was taken directly from our project taxonomy of patient safety [16].
A summary of the results from the development of the prototype tool can be found in the following tables each relating to one of the domains. Within each we define the domain, present the draft items, the rationale and evidence for each, the changes made as a result of the feedback collated from the questionnaires and semi-structured interviews and the related item as it appeared on the prototype checklist which numbered nine in total.
Phase Two - Testing of the prototype checklist
Eight participating practices within North Staffordshire (NS) agreed to test the checklist. A representation of the characteristics of the practices participating in the toolkit project can be found in Table 2.
Quantitative data
Table 3 shows the percentage of practices which answered yes to final checklist items. Items with a response of ‘No’ indicate that the practice might need to make a change to its systems if they do not feel that they have addressed a checklist item. The two items with the lowest percentage of ‘Yes’ responses (a quarter of the staff members did not think their practices achieved these safety goals) were item 6 regarding the failure to monitor the non-collection of prescriptions and item 7 around checking vulnerable patients following discharge from hospital. Several items received scores of 100% including the appropriate handling of incoming clinical information and the timely follow-up of abnormal results.
Qualitative data
We identified two key themes relating to the characteristics of the intervention and the environmental context. Within each a series of sub-themes were identified (summarised in Table 4) and below we describe each of these alongside exemplar quotes.
Intervention characteristics
This included the overall design and content of the checklist and the intended user and the frequency of its use.
Relative advantage
This describes the stakeholder’s perception of the advantage of implementing the Concise Safe-Systems Checklist (SSC) as opposed to maintaining existing practice [31]. Any tool or instrument designed to improve safety of care can also improve aspects of care in other respects as patient safety and quality of care are so intrinsically linked. In terms of the advantages of using the SSC, staff described how they improved patient safety directly in terms of prescribing safety and enabling the review of existing systems, but also indirectly by using it to provide a framework to discuss patient safety with the broader practice team.
Prescribing safety
A number of participants commented on the benefits of using the tool to improve medication safety and one practice manager felt the section on medications was the most useful.
“The medication things I thought was probably the most useful section… they say the most errors in a general practice are made on medicines…” Practice Manager, P03
The other area that the SSC appeared to be effective was at highlighting the non-collection of repeat prescriptions. One GP acknowledged how this item had raised awareness of the issue and a practice manager how it had encouraged them to discuss the issue with other members of the team
“Non-collection of prescriptions, that’s the one that we found that we weren’t doing very well… because we’re moving to electronic prescribing in a couple of weeks’ time, we’ll look into that, that way…” GP, P02
“The non-collection of prescriptions was good and that did encourage me to talk to the dispensing team—”what did they do with those?” " Practice Manager, P07
Staff engagement
Participants described how using the SSC indirectly benefitted patient safety by helping engage a range of staff. Although the tool was designed to be used by a single individual frequently, its completion would or could rely on other members of the practice team, helping raise awareness of patient safety.
“So we found it on several levels a really useful tool and not least, of course, patient safety, but in terms of actually being another vehicle to encourage cross-team understanding within the practice, as well.” Practice Manager, P01
One Practice Manager felt that the document could be used to frame a discussion with GPs on whether policies and procedures were implemented as expected.
“…it’s quite straightforward, I’ll just run through everything with the GPs instead of saying ‘yes, we do this’… I mean you can have policy and procedure and no-one can follow it.” Practice Manager, P06
Review existing systems
It was noted, how as a whole, the SSC provided the opportunity to look again at the safety of existing systems that due to familiarity might otherwise be overlooked.
“Actually, it gives you the chance to reflect that some of the things [we do] are a system and to think, ‘Oh, yes!’ Something like mail-handling is, like so embedded …we take 500 letters in… every day, scan them in, pass them round and whatever - that, you know, you can almost forget that that is a safe system.” Practice Manager P03
Training staff
Another way in which the SSC may indirectly benefit patient safety is by its use as a training tool for clinicians in the early part of their career. One practice manager described how it presented a useful overview for inexperienced clinicians.
“One thing I thought it would be …a good training tool for, like, an overview…These things would be good for, like, GP registrars and things, like in training… it’s a good overview position.” Practice Manager, P03
High level approach
The benefits of the high level approach adapted by the checklist as a way of immunising specific items against local or sporadic change were described.
“I think one of the things that’s hard … with the checklist, is… keeping it up to date as things change so fast in practice, but a lot of your sentences are quite high-level, so it means that it lasts…” Practice Manager, P03
Adaptability
Adaptability describes the degree to which an intervention can be tailored, refined or reinvented to better meet local needs. The flexibility of the SSC in terms of how frequently it could be used was noted.
Frequency of use
There was no prescribed time interval in between using the SSC, meaning that practices could decide how often it could be used. One practice manager described how they might use the tool monthly and another how they would use it on an annual basis.
“…If you’re doing it monthly, you’re more aware of the questions in your head, aren’t you, so it’ll become more of a routine. So, yes, I think it would [be monthly], in the long term.” Practice Manager, P04
“I think once you’ve checked through it, it might be worth just going through it on an annual basis, just to make sure that you are doing these things….” Practice Manager, P06
“I would very much like to see this as an annual event, in practice.” Practice Manager, P01
Design Quality
The design quality describes the perceptions of users of the quality of its design. The primary design element which participants commented on was its ease of use.
Ease of use
The SSC was considered well structured and easy to follow, which meant that it was quick and easy to use.
“I think because it is quite brief it’s quite a useful thing, just a pointer to go through it and make sure that these things are still being done as they should.” Practice Manager, P06
Environmental context
The theme of environmental context relates to the influence of factors external to the design of the tool and the organisation, specifically the ability of the practice to meet the needs of their patients.
Lack of capacity
One factor that may inhibit its further use was the limited capacity, in terms of time and workload in primary care. Despite not knowing the length of time it would take to use the tool, a GP at one practice asked a part-time member of staff to be responsible for the tool because of concerns over their own lack of time.
“Because we were just totally snowed under, so I knew I wouldn’t have time to do this so I asked my colleague who only works part time and did that for me. So he’s… done the Safe Systems questionnaire.”—GP P02
One practice manager was positive towards the SSC but cautioned that its future implementation might depend on the ability of practices to meet the twin pressures of time and resource.
“As much as I am a big fan of this tool, I think the two key issues are finding time and, if it involves any resources, is actually finding support for those resources because that’s always challenging in this day and age.” Practice Manager, P01