i. Results from Repeated Community Score Cards
At the public dispensary, the community identified 11 issues to address (Figure 1): five of high severity and two each of moderately high severity, moderately low severity, and low severity. Of the high priority issues, all five showed some improvement by the time the score card was repeated, with two (age discrimination and stockouts) showing large improvement. However, the other three high severity issues - the circulation of inaccurate family planning information, lack of a youth friendly center, and parental support for family planning use - remained moderately high severity issues. Of the other six issues, one saw modest improvement, three saw no change (although these were the three lowest priority issues) and low male involvement in family planning use worsened. Interestingly, the providers had also identified low male involvement as one of their issues, and also reported that the severity of the issue had increased. Another area of accord between community and provider was family planning stock-outs: both parties agreed that stock-outs were significantly improved by the repeated score card. Of the 12 issues identified by providers, all four of the high severity issues improved. There was some overlap between the issues identified by community and providers, but half of the issues identified by providers were behavioral changes desired in patients, or gaps in knowledge about family planning that were present in the community.
The community paired to the public health center reported that eight of their 15 identified issues were of high severity (Figure 2). Of these, 13 indicators improved after implementation of the action plan, with some indicators improving by a substantial amount, most notably respectful client treatment, informal payments, provider method preferences, and extensive wait times; this suggests substantial improvements to the quality of family planning provision. In contrast with the community-derived score card, only three of the 13 issues identified by providers were seen as high priorities, two of which were also big concerns for the community: lack of a youth friendly center, and poor parental support for youth family planning use. However, the providers did not feel that high priority issues identified by the community, such as long wait times or demand for informal payments were of the same importance for providers. While the public health center community reported that all issues improved or remained the same, the providers did not have similar results across the board. Providers reported that their high workloads had increased in severity and that other issues such as stockouts, family planning myths, and user preference for specific methods contraceptives had remained the same. The most drastic improvement seen by providers was a reduction in illegal abortion services, whereas the community saw the most improvement in the rudeness and quality of counseling given by providers.
The public hospital community identified 12 issues, of which only one increased in severity (Figure 3). Again, the inclusion of men in family planning was a larger issue than had been seen in the first score card. The public hospital providers also identified male inclusion as an issue but did not see a change in severity. Both the providers and the community agreed that poor parental support for family planning was a significant issue, as was demand for informal payments. Discrimination based on age was also an issue that appeared in both sets of score cards; however, providers ranked this as a low priority issue, and the community ranked this as a high priority issue and did not report the same improvement seen by the providers. Provider absenteeism was an issue identified only by the community, who reported some improvement. Providers were more concerned with the support, funding, and space available in their facility, while the community focused on provider and community behaviors and attitudes.
ii. Results from Mystery Clients and Unannounced Visitors
A comparison of data provided by mystery client and unannounced visits conducted in 2019 (baseline) and again in 2021 (endline) are summarized in Table 1 and described for each of the three participating facilities below.
Pilot Facility #1: Dispensary
At baseline, all mystery clients were denied a contraceptive method because the facility first required a pregnancy and/or HIV test prior to offering contraception; as such, we were unable to measure informal payments in this facility at baseline. Regarding provider bias, one of the three mystery clients reported she was treated with scorn by the provider, indicating “… when I assured her I am not pregnant she sneered at me and looked at me badly.” Despite arriving at the facility by 8:30am, mystery clients reported waiting, on average, just over one hour (62 minutes) to be seen because providers either arrived late or engaged in ‘storytelling’ until 9:30am. Although there were four providers on the duty roster, three (including the facility manager) were absent at both unannounced visits; the single provider present at both unannounced visits reported that the absent providers were running personal errands at the first unannounced visit and was unsure of the reason for their absence at the second unannounced visit.
Following the CSC intervention, mystery client visits were repeated, with both mystery clients offered family planning methods, all of which appeared to be in stock and offered free of charge – an improvement over baseline. However, a provider did caution one mystery client that the facility charges 200 shillings[1] for implant removal ‘because it requires a specialist for removal.’ Both mystery clients reported being treated with respect, although one was discouraged from using the intrauterine device because she is nulliparous, with the provider strongly recommending condoms. A similar pattern of wait time was observed at follow-up, with providers arriving at the facility shortly before 9:30am and attending to patients starting around 9:45am, resulting in mystery clients waiting 75 minutes, on average to be seen. At follow-up, no providers were present when the unannounced visitor arrived at 9am. Two providers arrived late (one at 9:30am, no reason offered, and one at 11:15am, due to running personal errands) while the other two were absent for the whole day, due to authorized personal leave and off-site duties.
Pilot Facility #2: Health Center
At baseline, one in three mystery clients was asked to pay an informal fee. One mystery client (implant user) reported that the provider “said family planning services are free but then I buy him tea at 100 shillings and then he will do the procedure for free.” No instances of disrespect or of providers refusing to offer family planning for reasons related to age, marriage, or parity were reported at this facility at baseline. However, one mystery client reported their provider discussed future plans to refuse family planning to patients who arrived at the facility without their partner. Two mystery clients waited over an hour to be seen while a third reported she was attended to within 15 minutes of arriving, for an average wait time of 69 minutes. There were five providers on the duty roster for this facility; three of these were absent at the first visit by an unannounced enumerator (two were on vacation and another, the manager, was at an off-site meeting) and all providers were absent at the second unannounced visit, as they were running personal errands or were absent for unknown reasons.
After the CSC intervention, both mystery clients were asked to pay an informal fee for family planning, in the amount of 50 shillings, and the provider indicated these fees were sanctioned by the Ministry of Health[2]. No instances of disrespect were reported and both mystery clients were seen promptly upon arrival (six-minute wait time, on average). At follow-up, our project identified seven providers as routinely offering family planning services and all seven were absent on the day of the unannounced visit at 9:00am, with the first provider arriving at 9:06am. Three of the seven providers were out all day on authorized leave while the other four were late due to running personal errands (with the last three arriving between 9:30 and 10:20am).
Pilot Facility #3: Sub-County Hospital
At baseline, informal fees ranging from 50 to 200 Kenyan Shillings and were solicited from two out of three mystery clients, who were attended to within 15 to 45 minutes of arriving at the facility. Although the baseline wait time was shorter at this facility (on average, 21 minutes), providers reportedly rushed during family planning counseling, with one mystery client reporting, “The providers came early but kept on story telling for about 19 minutes before calling me in. She was also in a hurry hence attended to me hurriedly despite the fact that I was the only one seeking the service.” No instances of provider disrespect or reluctance to offer family planning to young, unmarried, or nulliparous women were reported. This hospital had six family planning providers on their duty roster; half were absent at the first unannounced visit (primarily due to being on vacation) and all were absent at the second unannounced visit, due to a mix of sanctioned and unsanctioned reasons including being on leave, being on night duty the night before, and being late to work.
At follow-up, one of the mystery clients (implant user) was not offered a method – after waiting over one and a half hours to be seen because the provider was late to work – because implants were stocked out at the time of visit, although the mystery client was informed by her provider that all methods are offered for free at the facility. The other mystery client was attended to by a laboratory technician (untrained provider) who offered her the injectable and then asked her to pay 100 shillings, because ‘it took an effort’ to provide her with the service she needed. No instances of provider disrespect or age bias were reported by mystery clients. On average, the two mystery clients waited 64 minutes to be seen by a provider, a substantial increase over baseline. At follow-up nine providers were routinely offering family planning and three of these were present at the time of the unannounced visit. Of the other six, three arrived late, between 10am and 10:30am. The other three were absent for the entire day, two due to authorized leave and one due to personal errands.
iii. Results from Service Statistics
Facility-level service statistics are presented in Table 2. Both intervention sites and the average performance in other public facilities of the same level experienced a decrease in the number and proportion of women of reproductive age accessing family planning services. Four months pre-intervention, the proportion of women of reproductive age accessing family planning in the intervention hospital (62%) was higher than the average uptake in all hospitals in the subcounty (55%). The family planning uptake among all women of reproductive age in the intervention hospital reduced by a larger margin than the average of all hospitals in the subcounty. Similarly, the proportion of women of reproductive age accessing family planning services reduced by a larger margin in the intervention health center and dispensary compared to the average of all health centers and dispensaries in the subcounty, respectively. The drop in the number of women accessing family planning was consistent across all age groups. For example, the average number of women aged 15-19 years accessing FP services fell (and by a larger amount in the intervention health center and intervention dispensary compared to the average of facilities of the same level) while there was a modest increase in the number of women aged 20-24 years accessing FP in the intervention hospital and health center.
iv. Results from Focus Group Discussions
Table 3 describes the themes that emerged from this analysis, along with illustrative quotes.
Facilitators of the Community Score Card
As seen in Table 3, data from youth focus group discussions indicate that implementation of the CSC was aided by the help of community health volunteers (CHVs). CHVs assisted facilitators in encouraging youth to attend CSC meetings, especially when parents were reluctant or did not trust the youth facilitators. One youth facilitator said that “We were using the CHVs to help involve the community… It would be difficult to face a parent. Or let’s say it’s a male parent… they don’t believe in the youths. So... we were using the CHVs to help in that area.” According to youth facilitators and providers, support from community members also aided the score card process. Youth facilitators said community members were “ready” (422, 100), “had willingness to learn more,” “gave us an easy time to communicate with them,”, and “were very interactive.”
Some providers were also open to learning community perspectives and changing their own behaviors, which facilitated the score card process. One provider noted that participating in the score card “informed [them]. Like maybe we were weak. It was a good forum. Someone could talk what they have at heart.” When speaking of informal fees charged to community members seeking family planning services, another provider said, “If it is issues of charging clients and that is making clients not access the services, it should be improved on so that the clients get the services for free.”
Barriers to the Community Score Card
Youth facilitators reported barriers to recruiting youth to attend CSC meetings due to parental reluctance. Parents were sometimes unwilling to allow their children to attend CSC meetings because of stigma against youth using family planning and because past family planning programs recruited youth without parental consent. One facilitator said, “When you approach a ...female youth and the parent refuses: ‘You want to go…you want to take her to family planning without the parents’ consent?’ ... So even if you try to explain that this is like collection of data the parent won’t agree because sometimes back youths were picked from the community and taken ...for family planning without their parents’ consent.”
Providers also noted barriers to the extended score card process presented by the mobile and transitory nature of youth. One provider said that, “You know youths, if you pick them to work for a duration, … one who has gone to college, others have moved away, others have gone to Nairobi. So, you find out that it is difficult to work with them.” Providers suggested that facilitation by CHVs, rather than youth, could improve the process.
Youth facilitators observed that some negative provider behaviors continued even after providers agreed to address service barriers to family planning. After providers said “they valued clients in their facility” during the CSC, negative provider behaviors persisted. In one example, after a community member left a facility without receiving services due to a rude provider, CHVs raised the problem again with the facility. This CHV was told “the problem would be taken care of. But later on, it’s like they didn’t do anything about it.” Thus, providers were not only not addressing barriers brought up in the CSC meeting, but also ignoring CHVs who tried to help community members.
Lastly, some providers noted that the score card process was time consuming and noted wanting increased compensation for their time. One provider commented, “Yes, because you sit for long and you participate, the expectations from the program are high but... we are not appreciated. You know just finding time, sitting down talking, no refreshment, and the reimbursement is little.”
Successes of the Community Score Card
The focus groups with YWG members reported CSC successes included encouraging the community to speak openly about barriers to family planning use, identifying solutions to those barriers, and in some cases, successfully implementing solutions. As one youth facilitator said, “The experience was good: you get to interact with the youths, you know the problems, the challenges…and what people are going through to access medical services here… and what can be done to help the problems that they are going through.” The main barriers to family planning usage at the facility level identified by community members included negative provider behaviors, stock-outs, discrimination against adolescents, and informal fees. The CSC process was used to identify several possible solutions, some of which were successful in addressing barriers. For example, the CSC process led to youth being given “good time” at health centers, where they were “advised and talked to about the family planning before they insert it. [Adolescents were] told about the challenges and importance” of family planning. This led to youth, “even the ones that were being criticized ... now coming to get the service.” The CSC also identified that provider behaviors, especially speaking harshly to patients, were a significant deterrent. Youth facilitators reported one benefit of the CSC was “The way they [providers] were talking was just good as compared to the shouting before.”
From the provider focus group, the main success of the CSC was the increased dialogue and communication between the providers and the community. The providers felt that the CSC “was an educative experience because we were able to sit with the community representatives and understand how the community views the health facility.” The providers characterized the meetings as “so healthy” and agreed that it was an opportunity for the community to understand the issues facing the clinic, as well as for the healthcare providers to understand the perspective of the community members. Providers also felt that the CSC allowed them to educate the community in ways that improved the healthcare the community members later received, as the community “[is] able now to access these services with an informed mind.” Finally, providers were also able to identify areas to improve their practice: for example, one provider said, “we should also ensure that we provide quality and efficient services to the youth,” and identified increasing facility staff as a way to increase service quality and efficiency. On a more personal level, providers appreciated the CSC as a way to remind the community that they are only human: “They are able also to understand that even the health care workers who are working there are also human beings... we try and ensure they understand us as we understand them.”
Deviations from Original Community Score Card Plan
Unfortunately, both the youth facilitator focus groups and the provider focus groups reported some significant deviations from the original intention of the CSC. The reasons why youth facilitators opted to deviate from their extensive training is unclear. The major deviations revealed by the youth facilitator focus groups centered on two related issues: first, the community misunderstood the role of the score card meetings in the community, believing the meetings were intended as a forum for family planning education rather than a dialogue and action plan for family planning quality improvement. Secondly, the youth CSC facilitators misunderstood their role as CSC facilitators and mistakenly identified more as family planning educators than as CSC facilitators. The combination of these two deviations from the design of the CSC intervention led to several challenges. For example, in some communities, parents were reluctant to let their youth, especially their daughters, attend score card meetings because they believed that these meetings were intended to give their children family planning services or education. This may be in part because so many youth facilitators spoke about educating others about family planning as part of their role as CSC facilitators. Some of this “family planning education” occurred as part of encouraging youth to come to CSC meetings by addressing myths about family planning users, as explained by one facilitator: “And then we tried to make them now come... like for example they were talking of myths and lack of information. Some people are not…they don’t know anything about the family planning, so talking to them about it made them to change their minds.” In other cases, it appears that during CSC meetings, youth were asking questions about family planning: “They wanted to know more... like the best methods that they can use for family planning,” which led to youth facilitators “teaching them maybe how to use condoms” or engaging in a ‘family planning educator’ role that they had not been trained for.
The provider focus groups also showed misunderstandings in the purpose and goals of the CSC. While the CSC was intended to allow the community to hold their healthcare clinics and providers accountable for delivering a high quality of care, some providers viewed the CSC as a way to justify or explain poor quality of care and avoid responsibility for quality improvements. When community members expressed frustration at long wait times, providers responded by “telling” and “mak[ing] them understand that it is a service like any other...so when they come and find a client coming for another service, it is okay for them to wait.” Providers also used the CSC to justify informal fees to the community as a consequence of stock-outs or a lack of supplies. It appears that some of the providers viewed the CSC as a forum for teaching the community to be more patient or calm. Providers also wanted the community to know that “if there is anything wrong, they should not escalate it as such because the health care workers are also human beings and they are doing different things so you come and find someone with his or her own…you can never know how they woke up.” This shows some of the discomfort providers felt with the changing power dynamics brought by the CSC, as they characterized themselves as “trying to moderate” as the CSC facilitators had “some of that authority [to go] overboard a little bit.”
While discomfort of some providers with CSC activities emerged slightly in the focus groups, providers were reluctant to clearly criticize the CSC when responding to the short provider questionnaire, with none of the providers characterizing the experience as negative or unsettling. Instead, provider responses during the short provider questionnaire were overwhelmingly positive, such as, “Love feedbacks both positive and negative…Take negative comments positively in order to change and improve services for clients… Would wish for more activities of score card because they make service provision improve” (dispensary provider); “Made providers know weaknesses as well as strengths” (health center provider); and “It [the CSC] became an opportunity to be better because gaps were highlighted and as a provider got to work on them. It is a good initiative. Given chance, should be done more often” (hospital provider).