Baseline characteristics of the qualitative study phases
For phase one, the target was a cohort of nine study participants attending a continuous professional development nursing course. This cohort had only female nurses but the gender disparity is not unusual as more than 75% of nurses in Kenya are female [44]. Their median age and median experience in years was 34 and 6 respectively (Table 2). In phase two, five participants were interviewed, all of whom were paediatricians or paediatricians-in-training, including three females and two males (see Table 1–2 for group characteristics). In phase three, 19 participants made up of junior doctors, in-charge paediatric nurses, and nursing students were interviewed by TT. The median age of participants in this phase was 25 years with three years of experience. Across all the phases of the qualitative study, participants were drawn from seventeen different health facilities’ paediatric units and departments spanning private clinics to public referral hospitals, geographically representing different regions of Kenya and reflective of the various professional cadres.
Table 2
Summary characteristics of study participants
Research Phase | Clinical Cadre | Institutions represented | N (% Female) | Age (years) | Experience* (years) |
| | | | Median (IQR) | Median (IQR) |
One | Sub-speciality nurses in-training | 8 | 9 (100%) | 34.5 (30-38.75) | 6 (3–8) |
Two | paediatricians | 4 | 5 (60%) | 32 (31–33) | 8 (7–8) |
Three** | Senior nurses, junior doctors, and students | 5 | 19 (52.6%) | 25 (23.5–33.5) | 3 (2-7.5) |
Total | 17 | 33 (66.7%) | 31 (25-35.5) | 5 (2–8) |
Note: *Experience reflects cumulative years of front-line service, including internship. ** 4/19 of these participants were junior doctors. |
From the silent phase and item generation phase detailed of Nominal Group Technique (NGT) in supplementary 2 (appendix), the participants went through the listed items, discussing with and asking for clarification from each other, with the facilitator (TT) summarising the generated items as listed in Table 3. The participants tended to express their learning experiences using the app in a manner unlinked to specific in-app feedback design instruments even though that was what they were being asked to do. An example of this is item C, G or I in Table 3, where phase one participants attached value to how LIFE influences their cognitive processes. Additionally, during NGT, we felt that the brevity in the descriptions of the items generated by the participants was purposely done to avoid in-depth interrogation by their peers.
Table 3
Summary of design components derived through consensus from item - generation phase of the Nominal Group Technique.
Option | Description |
A | Promote challenge using almost similar multiple-choice options |
B | Accommodate differences in actual workplace practice in the scenarios |
C | The timer adds pressure, anxiety, [*viewed positively, to be retained as a feature] |
D | Give immediate feedback due to the "urgency" of the learning scenario [*as opposed to providing delayed feedback at the end of the scenario] |
E | Improve the utility of feedback to help remember the correct answer [*e.g. by adding more information links] |
F | Give immediate correct answer on first error screen but weight the final score when feedback message is provided to the wrong entry. |
G | Sharpen critical thinking when playing the game [*in reference to use of feedback messages in error dialogues, providing hints etc.] |
H | Add emojis to make enjoyable [* e.g. When reflecting LIFE’s evaluation of learner’s progress] |
I | Challenging, if it’s the learner’s first time interacting with [*ETAT+] materials |
J | Give a rationale for more task information. [*Helps to build learner’s appreciation of task importance] |
K | Add continuation to scenarios [*quantity of training content] |
L | Cascade challenge and complexity level of the content |
Note: * Clarification added through discussions with study participants. **Two options (Give more demonstration of tasks; Reduce navigation difficulty;) were not discussed due to accidental omission by the session facilitator. |
The identified thematic areas from the NGT discussions that generated Table 3 are expounded upon from the next section, and in juxtaposition to results from phase two and three analyses, for purposes of comparison and contrast. Participants in phase one ranked the items listed in Table 4 in order of decreasing priority, based on which items they were most inclined towards, impressed with, or would recommend being included (i.e. voting phase of NGT).
From the voting phase, each cell in Table 4 in rank columns represents the number of participants who voted for that item at that rank position. Given that there were 12 items to be voted upon, and participants were limited to only voting for their top 10 items, row-wise sums of the item votes could be less than or equal to the total number of participants. The weighted average rank, given as the sum of the ranks adjusted for the votes per rank, was calculated as illustrated in Eq. 1.
$$Weighted Rank = \frac{\sum \left(row\left(cellwise votes*rank position\right)\right)}{row \left(sum of votes\right)} \left(1\right)$$
Table 4
Weighted rank of items generated from the voting phase of Nominal Group Technique
Item | Item description* | Rank | Average Weighted Rank** |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
F | immediate corrective feedback on first incorrect try | 1 | 1 | 1 | 2 | 1 | 0 | 1 | 0 | 0 | 0 | 3.71 |
J | Task rationale with more information | 2 | 2 | 0 | 2 | 0 | 0 | 1 | 0 | 1 | 0 | 3.75 |
G | Sharpening critical thinking using feedback | 2 | 1 | 0 | 2 | 1 | 0 | 1 | 1 | 0 | 0 | 4 |
I | Challenging Content | 0 | 2 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 4.5 |
C | Task timing | 1 | 1 | 2 | 0 | 0 | 1 | 1 | 2 | 0 | 0 | 4.75 |
D | Immediate feedback due to learning task urgency | 2 | 1 | 0 | 1 | 1 | 0 | 2 | 1 | 1 | 0 | 4.89 |
E | Improved feedback utility | 0 | 1 | 0 | 1 | 1 | 2 | 0 | 2 | 0 | 1 | 6.13 |
A | Promote challenge using almost similar multiple-choice options | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 2 | 0 | 6.17 |
K | Add continuation to scenarios | 0 | 0 | 2 | 0 | 1 | 2 | 1 | 0 | 0 | 2 | 6.25 |
L | Cascade challenge and complexity level of the content | 0 | 0 | 1 | 1 | 1 | 2 | 0 | 0 | 3 | 0 | 6.38 |
B | Scenarios accommodate differences in workplace practice | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 2 | 1 | 0 | 6.6 |
H | Add emojis to make enjoyable | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 5 | 8.5 |
Note: *The item description provided as shorthand of descriptions given in Table 9.2.1. **The average weighted rank is given as the sum of the ranks adjusted for the votes per rank, with the formula provided in Eq. 9.3.1 |
The next sections of the results analyse responses from all the qualitative study phases through a thematic lens [42, 43], reflecting how experiences within different cadres and levels of speciality training in neonatal emergency care might map onto personalised training using LIFE. The themes identified from semi-structured interviews with participants are discussed in the following sections and are linked to Winnie Hadwins’ SRL model [43].
Learning task definitions, elaboration and performance imperatives
Nursing and student participants interpretation of the aims of learning tasks within LIFE implied that from the game features provided within an assessed task, their attention was skewed towards performance (as indicated by their fixation with final score), rather than task elaboration. Specifically, time spent on a task appeared to be intrinsically associated with the urgency of the learning task, considered inseparable from performance goal. This correlates with the high rank item F (Immediate corrective feedback) in Table 3.
“…As you begin, it took a little bit of time because you had to read the explanations and the questions...So, at the end of it you find that your resuscitation took a bit of a long period. And even as you start the game, you know at the end you are being timed which again I think is a good thing because again with resuscitation, you are dealing with life and death and you have a very short period of time, so your interventions have to be really correct and they have to be timely…” LIFEEXP12
However, paediatrician participants in phase two tended towards downplaying time as a task strategy and instead, sought to use feedback to inspire goals linked to refining skill competencies. In phase two, timing was perceived to be relevant when conditioned on pre-existing learner knowledge or practice and is dealt with in a later section of this paper. This would be more reflective of the rank of item I in combination with item C (Task timing) in Table 3.
“…timing should not be [there], but when you have the pop-up message telling you ‘this is a life-saving procedure, you need to move, you need to make a decision, you need to make a step’ it’s good enough…” KPA002
In contrast, perceptions of phase three participants combined the emphasis of elaboration of the learning task together with its time-linked consequences.
“…Ideally, as nurses, …you need to know because if you mess, if you do not do the right things at the right time the first time, most of the times you will mess up with those children. So, it is good to understand, ‘why do I have to do this?’...” LIFEEXP06
Learning task elaboration (a “More information” button within the LIFE app) and as indicated by item J (Task rationale) in Table 3 was highly valued among the participants in phase one and three, hinged on the perception of LIFE not as a game first, but as a learning tool. This is illustrated in the quotes below.
“…They [feedback messages] are explaining why you should do that so I found it quite educative because there are times when you are not even sure, ‘why would I need to do this?’ but they are able to elaborate it further. If they tell you it is a neutral position, why should the head be in a neutral position? Then they explain to you why. ‘Why would I use this kind of mask rather than this other one?’, you know...” LIFEEXP06
However, phase two participants preferred a standardised learning process that was not seeking within-game tasks elaboration as the preferred emphasis of the game. Given that phase two participants were paediatricians, they would have been exposed to ETAT + hence their agreement on this. They argued for elaboration of the learning goals to be articulated either before or after, but outside the LIFE game itself. This might be attributable to the differences in the level of training, prior experience and specialisation in the participants between the phases.
“…I think you can gauge how much people have internalised if you’ve given the talk and now use their scores and what mistakes they make to like correct things at the lab. So, I feel like it can still be used even without the ETAT setting, if you were just given like a CMEor something. Like do a talk and then tell them, ‘Okay fine I want you to play this game’, then give them their scores at the end, you go like, ‘I realised that this are some of the few things that were hard to clear…’ I feel like it’s giving that talk before and giving them the goals of the game and what you expect them to achieve…” KPA003
Additionally, phase two participants appeared to be in agreement that an unintended consequence of tasks being elaborated in similar fashion to traditional educational tutorials diminishes the gaming experience, which is explained further in the Games and engagement results sub-section. Healthcare providers perceive limited internalisation of learning where there is no elaboration. The elaboration being referenced here not only emphasises the expected goal of the learning task being taught but also a nuanced breakdown of the consequences of [in]action or erroneous action.
Context of practice, cognitive adaptations and goal setting
Phase one and three participants perceptions on how LIFE might encourage personal evaluations were arguably a result of reflections on (1) context of resources and frontline work encountered so far, and (2) the cognitive effort put in weighing options available predicated on prior knowledge.
“…You know down there in the villages, things like a resuscitaire are not available and they actually have to do the resuscitation on a desk and not on a resuscitaire because the hospital doesn’t have a resuscitaire. Or even if they are available, they are not working. A normal resuscitaire should have a warmer at the bottom and the warmers are not even working in the hospital…” LIFEEXP03
“…Stimulate critical thinking…you think about resuscitation materials…I compare with what sometimes is practiced at place of work, like tickling the babies’ feet so I was like ‘I would like to tickle it also’…” NGTCONV
“…We have the scenario, like this is a new-born resuscitation, this is the flow chart, this triggers you, if at all you have done this before, it triggers you because you have the three Ss like shout for help, prepare your setting, wash your hands and all that then examine. So, when you are coming to the game itself like the scenario, when it comes to the choices you are like ‘this is off, this is off…’. But the way [LIFE] answers are, all the enticing ones are there: ‘this looks good, I will do this’. Then you are like ‘Oh! it is not correct’…” LIFEEXP11
Learning content relatability appears to influence self-evaluation of learning progression given a contextualised framing of learning objectives. Admittedly, it is worth noting that such necessary equipment for providing quality healthcare might not be readily available in typical rural facilities, but there is a need to keep the app in line with what should be done so as to continually reinforce best practice [45]. Curiously, the typical learning task strategies especially utilised by participants who had limited prior knowledge of task was use of guesstimation (educated guess) approach as illustrated below.
“…Yeah maybe one or two I guessed, and got... I eliminated the wrong and I arrived on the right…” KPA002
But guesstimation, when linked to the need for elaboration earlier addressed can be viewed as learning by understanding what not to do. Healthcare providers evaluated this learning strategy to be successful if it produced perfect conformity to the clinical care guidelines as evidenced in the quotes below.
“… [The score] represents the kind of management I am doing. Because if you do not score well that means the care you are giving is not right… let’s say you have scored 20%, what is going to be the outcome of this baby? You are going to have a baby with severe cerebral palsy. You are managing in terms of, you know… to come out with somebody who is alive and not having any major complications…” LIFEEXP01
“…I think scoring 70% means you never understood the full training. It doesn’t mean you can’t provide all the care, because even scoring that 70% is still actually part of the training. But a person who scores 100%, there is a tendency for this person to be better trained as compared to the person who is scoring 70. But it doesn’t mean that the person who is scoring 70 cannot take part in resuscitation…” LIFEEXP03
Even in the gamified smartphone-based training linked to clinical scenarios, there is limited perception of internalisation of learning where performance is not ‘automatic’ (i.e. characterised by very quick response to the learning task) and have a perfect score. Performance, while being regarded as not being exclusively constitutive of knowledge mastery, cannot be dissociated from the perception of knowledge mastery. Healthcare providers concern for patient outcomes was heightened by how they perform, but they do not consider this to be representative of their knowledge mastery, which they implicitly referred to as not being directly observable i.e. latent.
Phase two participants made a distinction of whether personal learning goals needed to be set based on (1) previous exposure to ETAT+, and (2) the number of previous plays on LIFE. These they argued, would mitigate any negative effects stemming from learner’s first experience with ETAT + content, and once acclimatised to LIFE and ETAT+, focus on implications of erroneous knowledge on clinical outcomes. Where there was no previous knowledge of the content, the emphasis was on an exploratory learning approach, with goal-setting exercises building on previous rounds of play.
“…For beginners, maybe no, but [maybe] if someone has either played it before or has done it before. I feel like because you, you know the sequence, you’d be like, ‘Okay fine the next step is I want to do it in less time’ … for the purpose of this learning, it, it wouldn’t be that useful…doing the game the second time, I think that will improve on your time and like improve on your skills. Yeah maybe it will be useful for those people, for first-timers it is discouraging…” KPA 003
“…Well for me, I think getting a perfect score, translated to how well I understood the scenarios… like I was reminding myself what I had forgotten about ETAT+. So for me the motivation was other than the high score, whenever I scored higher than I did initially, I was like ‘Wow, I think I am getting better’, you know? it is like I am beginning to showcase whatever I have learnt in that particular scenario and putting it into practice now, like ‘…how well had I gotten the concept and how well I’m I replicating it in the scenario now…’ LIFEEXP10
At a global level, LIFE’s goal-oriented imperative of saving lives at birth was well understood and internalised by the participants in all the qualitative study phases in general, with the clear intended consequence being building confidence to provide care.
“…If I had a life, in front of me that means I would have gotten it wrong when I’m doing an emergency procedure that I need to know, so that was scary… I know it’s more of goal-oriented, to have more confidence, you need to pass, you need to get better…” KPA002
These strategic learning adaptations informed by personalised learning goals which constantly evolve based on perceptions of knowledge mastery have implications on the type of instructional support expected and to what end. We expound on this in the next section.
Feedback and motivation, and transfer to practice
Feedback is the key mechanism for maintaining high SRL behaviour by allowing learners to monitor their engagement and attainment of their learning goals [46]. In LIFE’s case, it served to highlight the extrinsic motivation for learning, which healthcare providers understood to be the adequate management of the neonatal emergency cases. It also unearthed self-efficacy concerns from healthcare providers’ reflections:
”…. I am a healthcare worker, how do I miss that, how do I get that wrong in resuscitating a new-born? But it’s good it’s there because it helped me know, ‘you are wrong on this one’. But on the other hand, … I think I’m very bad if I can get a wrong on this…” KPA002
They perceived their self-efficacy concerns to be inseparable from their extrinsic motivation linked to a higher game score. Feedback needed to build up motivation for personal goal achievement at the individual learning task level in addition to encouraging the global goal of perfect automatic performance. This was reflected in phase one by items D (Immediate feedback due to the learning scenario "urgency") and E (Improved feedback utility) in Table 3.
“Q: Having played it, in terms of the type of feedback you got from LIFE, what would you like to see?
A
Maybe for them [the feedback messages] to say what I could do better, Okay, the overall point of the game was that we did it and the baby cried, I don’t know if it’s possible for them to, to say maybe he should have bagged faster, is there an option for that?...” KPA001
“…maybe it’s to say that ‘you’ve gotten at least X right ...It’s essential that you get this one also right...’ A way to imply that ‘Yes, you’re wrong, still, but try... we need you to improve’. What is the motivation after I get the wrong things? That’s what I’m thinking, what is the motivation? ...” KPA002
“…you can just give feedback at the end of it and tell me ‘at this phase went wrong and this was supposed to be the answer’ and then after revising something and then you go and start doing it, after getting where you went wrong, I will try and remember. Instead if you are done with this step and it tells you immediately… you feel stupid. Personally, I felt so stupid and I was like ‘you don’t know this thing?’...” LIFEEXP02
This illustrates how the corrective feedback messages (a) motivate individual learners to keep reflecting in and on their action and (b) might encourage healthcare providers to apply what they learn through LIFE in routine hospital settings as illustrated in the comments below.
“…[LIFE] is better than nothing because when we see that and when you call your consultant over the phone because you can be doing a resuscitation and you are like ‘this child is actually not responding to the bagging that I am actually doing’, once they talk to you over the phone and tell you that you need to actually give thirty breaths per minutes, if you have actually gone through the [LIFE] app …like three times probably you would have seen the messages somewhere, and it would be better than that someone who has never heard about thirty breaths per minutes…” LIFEEXP03
“…But you know, at first before I played the game, I could have killed so many babies, but right now actually I am so confident, so I can handle it…” LIFEEXP02
Within gamified learning, targeted approaches leveraging on heightened learner emotions are needed to enhance learner engagement [47]. Through such targeting, gamification makes it easy to repeat cycles of game play, evaluation and reflection on learning performance, and adapting the learning strategy for next iteration, free from associated real-life risks associated with clinical learning content.
“…Because I understand that with such games, some of them actually they are more intense in the fact that they tell you that the baby has died… I’ve seen it before in other games and I would think when on emotional level it’s somehow useful, you would want to save the life again. You know, through applications and games, it’s the only time you can revive, resurrect someone to try and save them again…” KPA004
“…You know when you play it [LIFE], it gets so interesting and then you are just like ‘why did I get 75 and I want to get to 100?’...That was my goal! And then the more you play it and the more you want to hate it, but the more you are learning from it. And the more it gets easier…” LIFEEXP02
The reflection and adaptation within the repeated learning cycles are exemplified in the previously mentioned ‘guesstimation’ strategy where healthcare providers with partial understanding of care giving procedures try to deduce the gap in their knowledge by a partial elimination mechanism when responding to learning tasks, or reflect on their experience in routine clinical practice. Healthcare providers perceive their smartphone based SRL to be limited when their individual motivation for reflective learning is not targeted by the platform nor does learning address confidence and courage to act in the routine clinical setting. This perception of limited learning reflects on their views of whether they are acquiring knowledge mastery (despite good performance) through SRL using gamified platforms.
It is peculiar that this subsection was barely mentioned in phase one, except as primarily intended to produce better scores instead of better capacity to respond to personal growth areas in delivering emergency care. These differences might be partially attributable to differing level of experience mediated by specialisation training, or due to the methodological differences between phase one NGT and the complimentary phases two and three.
Game experience and engagement
As a gamified smartphone-application, healthcare providers provided their views on the trade-off between content, design, and provision of supplementary learning resources in enhancing the game experience. In this regard, while phase one participants would want more task elaboration, phase two participants reflected on its perceived negative impact on the gamified experience.
“…You don’t want to go so far from being in the game and actually is a bit of a game and a massive scientific quiz and research and information overload. Because people can get information from books and things…keep the game as a ‘this is how we resuscitate babies, lets practice doing it, let’s get into a routine, this is the protocol to follow’, not digging so much in to the science behind it….” KPA005
“…I feel like if you added anything it would be too long, and someone would get bored in-between, but it would be a tough balance to add something and still keep people interested…” KPA003
However, the effort healthcare providers applied in learning tended to be low due to perceiving the platform as not requiring utmost serious-mindedness while using it. This could be attributed to a limited experience of immersion while using LIFE [47]. Such a learning posture towards gamified learning might hamper its acceptability as a novel modality of teaching if does not target to deeply immerse the learners [47].
“…You know like the way I was playing it, at first, I was playing it while watching movies but now like when someone tells you play this game there is a qualification you will do it with a lot of seriousness in it… I would play it with a lot of care and a lot of concentration…” LIFEEXP02
“…Probably I would have been keener because I told you that in one of the scenarios, class was going on and I was just here [playing it]. So, I wasn’t concentrating, so probably I should have been keen and taken it as more of life and death situation. As opposed to treating it as just a game. So, I think if I would have been keener… I would have been more meticulous…” LIFEEXP10
“…I think there will be some initial resistance from people trained on mannequins and real babies…people saying, ‘is this [form of] paediatrics training better?’ …” KPA005
Effort regulation will produce higher goal attainment where gamified platforms such as LIFE ensure heightened keenness and sombreness of healthcare providers learning through it. Additional incentives to foster such postures towards SRL behaviours need to be further explored.
From the key findings, internalisation of the learning task would be maximised in the presence of feedback that elaborated on the consequences of [in]action or erroneous action. Such internalisation would be in response to learning content that was relatable due to its reflection of the clinical context and would help foster an engaging learning journey. Healthcare providers perceived themselves to have not internalised the learning content where their performance was not perfect nor were their responses to the learning tasks automatic. They also perceived achieving a perfect score as not being fully constitutive of knowledge mastery. Heightening the keenness and emotions of healthcare providers while learning was perceived to lead to enhanced learning and optimised effort regulation [47]. From the study participants, the validation that they had met their learning objectives was when they perceived that their courage to act in a routine clinical setting had been enhanced from the confidence in the knowledge gained produced in a healthcare provider. These key findings, which represent how the interviewed healthcare providers perceive SRL to be linked to their personal learning objectives, are illustrated by Fig. 1.