Validation
Validation of the E-MPATHY Draft I demonstrated good content and face validity. The I-CVI was 1.00 from all items in introduction, verbal autopsy, form for maternal death in facilities, clinical standards, reviews, narrative summaries of a committee worksheet, and plan of action. The S-CVIs for all items indicated an excellent content validity (S-CVI=1.00).
Inter-rater reliability
One-hundred-thirteen (113) maternal deaths were reviewed and included in the inter-rater reliability analysis between the district and provience reviewers using E-MPATHY. Using the ICD-MM, the most common underlying causes of death were considered. Table 1 compares the ICD-MM underlying cause groups and summarizes levels of agreement between the two MDR committees. The overall kappa (k) was 0.86 (95% CI: 0.76-0.96); p<0.001), with the highest agreement for abortive outcomes (k=1), obstetric hemorrhage (k=0.96), and non-obstetric complications (k=0.82). The lowest agreement was for pregnancy-related infection and other obstetric complications with k=0.71.
Table 1. Level of agreement of ICD-MM group classification for underlying causes of death by district and province MDR committees
ICD-MM Group number
|
ICD-MM group name
|
District team
n (%)
|
Province team
n (%)
|
Kappa
(95% CI)a
|
Kappa
(95% CI)b
|
1
|
Pregnancy with abortive outcome
|
2 (1.8)
|
2 (1.8)
|
1.00
(0.87-1.00)
|
|
2
|
Hypertensive disorders in pregnancy, childbirth and the puerperium
|
31 (27.4)
|
31 (27.4)
|
0.82
(0.64-1.00)
|
|
3
|
Obstetric hemorrhage
|
41 (36.3)
|
41 (36.3)
|
0.96
(0.78-1.00)
|
|
4
|
Pregnancy-related infection
|
5 (4.4)
|
6 (5.3)
|
0.71
(0.53-0.89)
|
|
5
|
Other obstetric complications
|
6 (5.3)
|
5 (4.4)
|
0.71
(0.53-0.89)
|
0.86
(0.76-0.96)
|
6
|
Unanticipated complications of management
|
0
|
2 (1.8)
|
n/a
|
|
7
|
Non-obstetric complications
|
18 (15.9)
|
20 (17.7)
|
0.94
(0.75-1.00)
|
|
8
|
Unknown/undetermined causes of death
|
10 (8.8)
|
6 (5.3)
|
0.73
(0.55-0.91)
|
|
9
|
Coincidental causes
|
0
|
0
|
n/a
|
|
n/a, not accounted; a, Inter-raters’ Fleiss kappa for individual categories; b, Inter-raters’ Fleiss kappa for overall; CI, confidence interval; ICD-MM, International Classification of Diseases-Maternal Mortality; MDR, maternal death review.
Table 2. Level of agreement of preventability determined by district and province MDR committees
|
District team
n(%)
|
Province team
n(%)
|
Kappa (95%CI)b
|
Preventable
|
86 (76.1)
|
97 (85.8)
|
0.68 (0.50-0.87)
|
Not preventable
|
27 (23.9)
|
16 (14.2)
|
|
b, Inter-raters’ Fleiss kappa for overall; CI, confidence interval; MDR, maternal death review.
The district committee identified 86 of the 113 cases (76.1%) were preventable, while the province MDR committees judged that 97 (85.8%) were preventable (Table 2). Overall, there was substantial agreement between preventable events assigned by the district and province MDR committees (k=0.68, 95% CI: 0.50-0.87; p<0.001).
Furthermore, to determine the underlying preventable factors, the 3-delays model was applied to identify the delays in each pregnancy period (Table 3). The province reviewers identified more delays in all three categories than the district reviewers. The highest agreement in identification of delays in facilities was found in the third phase delay of the postpartum period (κ=0.61, 95%CI: 0.46-0.75; p<0.001). The ante-intrapartum period had the lowest agreement (κ=0.21, 95%CI: 0.05-0.36; p<0.001).
Table 3. Agreement level of identification of the three delays in each pregnancy period between district and province MDR committees
|
District team
n (%)
|
Province team
n (%)
|
Kappa (95% CI)
|
Antenatal
|
57 (50.4)
|
80 (70.8)
|
0.45 (0.33-0.57)b
|
Phase one delay
|
27 (23.9)
|
26 (23.0)
|
0.63 (0.46-0.80)a
|
Phase two delay
|
0
|
0
|
n/a
|
Phase three delay
|
40 (35.4)
|
78 (69.0)
|
0.39 (0.27-0.52)a
|
Early pregnancy
|
4 (3.5)
|
8 (7.1)
|
0.48 (0.33-0.62)b
|
Phase one delay
|
3 (2.7)
|
1 (0.9)
|
0.49 (0.11-1.00)a
|
Phase two delay
|
1 (0.9)
|
0
|
n/a
|
Phase three delay
|
1 (0.9)
|
7 (6.2)
|
0.24 (0.15-0.66)a
|
Ante-intrapartum
|
47 (41.6)
|
79 (69.9)
|
0.21 (0.05-0.36)b
|
Phase one delay
|
12 (10.6)
|
5 (4.4)
|
0.31 (0.01-0.61)a
|
Phase two delay
|
0
|
0
|
n/a
|
Phase three delay
|
41 (36.3)
|
77 (68.1)
|
0.29 (0.16-0.43)a
|
Postpartum
|
54 (47.8)
|
58 (51.3)
|
0.58 (0.42-0.74)b
|
Phase one delay
|
10 (8.8)
|
3 (2.7)
|
0.28 (0.04-0.60)a
|
Phase two delay
|
0
|
0
|
n/a
|
Phase three delay
|
48 (42.5)
|
54 (47.8)
|
0.61 (0.46-0.75)a
|
n/a, not accounted; a, Inter-raters’ Fleiss kappa for individual categories; b, Inter-raters’ Fleiss kappa for overall. CI, confidence interval; MDR, maternal death review.
Figure 1 presents the excerpts of the recommendations for the identified underlying factors related to the delays. Due to the diversity of the recommendations, we did not calculate the Fleiss k statistics for the level of interrater agreement related to these responses.
[Insert Figure 1 here]
Figure 1. Examples of the chosen reviewer's recommendations
Underlying cause of maternal death
Category
|
3 Obstetric hemorrhage
O72.1 Other immediate postpartum hemorrhage
|
Contributing condition
|
O66 Other obstructed labor
E66 Obesity
|
Preventability
|
Preventable
Fail to meet standard of care
|
Component
|
Delays
|
Target
|
Recommendation
|
Person in Charge
|
Scheduled time
|
Antenatal
|
Third delay
Early detection:
Health professional: misdiagnosis, mismanagement of follow-up care
|
Community health center
|
Quality improvement of integrated ANC with supervision and periodic training facilitated by the district MCH team
|
Head of community health center, MCH coordinator
|
Three months
|
Ensuring completeness of data MCH handbook, including obstetricians' and gynecologists' examination data
|
POGI, IDI
|
Three months
|
Professional organization
|
Heath professional organization’s feedback to their members
|
POGI
|
One month
|
Quality improvement of clinical management based on evidence-based, competency-based training and supportive supervision
|
Continuing medical education based on MDR recommendation of the current preventive measures at the facility and community level
|
Intrapartum
|
Third delay
Standard management:
Labor monitoring,
Abnormal labor identification,
Diagnosis
Emergency condition:
Early detection, problem identification/diagnosis, response time
Policy and Guideline: compliance with clinical practice guideline
|
Professional organization
|
Heath professional organization’s feedback to their members
|
POGI
|
One month
|
Quality improvement of clinical management based on evidence-based, competency-based training and supportive supervision
|
POGI
|
Three months
|
Continuing medical education based on MDR recommendation of the current preventive measures at the facility and community level
|
Postpartum
|
Third delay
Monitoring at health facility:
Health professional:
Emergency condition:
early detection, stabilization, obstetrics’ procedure
Policy and Guideline: compliance with clinical practice guideline
|
Hospital
|
Quality improvement in the management of obstetric emergencies through monthly EMOC team meetings
Accelerating interval between admission to treatment, with targeting that all patients treated in less than 30 minutes when arriving at the hospital
|
Medical director
|
One month
|
ANC, antenatal care; EMOC, emergency obstetric care; IDI, Ikatan Dokter Indonesia (Indonesian Medical Association); MCH, mother/child health; MDR, maternal death review; POGI, Perkumpulan Obstetri dan Ginekologi (Indonesian Society of Obstetrics & Gynecology).
Acceptability
[Insert Table 4 here]
Table 4. Assessments on seven dimensions of acceptability
Key Themes
|
Affective Attitude
|
Burden
|
Ethicality
|
Intervention coherence
|
Opportunity Costs
|
Perceived Effectiveness
|
Self-efficacy
|
Representing each TFA construct
|
How an individual feels about the intervention
|
The perceived amount of effort that is required to participate in the intervention
|
The extent to which the intervention has a good fit with an individual's value system
|
The extent to which a participant understands the intervention
|
The extent to which benefits, profits, or value must be given up to engage in the intervention
|
The extent to which the intervention is perceived as likely to achieve its goal
|
The participants’ confidence that they can perform the behavior(s) required to participate in the intervention
|
E-MPATHY tool
|
Perceived benefits in conducting a review’s decision based on scientific evidence
|
Desire to use the E-MPATHY for reviews at districts
|
Perception of behaviour changes in daily practices
|
The reviewers constantly use clinical guidelines when disagreement occurs
|
Reviewers provided time to learn updated clinical guidelines
|
The first experience to adapt reviews with the utilization of clinical guidelines for assigning the cause of death and identifying the delay
|
The reviewers were able to review more cases per day routinely
|
Unoptimized clinical management lead to fatal results
|
|
Self-reflection after being able to identify the third delay
|
Reviewer perception of an E-MPATHY as an easy tool for guiding the reviews
|
The review model encouraged the reviewers to continuously learn the new contents of the maternal and child health handbook
|
Perceived difficulties when starting to search for new guidelines for non-obstetric cases
|
|
Perceived re-learning and updating review’s knowledge
|
|
Changes the attitudes to prevent the delays in the intervention in the referral hospital.
|
Using E-MPATHY for reviews guided more appropriate alternative recommendations.
|
Reviews were more concise and accurate with guidelines
|
Using E-MPATHY in the periodic MDR, the reviewers could decide when disagreements occurred with conducting literature searches when there was an unavailable guideline
|
.
|
|
|
|
|
|
Minimal misclassification of the cause of maternal death
|
Differences in identifying the preventability when there were frequent clinical updates, task-shifting issues, and indirect delays of inadequate standard laboratory
|
|
E-MPATHY, Electronic-Implementation of Maternal Death/Mortality Review; MDR, maternal death review; TFA, Theoretical Framework of Acceptability.
Table 4 displays the key themes and sub-themes generated from the TFA analysis reflecting participants’ concurrent acceptability of E-MPATHY for decision support system in MDR. All the reviewers perceived a positive learning about the advantages of E-MPATHY. They highlighted the familiarity of doing reviews using clinical standards and evidence-based guidelines to reveal maternal care deficiencies and proposed suitable recommendations. Furthermore, in the construct of opportunity costs, the reviewers expressed their perceived benefits of the E-MPATHY, such as shorter review time, accuracy, lower risk of misclassification, and opportunity of updating their knowledge to the latest evidence or guideline. Interestingly, in term of the ethicality construct, some reviewers reported that their experiences in using this tool promoted self-reflection to improve their awareness of the possible outcome of delays in the clinical practice setting. One reviewer shared the following:
“[my experience with E-MPATHY] reviewing cases makes me aware of my own practices. I intended to delay cesarean section for my patient with severe preeclampsia until the folowing morning [but] I decided to do the emergency cesarean section considering the high risk of death in such a case.” (A district reviewer, 42 years old)
However, frequent updates to the clinical procedures especially during the COVID-19 pandemic and inadequate HIV screening, indirect delay of syphilis and hepatitis testing (triple elimination) and lack of follow-up examinations in antenatal care during the time which this study was conducted interfered with the perceived effectiveness in reviewing the preventability of fatal events. The reviewers highlighted the discrepancy in their review findings of cases of patients with COVID-19. The district team members considered that the review based on the updated clinical procedures to be unoptimized clinical management. In contrast, the members of the province team preferred to use the updated procedures to identify the deficiencies for improvement in maternal care. Concerning the inadequate triple elimination, the district team did not perceive the cases with inadequate testing as contributing factors to maternal death. On the other hand, the province team associated this inadequacy as one of the deficiencies in maternal care.
Besides the two factors related to clinical procedures, task-shifting to midwives of selected interventions also interfered with the perceived effectiveness. This present study found that the lowest agreement was found in the intrapartum period related to professional birth attendants. While the province reviewers perceived the task shifting as a delay, the district reviewers considered it as acceptable considering the limited number of attending OB/GYN professionals.