Table 1: Mean indicators’ scores (range 1-3) for each attribute used for the evaluation of the influenza sentinel surveillance system in Zambia, 2011-2017a.
Attributes
|
Number of evaluated indicators
|
Mean score
|
Performance
|
· Data quality and completeness
|
7
|
2.9
|
Moderate to good
|
· Timeliness
|
2
|
2.5
|
Moderate
|
· Representativeness
|
2
|
2.0
|
Moderate to weak
|
· Flexibility
|
2
|
3.0
|
Good
|
· Simplicity
|
7
|
2.8
|
Moderate to good
|
· Acceptability
|
4
|
3.0
|
Good
|
· Stability
|
8
|
2.6
|
Moderate to good
|
· Utility
|
4
|
2.7
|
Moderate to good
|
· Sustainability
|
2
|
1.0
|
Weak
|
· Overall
|
38
|
2.6
|
Moderate to good
|
a Each quantitative indicator was evaluated as the proportion (expressed as percentage) of the outcome of interest over the total. A scale from 1 to 3 was used to provide a score for each quantitative indicator as follows: <60% (from the above calculation) scored 1 (weak performance); 60-79% scored 2 (moderate performance); ≥80% scored 3 (good performance). For indicators for which a proportion over a total could not be obtained (qualitative indicators) a score was assigned based on the same scale using expert consensus. The scores assigned to each indicator were averaged for all indicators evaluated for each attribute to provide a mean score for each surveillance attribute. An overall score for the surveillance system was obtained by averaging the scores of all evaluated indicators.
Table 2: Indicators for data quality and completeness, timeliness, representativeness and flexibility used for the evaluation of the influenza sentinel surveillance system in Zambia, 2011-2017a.
Indicator
|
Calculation/data inputs
|
Data source
|
Indicator value
|
Score
|
Data quality and completeness
|
· Proportion of enrolled patients with ILI against set target (5 patients per week per site).
|
Number of enrolled patients with ILI / Estimated target.
|
ISS database and estimated target
|
140.4%
|
3
|
· Proportion of enrolled patients with SARI against set target (all eligible patients).
|
Number of enrolled patients with SARI / All patients with SARI.
|
ISS database
|
69.8%
|
2
|
· Proportion of SARI/ILI cases that meet the case definition
|
Number of ILI/SARI cases that meet the case definition / Total number of enrolled ILI/SARI cases
|
ISS database
|
86.5%
|
3
|
· Proportion of forms without at least one inconsistent or missing value for key variablesb
|
Number of forms without at least one incorrect or missing value / Total number of forms
|
ISS database
|
93.8%
|
3
|
· Proportion of sampled ILI/SARI cases with available laboratory results
|
Number of ILI/SARI cases with available laboratory results / Number of sampled ILI/SARI cases
|
ISS database
|
93.2%
|
3
|
· Proportion of sample with positive RNP results
|
Number of samples with a positive RNP result / Total number of samples tested
|
ISS database
|
99.8%
|
3
|
· Proportion of collected variables included in the WHO recommended minimum data collection standard
|
Number of collected variables within the WHO list / Number of WHO recommended variables.
|
CIF and WHO guidelines for influenza sentinel surveillance.
|
82.3%
|
3
|
Timeliness
|
· Proportion of samples received within the target period from collectionc
|
Number of samples received at the laboratory within 14 days from collection / Number of samples received
|
ISS database
|
73.2%
|
2
|
· Proportion of samples tested within one week from receipt
|
Number of samples tested within two weeks from receipt / Number of samples tested
|
ISS database
|
87.3%
|
3
|
Representativeness
|
· Geographical coverage
|
Number of provinces covered by the influenza sentinel surveillance network / Total number of provinces of the country
|
Geographic distribution of sentinel sites.
|
20.0%
|
1
|
· Inclusion of all age groups
|
Age distribution of ILI/SARI cases (median, minimum and maximum)
|
ISS database
|
Med. : 4 Y
Min. : 0 Y
Max. : 97 Y
|
3
|
Flexibility
|
· Expansion of sentinel sites participating to the ISSS since inception
|
Number of new sentinel sites since inception
|
Protocol
|
5
|
3
|
· Surveillance for pathogens other than influenza
|
Number of investigated pathogens other than influenza
|
Protocol
|
10
|
3
|
Abbreviations: ILI: influenza-like-illness; SARI: severe acute respiratory illness; CIF: case investigation form; RNP: RiboNucleic Protein; WHO: World Health Organization; ISS: influenza sentinel surveillance.
a Each quantitative indicator was evaluated as the proportion (expressed as percentage) of the outcome of interest over the total (indicator value). A scale from 1 to 3 was used to provide a score for each quantitative indicator as follows: <60% (from the above calculation) scored 1 (weak performance); 60-79% scored 2 (moderate performance); ≥80% scored 3 (good performance).
b Key variables evaluated for completeness and consistency of data: site, age/date of birth, sex, date of consultation/admission, date of symptoms onset, date of sample collection and signs and symptoms included in the case definitions.
c The target period was 48 hours from sites situated in Lusaka Province and 14 days from sites situated in Copperbelt Province.
Table 3: Indicators for simplicity and acceptability used for the evaluation of the influenza sentinel surveillance system in Zambia, 2011-2017a.
Indicator
|
Calculation/data inputs
|
Data source
|
Indicator value
|
Score
|
Simplicity
|
· Perception of surveillance staff on identification of cases
|
Number of surveillance staff within each reported category [very difficult (VD), difficult (D), easy (E), very easy (VE)] / Number of surveillance staff interviewed
|
Questionnaire survey among surveillance staff at sentinel sites
|
VD : 9.3%
D: 7.0%
E: 27.9%
VE: 55.8%
|
3
|
· Perception of surveillance staff on obtaining consent
|
Number of surveillance staff within each reported category [very difficult (VD), difficult (D), easy (E), very easy (VE)] / Number of surveillance staff interviewed
|
Questionnaire survey among surveillance staff at sentinel sites
|
VD: 4.6%
D: 13.3%
E: 35.6%
VE: 46.5%
|
3
|
· Perception of surveillance staff on completing the CIF
|
Number of surveillance staff within each reported category [very difficult (VD), difficult (D), easy (E), very easy (VE)] / Number of surveillance staff interviewed
|
Questionnaire survey among surveillance staff at sentinel sites
|
VD: 0.0%
D: 0.0%
E: 27.9%
VD: 72.1%
|
3
|
· Perception of surveillance staff on sample collection
|
Number of surveillance staff within each reported category [very difficult (VD), difficult (D), easy (E), very easy (VE)] / Number of surveillance staff interviewed
|
Questionnaire survey among surveillance staff at sentinel sites
|
VD: 0.0%
D: 5.7%
E: 80.0%
VE: 14.3%
|
3
|
· Perception of surveillance staff on packaging and storage of samples
|
Number of surveillance staff within each reported category [very difficult (VD), difficult (D), easy (E), very easy (VE)] / Number of surveillance staff interviewed
|
Questionnaire survey among surveillance staff at sentinel sites
|
VD: 0.0%
D: 0.0%
E: 20.9%
VE: 79.1%
|
3
|
· Perception of surveillance staff on completing the screening/enrollment logbook
|
Number of surveillance staff within each reported category [very difficult (VD), difficult (D), easy (E), very easy (VE)] / Number of surveillance staff interviewed
|
Questionnaire survey among surveillance staff at sentinel sites
|
VD: 0.0%
D: 0.0%
E: 72.4%
VE: 27.6%
|
3
|
· Time to enroll a SARI/ILI case from patient’s identification to the sample packaging
|
Number of surveillance staff within each reported category (<30 min, 30-60 min, >60 min) / Number of surveillance staff interviewed
|
Questionnaire for surveillance staff at sentinel sites
|
<30: 68.6%
30-60: 20.0%
>60: 11.4%
|
2
|
Acceptability
|
· Proportion of surveillance staff that is satisfied with the weekly bulletins
|
Number of surveillance staff within each reported category [not satisfied (NS), poorly satisfied (PS), satisfied (S), very satisfied (VS)] / Number of surveillance staff interviewed
|
Questionnaire for surveillance staff at sentinel sites
|
NS: 0.0%
PS: 0.0%
S: 25.6%
VS: 74.4%
|
3
|
· Proportion of surveillance staff that is satisfied with the feedback of laboratory results
|
Number of surveillance staff within each reported category [not satisfied (NS), poorly satisfied (PS), satisfied (S), very satisfied (VS)] / Number of surveillance staff interviewed
|
Questionnaire for surveillance staff at sentinel sites
|
NS: 0.0%
PS: 9.3%
S: 69.8%
VS: 20.9%
|
3
|
· Proportion of time allocated to influenza surveillance activities per week
|
Number of hours allocated to influenza surveillance activities per week / Number of working hour per week
|
Questionnaire for surveillance staff at sentinel sites
|
22.5%
|
3
|
· Number of ILI/SARI patients enrolled per day
|
Number of surveillance staff within each reported category [≤5 patients (≤5), 6-10 patients (6-10), >10 (>10)] / Number of surveillance staff interviewed
|
Questionnaire for surveillance staff at sentinel sites
|
≤5: 95.2%
6-10: 3.2%
>10: 1.2%
|
3
|
|
|
|
|
|
|
|
|
Abbreviations: ILI: influenza-like-illness; SARI: severe acute respiratory illness; CIF: Case Investigation Form.
a Each quantitative indicator was evaluated as the proportion (expressed as percentage) of the outcome of interest over the total (indicator value). A scale from 1 to 3 was used to provide a score for each quantitative indicator as follows: <60% (from the above calculation) scored 1 (weak performance); 60-79% scored 2 (moderate performance); ≥80% scored 3 (good performance).
Table 4: Indicators for stability used for the evaluation of the influenza sentinel surveillance system in Zambia, 2011-2017a.
Indicator
|
Calculation/data inputs
|
Data source
|
Indicator value
|
Score
|
Stability
|
· Frequency of lack of data collection forms
|
Number of surveillance sites within each reported category [never (0), once per year (1), 2-3 times per year (2-3), ≥4 times per year(≥4)] / Number of surveillance sites
|
Questionnaire for surveillance staff at sentinel sites
|
0: 83.3%
1: 16.7%
2-3: 0.0%
≥4: 0.0%
|
3
|
· Frequency of lack of sampling material
|
Number of surveillance sites within each reported category [never (0), once per year (1), 2-3 times per year (2-3), ≥4 times per year(≥4)] / Number of surveillance sites
|
Questionnaire for surveillance staff at sentinel sites
|
0: 83.3%
1: 16.7%
2-3: 0.0%
≥4: 0.0%
|
3
|
· Frequency at which a power failure, including the generator, occurred at the surveillance sites
|
Number of surveillance sites within each reported category [never (N), seldom (S), often (O), regularly (R)] / Number of surveillance sites
|
Questionnaire for surveillance staff at sentinel sites
|
N: 16.7%
S: 50.0%
O: 33.3%
R: 0.0%
|
2
|
· Proportion of sentinel sites with at least one member of staff trained on sentinel surveillance procedures
|
Number of sentinel sites with at least one trained member of staff / Number of surveillance sites
|
Questionnaire for surveillance staff at sentinel sites
|
100.0%
|
3
|
· Proportion of sentinel surveillance staff ever trained on sentinel surveillance procedures
|
Number of surveillance staff ever trained / Number of surveillance staff
|
Questionnaire for surveillance staff at sentinel sites
|
90.7%
|
3
|
· Proportion of sentinel surveillance staff trained on sentinel surveillance procedures during the last one year
|
Number of surveillance staff during the last one year / Number of surveillance staff
|
Questionnaire for surveillance staff at sentinel sites
|
69.7%
|
2
|
· Availability and use of standard operating procedures (SOPs) by surveillance staff
|
Number of surveillance staff with access and use of SOPs / Number of surveillance staff
|
Questionnaire for surveillance staff at sentinel sites
|
97.7%
|
3
|
· Proportion of surveillance sites providing samples weekly after 3 months from inception
|
Number of surveillance sites providing samples weekly/ Number of surveillance sites
|
ISS database
|
100.0%
|
3
|
· Proportion of weekly surveillance reports sent to MoH
|
Number of surveillance reports sent to MoH / Number of reporting weeks
|
Surveillance reports
|
71.4%
|
2
|
Abbreviations: ISS: influenza sentinel surveillance; MoH: Ministry of Health.
a Each quantitative indicator was evaluated as the proportion (expressed as percentage) of the outcome of interest over the total (indicator value). A scale from 1 to 3 was used to provide a score for each quantitative indicator as follows: <60% (from the above calculation) scored 1 (weak performance); 60-79% scored 2 (moderate performance); ≥80% scored 3 (good performance).
Table 5: Indicators for utility and sustainability used for the evaluation of the influenza sentinel surveillance system in Zambia, 2011-2017a.
Indicator
|
Calculation/data inputs
|
Data source
|
Indicator value
|
Score
|
Utility
|
International
· Proportion of weeks with data reported to WHO FluNet
|
Number of weeks with data reported to WHO FluNet / Number of weeks during the evaluated period
|
Weekly FluNet submissions
|
98.4%
|
3
|
· Mean annual number of samples shared with WHO Collaborating Centers (WHO CC)
|
Number of samples/isolates shared with WHO CC / Number of years with samples shipped.
|
Shipment logs to WHO CC London
|
23 (range 14-33) shipped during 2012 and 2014-2017
|
3
|
· Number of contributions to influenza Regional/Global studies
|
Number of publications on Regional/Global studies with influenza data from Zambia
|
PubMed
|
2 [8,25]
|
2
|
Domestic
· Ability to assess important influenza epidemiological features/public health outcomes
|
· Temporal patters of influenza virus circulation (Yes) [3]
· Circulating influenza types/subtypes, including pandemic strains (Yes) [3]
· Proportion of ILI/SARI illness attributable to influenza virus infection (Yes) [3]
· Risk factors for influenza-associated severe illness (No)
· Burden of influenza-associated illness (Yes) [14]
|
Publications and reports
|
80.0%b
|
3
|
Sustainability
|
· Proportion of the ISSS cost covered by the Zambia-MoH
|
Cost covered by the Zambia-MoH / Total cost
|
Budget report
|
16.9%
|
1
|
· Availability and implementation of a sustainability plan
|
· Drafted (Yes)
· Finalized (No)
· Approved (No)
· Implemented (No)
|
Sustainability plan
|
25.0%b
|
1
|
Abbreviations: MoH: Ministry of Health.
a Each quantitative indicator was evaluated as the proportion (expressed as percentage) of the outcome of interest over the total. A scale from 1 to 3 was used to provide a score for each quantitative indicator as follows: <60% (from the above calculation) scored 1 (weak performance); 60-79% scored 2 (moderate performance); ≥80% scored 3 (good performance). For indicators for which a proportion over a total could not be obtained (qualitative indicators) a score was assigned based on the same scale using expert consensus.
b Indicator value calculated by dividing the number of achieved outcome by the total number of outcome considered (i.e. 4/5=80.0% or 1/4=25.0%).