Vector control interventions play a fundamental role in the control and elimination of vector-borne diseases. The evaluation of vector control products relies on bioassays, laboratory and semi-field tests that use live insects, to assess the product’s effectiveness. Bioassay method development requires a rigorous validation process to ensure that relevant methods are used that capture appropriate entomological endpoints which accurately and precisely describe likely efficacy against disease vectors as well as product characteristics within the manufacturing tolerance ranges for insecticide content specified by the World Health Organisation. Currently, there are no standardised guidelines for bioassay method validation in vector control. This report presents a framework for bioassay validation that draws on accepted validation processes from the chemical and healthcare fields and which can be applied for evaluating bioassays and semi-field tests in vector control. The validation process has been categorised into four stages: preliminary development; feasibility experiments; internal validation, and external validation. A properly validated method combined with an appropriate experimental design and data analyses that account for both the variability of the method and the product is needed to generate reliable estimates of product efficacy to ensure that at-risk communities have timely access to safe and reliable vector control products.
Research Article
A bioassay method validation framework for laboratory and semi-field tests used to evaluate vector control tools
https://doi.org/10.21203/rs.3.rs-2706711/v1
This work is licensed under a CC BY 4.0 License
Journal Publication
published 28 Sep, 2023
You are reading this latest preprint version
Vector
bioassay
insecticide
evaluation
validation
verification
precision
variability
feasibility
insects
ITN
control
Vector control interventions play a fundamental role in the control and elimination of vector-borne diseases, due to their effectiveness in preventing infection and reducing disease transmission [1–3]. Products for use in vector control interventions undergo a standardised process through WHO pre-qualification to ensure each product meets specified standards [4–7]. For many years vector control relied upon insecticide classes (pyrethroids, organochlorines and carbamates) that induce rapid neural intoxication and death [8]. Therefore, during the laboratory phase of this evaluation, product characteristics are assessed using bioassays that typically measure the knockdown and mortality of exposed insects, usually mosquitoes. There has been observed high heterogeneity in measured outcomes and poor agreement between methods [9–12]. The observed variability in bioassays is exacerbated by the high tolerance ranges for the insecticide content of insecticide treated nets (ITNs), because ITNs are non-homogeneous products that can vary within and between each panel used for net construction [7,13].
New classes of insecticide are now available for use on ITNs, and for use in indoor residual spraying (IRS). Therefore, new bioassay methods are required that accurately describe the performance characteristics of products considering the mode of action of the active ingredient (AI) and which are sufficiently sensitive to differentiate product performance against vector populations with different physiologies [14]. A method refers here to a new or modified bioassay, a technique using commercially available or in-house reagents, a prototype kit or a semi-field test. Bioassay method development requires a rigorous validation process to ensure that the methods are able to use entomological endpoints to accurately and precisely describe product characteristics within the manufacturing tolerance range currently specified by WHO for that product [13]. This requires validation of the method’s quality, reliability, and consistency of results prior to implementation in multiple facilities [15].
Method validation is a process that is used to demonstrate that a procedure is suitable for its intended purpose and that the results obtained are reliable [16,17]Experimental design for method validation is centred on defining and measuring analytical error[18,19]. These analytical errors are broadly categorised into two: 1) random errors or imprecision, which refers to the distribution of test values, and 2) systematic errors or inaccuracy, which is a shift or bias between the central measure value and the correct value [19]. The total error is a combination of 1) + 2) and plays a vital role in assessing the test’s performance based on the allowable measurement error defined during the preliminary development and feasibility stages.
Although standardised method validation guidelines for assessing novel methods exist within the healthcare and pharmaceutical industries (among others) [20–22], standardised validation practices are not typically used in the vector control field. This is partly because the widely used validation study design methodologies in use in other fields are not necessarily directly transferable to bioassays using live biological material or to tests performed in variable field conditions, resulting in a lack of relevant method validation guidelines [22,23].
This report describes a framework for bioassay method validation and presents case studies for its use in bioassay development for vector control products. The framework draws on formalised validation processes used in other scientific disciplines and adapts them into a modular approach that recognises the unique nature of bioassays that use entomological endpoints to characterise the insecticidal properties of vector control tools. The framework is suitable for use to validate methods that might be used in product testing, non-bioassay methods that might be used for routine quality surveillance, and for other bioassays with an entomological endpoint, such as resistance monitoring or methods used to test non-product-based vector control. It may also be used to think through issues that are causing variability in bioassays within a product testing facility. Within each process stage, validating laboratories can select the most relevant sub-studies to conduct based on the intended purpose of the method under development or in use and apply the framework to evaluate method precision, accuracy and replicability of both laboratory and semi-field tests.
Method validation and method verification are two distinct processes that may be required at different points in the development of a method from initial design to implementation in multiple laboratories and/or semi-field sites. A laboratory is required to carry out validation when:
- A laboratory has designed or developed a new method
- A laboratory is required to demonstrate comparability between a novel method and an existing standard method.
- A standard method has been modified
- A standard method is used for a new purpose
Full laboratory design and development involve the conception of the method from scratch, including preliminary testing to define whether the method is logistically feasible and can measure the desired outputs. Where the novel method measures the same outputs as an existing method, the novel and existing methods should be compared during validation. Modification of an existing standard method alters one aspect, for example extending the exposure time used in a test, and requires the internal and external validation process stages (see below) to be repeated. To use a method for a new purpose might involve the use of a method validated for use with one product class being used to characterise a different product class. Validation in such cases is dependent on the magnitude of the change but may require conducting feasibility experiments to demonstrate that the change in scope has not affected the capacity of the method to reliably capture its endpoints.
Laboratories adopting a validated method should conduct method verification [19], which can be conducted using controls of a known value and/or response and ensures that the implementing laboratory can reproduce the established method performance.
Stages of the method validation process
Four stages for evaluating bioassays and semi-field tests are proposed: 1) preliminary development; 2) feasibility experiments; 3) internal validation, and 4) external validation (Table 1). The stages are designed to ensure that the method is scientifically sound and reproducible within the variation exhibited in biological tests [5,19]. During preliminary development, the method is devised, and endpoints and analytical requirements are defined and tested [22]. At the feasibility stage, the performance parameters and endpoints are verified, and a standard operating procedure (SOP) is drafted. In internal validation, the analytical performance of the method is tested, the method claim is drafted and a data package for external validation sites is compiled. During external validation, the method is evaluated in multiple laboratories/sites and the final method claim is produced. Once external validation is successful, the method can be implemented.
[Table 1 to appear here, table included in the Tables and Legends section]
Preliminary development
The purpose of the preliminary development stage is to assess the proposed method design for suitability for a defined purpose in a defined setting, define the endpoints and the level of allowable analytical error (both imprecision and inaccuracy) for each, and to build robustness (minimise the impact of changes in variables or testing conditions on results) into the method. Experiments conducted as part of preliminary development typically use small sample sizes so that data on a range of conditions and variables (for examples, refer to the testing conditions below) can be generated and used to refine the method parameters and guide the experimental design for feasibility and internal validation.
Define method scope and endpoints
The method design, application, and endpoints to be used to assess method performance should be clearly defined. An endpoint is a quantifiable output that can be recorded using the method, e.g., oviposition inhibition in female mosquitoes exposed to an insect growth regulator. Every endpoint that is intended to become part of the eventual method claim must be defined. The definition should state precisely what is to be measured, when, and the desired range of measurement, e.g., number of eggs laid per female, up to five days post-exposure, from 0-300 eggs’.
Define acceptability criteria
Acceptability criteria define the allowable error within the method and are dependent on the effect size of each endpoint. In the example from the previous section, for the measurement outcome: ‘50% reduction in the number of eggs laid per female, up to five days post-exposure, from 0-300 eggs’ an acceptability criterion might be: ‘Measure a 50% reduction in the number of eggs laid with 10% precision within the reportable range’. The allowable error should be as small as possible yet align with what is practically achievable and scientifically justifiable [24].
Factors to consider when defining acceptability criteria:
- Within-day imprecision should be less than ¼ of the total allowable error or the coefficient of variation (CV) <20% [19,25,26]. Between-day imprecision typically has the same error level, but can be increased if justified [22]
- For measurement outcomes relating to target values, criteria can be set either as a multiple of standard deviation (SD), e.g., within 3SD of the mean, or within a percentage range of the target value e.g. +/- 25%
- For phenotypic measurement outcomes, an indicative threshold can be used although such thresholds should be used with care [27]. For example, 98% mortality in a susceptibility test using a discriminating concentration in monitoring for insecticide resistance.
Identify the analytical parameter/s to be measured
At least one analytical parameter must be evaluated [17,20]. Within the common analytical parameters of accuracy/trueness, precision, linearity, range and robustness, the most useful parameters for bioassay and semi-field validations are typically precision, robustness, linearity (concentration dependence), and range (reliable range of test values) [17,20,21,28–30].
Define testing conditions
Test conditions encompass conditions critical for method performance. These can be identified from literature or in-house laboratory data [17]. Bioassay testing conditions include:
- Vector age
- Vector status (sex, fed/unfed)
- Preparation conditions for the vector, e.g., sugar starvation
- Vector holding conditions post-exposure
- Time of day
- Environmental conditions
- Maximum/minimum number of vectors per replicate assay
- Sample handling conditions
- Storage pre- and post-test
- Time to reach ambient temperature prior to testing
- Sample preparation (age, washing)
To determine the optimum conditions for the method, experiments varying the testing conditions should be conducted, for example, changing the time of day that the bioassay is conducted to determine whether a mosquito’s circadian rhythm affects the results of the test [31]. A method is deemed robust if small variations in testing conditions do not heavily impact the performance of the method [22] for the purpose selected e.g., evaluation of pyrethroid content on an ITN. Testing conditions may have a small or large impact on the assay results depending on the specific mode of action of the chemistry being bioassayed.
Select a comparison method (where applicable)
If the novel method has been designed to measure the same outcomes as an established standard method the new method should be compared to this. Standardised methods currently recommended in WHO guidelines are the WHO cylinder test and bottle bioassay to measure insecticide susceptibility, the WHO cone test and the tunnel test to characterise ITN fabrics, and Ifakara Ambient Chamber Test and experimental hut trials to measure entomological efficacy of ITNs [4,27]. Select the method with the most similar test conditions and/or entomological endpoint(s) to the putative new method as the comparator method.
Define controls and control ranges
Negative (baseline) and positive controls must be defined. Since it is not always known what non-insecticidal features of a product may impact the measured endpoint, the negative control should be as close as possible to the product under evaluation. The positive control should induce a known and significant impact on the endpoint under evaluation. Methods designed for characterisation of dual AI products must include controls which contain each AI separately and in combination [32–34].
Conduct baseline and robustness experiments
Baseline experiments assess the performance of a method under assumed optimum testing conditions. Robustness experiments test the robustness of a method by identifying variables or testing conditions that might affect the method’s results [17,35].
For baseline experiments:
- Consider the testing conditions that can potentially affect results and define standard measurement levels, for example a specified temperature range, to control for such effects
- Conduct trial experiments using the simplest design possible, for example, tests using negative controls such as an untreated net for an ITN method
For robustness testing:
- Alter testing conditions or variables one at a time whilst keeping all other parameters unchanged. Although it is possible to vary multiple conditions simultaneously [17,36], due to the high variability in bioassays, one variable at a time is recommended
- Evaluate the degree of robustness: significance testing, a procedure used to quantify whether a result is likely due to chance or to some factor of interest, can be employed to determine the important factors for future consideration for assessing the method’s performance
Sample size:
A sample size of at least twenty replicates per group should be used for baseline or robustness experiments [37].
When designing the experiments, apply the following definitions:
- Replicate: for example, a single set of five individual mosquitoes in a WHO cone test or mosquitoes exposed together in a Tunnel Test
- Sample: for example, a single piece of a net
- Testing system: for example, the mosquitoes being tested. Mosquitoes reared together under the same controlled conditions are referred to as the same testing system. This can be a single colony at a point in time or one colony maintained over time that is characterised and maintains fitness parameters within defined limits.
The data from the baseline or robustness experiments should be analysed and compared to the acceptability criteria. Where necessary the method can be modified, the outcome(s) and acceptability criteria refined and retested before proceeding to the feasibility stage. Figure 1 outlines a decision tree that can be used at each process evaluation stage to determine whether progression to the next stage is appropriate.
[Figure 1 to appear here, figure included in the Figures and Legends section after references]
Text box 1. Terminology
Feasibility experiments
Feasibility experiments are employed to understand the inherent variability of a method, to obtain values that can be used for estimating sample size for the internal validation experiments and to assess the utility and logistical ease of the proposed technique. Where two tests have equivalent performance characteristics, the one which is easier to use, cheaper, faster, more sensitive or more accurate might be preferred.
Estimating an appropriate sample size
Testing 20-30 replicates in a feasibility study is usually enough to obtain an estimate for variability/precision for use in formal sample size calculations [37,38]. Ideally, 20-30 replicates in each study arm (WHO cones or cylinders, for example) would be tested on a single day to estimate within-day precision, followed by testing at least one replicate per day over a period of 20 days whilst holding all conditions constant to estimate between-day precision [19]. However, for bioassays that use long exposure times, such as the tunnel test, this study design is not possible. The sample size should be adjusted appropriately to suit the design of the method, performing at least four replicates per day. Additionally, the use of insects as the test system in bioassays means that it is not possible to hold the test system constant, i.e., use the same mosquitoes each day. Rigorous colony rearing procedures should be followed to ensure colony stability to minimise insect variability, and data on fitness parameters should be collected for consideration as a potential source of variability [34]. To account for this variability, we recommend that at least four replicates should be tested for a minimum of five days wherever possible and any analysis should include day of testing as a variable to account for the temporal bias inherent in bioassays using live insects.
Describe testing pattern and testing period
The testing schema and testing period in the experimental design of feasibility experiments defines how the within and between-day error of the method will be measured, and are typically determined during replication sub-studies (refer to Replication Studies). The testing pattern should be balanced with respect to the number of replicates tested in a single day and the number of replicates tested each day over multiple days so that reliable estimates for the within and between-day precision are obtained.
Defining final endpoints for validation and drafting an SOP
During preliminary development and feasibility stages, multiple endpoints might be trialled. The data from feasibility studies is used to identify which of those endpoints are reliable and suitable for use in assessing the method’s performance during internal and external validation. All selected endpoints and their acceptability criteria should be included in a draft SOP.
Select strains for use in validation experiments
Both pyrethroid-susceptible and pyrethroid-resistant mosquito strains can be used in validation experiments. Where relevant, strains should be selected with reference to existing WHO testing guidelines [27], WHO implementation guidance, and published works. For example, Lees et al [34] provides a strain characterisation SOP which can be used for dual-AI ITNs and adapted as appropriate for other studies that require resistant mosquito strains.
Internal validation
The purpose of the internal validation phase is to ensure that the method is reproducible within a laboratory, i.e., minimally validated, and to compile a data package that can be used by external laboratories/sites to externally validate the method.
Determining appropriate sample size and study design
Data from feasibility studies are used in a formal power calculation to determine the sample size for internal validation. This can be achieved by using standard formulas for sample size estimation or simulation studies for complex designs involving multiple varying factors and testing schema [39–41]. The predefined effect size for the primary endpoint of interest together with the SD/variability estimated from the feasibility experiments should be used to estimate the sample size. In a case where multiple endpoints are of primary interest, we recommend that the endpoint with the smallest effect size and greatest variability in the feasibility experiments is used in the calculation [42].
Draft the method claim
This is a statement that clearly states the scope of the method, the outcomes, analytical parameters, and acceptability criteria associated with the method. Considerations to be taken into account when employing the method, for example, incorporating the variability of sample materials into sample size calculations, should be stated as part of the claim. For example, Video Cone Test (VCT) PLUS, an extension of the standard WHO cone test designed to characterise the effects of the co-formulations of pyrethroids and non-pyrethroid insecticides based on mosquitoes’ activity in the cone (imprecision/CV < 30%) and 24h mortality within ±3% the standard WHO Cone Assay. A detailed example of a method claim can be found on the Innovation to Impact (I2I) website [43].
Compile a data package
A data package must be produced by the laboratory that developed the method and provided to the external validating laboratories. The data package must include:
- SOP
- Method claim
- Applicability
- Measurement outcomes
- Analytical parameters
- Acceptability criteria and justification
- Study designs including sample sizes and testing schema
- Define controls/criteria for selecting controls
- Criteria for strain selection
The method-developing laboratory should ensure that the product(s) and strain(s) used during the internal validation phase are characterised, and the results are provided together with the data package to assist with the interpretation of the validation results.
External Validation
As methods that use entomological endpoints to evaluate vector control tools are usually implemented in multiple laboratories and/or sites, to ensure reproducibility at least two external laboratories should validate the method [22,44]. These laboratories extensively validate the method by ensuring that the method claim is reproducible at multiple sites/laboratories using a standardised SOP. This allows different levels of precision to be assessed, for example, within-day, within-laboratory, between-day and between-laboratory. The external validation sites should follow the experimental design proposed associated with the method claim that was defined following the internal validation stage. All the outcomes and analytical parameters associated with the method claim should be assessed.
A statement of the final claim and a full validation report is produced once external validation is complete. Ongoing quality assurance procedures or method verification in implementing sites certify results produced using the method.
Validation sub-studies
In each of the process stages of feasibility, internal validation and external validation, different relevant sub-studies are conducted depending on the intended purpose of the method and the design of the bioassay. Figure 2 shows each of the process stages and the possible sub-studies that might be employed.
When designing sub-studies, a single experiment can be designed for multiple purposes or to assess multiple analytical parameters. For example, an experiment designed to measure precision, i.e., a replication experiment, can include a comparator method, i.e., comparison experiments. Table 2 provides a summary of typical categories of the methods used to assess vector control tools and their associated studies and performance parameters.
[Figure 2 to appear here, figure included in the Figures and Legends section after references]
Table 2. Examples of experimental types which could be applied as validation sub-studies for methods used to evaluate vector control products.
Linearity or reportable range experiments
The reportable range of a method is the span of test values for which reliable results can be obtained; linearity is the ability of a method to obtain results which are directly proportional to a given concentration [19,44]. These studies can be implemented at all the stages of the validation process including baseline experiments. The purpose of these experiments is to determine a working range of the method’s results that is accurate and precise. For example, a reportable range for a method to measure the characteristics of an ITN might be the minimum to maximum level of 24-hour mortality which can be reliably measured by the method and the variability within the range.
For methods with phenotypic outcomes, establishing LD50 and/or LD90 for each active ingredient can be a substitute. Methods intended to be used for durability monitoring of products should be assessed using, e.g., ITNs that have undergone various numbers of washes, with accompanying chemical analysis of treatment concentration. This will approximate testing at different concentrations and ensure that method performance is validated against a range of different product conditions.
At least five replicates of known values at each concentration/number of washes (where appropriate) should be analysed by bioassay and chemical methods in triplicate to define the reportable range.
Data analysis
Linearity of the reportable range can be visually inspected using a scatter plot and line of best fit that fits the lowest points in the data series or fitting a regression line through the points in the linear range [19,21,45]. To control for potential confounding factors, we recommend the latter. For methods that are non-linear, a non-linear curve or a non-linear regression line can be fitted. For methods used to assess durability, precision should be evaluated throughout the range to determine the method’s reliability for estimating entomological outcomes over time/number of washes.
Replication experiments
Replication experiments are conducted during the feasibility, internal validation, and external validation process stages. During replication experiments, estimates are obtained for random error [19]. The goal is to determine the typical variability of the method during normal usage through measuring precision [22] and therefore the experimental design should encompass routine day-to-day variations.
Precision can be evaluated at different levels [22,44]:
- Repeatability/intra-assay/within-run: precision observed among replicate bioassays performed under the same operating conditions within a day
- Intermediate precision expresses within-laboratory variations: different days, different operators, different mosquito-rearing cages, etc.
- Reproducibility: expresses the precision of agreement between laboratories
Repeatability and intermediate precision are evaluated during the feasibility and internal validation stages while all the levels of precision should be evaluated during the external validation. Repeatability variability is usually smaller compared to the other two levels of precision due to the many sources of variation that exist within and between laboratories contributing to the inter-laboratory variation than the within-laboratory or testing day [18,22]. Therefore, careful attention should be paid when defining the acceptability criteria for the different levels of precision.
Table 3 gives a summary of stages involved when conducting replication experiments.
[Table 3 to appear here, table included in the Tables and Legends section]
Data analysis
Common measures for precision are SD or CV, also known as the relative standard deviation. However, these measures are not ideal if the data are non-normally distributed, contain a high proportion of outliers, and if the number of replicates per group are not equal [46]. In such cases, alternatives to the CV can be used, such as the Geometric Coefficient of Variation (GCV), Coefficient of Quartile Variation (CQV), Coefficient of Variation based on the Median Absolute Deviation (CVMAD), and Coefficient of Variation based on the Interquartile Range (CVIQR) for simple estimates [47–49] or the intra-class correlation coefficient (ICC) [50,51]. More details about the formulas, pros and cons of each method, and examples of R packages (where possible) are contained in Table S1 (Additional file 2).
The data analysis performed should reflect the study design that was implemented, and a data analysis plan should be produced in advance alongside the study protocol. Usually, there are different sources of variation in replication studies, and it is important to estimate precision whilst accounting for the variability of all possible factors. These factors can be fixed and/or random variables, for example, estimating the within-day variability while accounting for the testing days, operators, and site variability. The most powerful approach for estimating precision for replication studies is using mixed-effects models, and the CV and/or ICC and their associated 95% confidence intervals (CI) [18,50,51]. The incorporation of 95% CIs is critical given the many unknown factors that can influence the results of a study but cannot be controlled for in the study design [42]. These analysis methods are applicable for various types of data including continuous, proportions, binary and counts. For example, this can be implemented by using the VCA (normal data only) and rptR R-packages among other software or packages [51,52].
Comparison experiments
These experiments are conducted during the feasibility, internal validation and external validation phases and determine if there are any differences between an existing method and a new method. For example, the WHO cone test is the standard method to measure the impact of mosquito tarsal contact with an AI applied for vector control; a novel method developed to measure the impact of exposure using a different approach could be compared to the cone test to determine the comparability of the two methods. Usually, this is performed by testing the same sample by both methods [19]. However, such designs are not feasible for bioassays as the same insects/replicate samples cannot be used/measured twice using different methods/tests since preexposure will influence the outcome of a second exposure [5]. Therefore, comparison experiments for bioassays should be conducted in parallel using the same test system under the same conditions for both the pre-existing and novel methods to allow comparison.
Comparison studies for methods designed to evaluate products with new modes of action should be undertaken in parallel with a product of known performance using existing methods. Table 4 gives a summary for implementing comparison experiments.
Table 4. Comparison experiments
|
Stage |
Notes |
| 1. Select comparison method |
|
| 2. Determine the maximum number of replicates that can be performed in a single day |
Four replicates are the minimum to calculate within-run imprecision |
| 3. Perform 30 replicates each of new method and comparator method in parallel over the smallest possible number of days |
Note that for methods with extremely high variation, 30 replicates may be insufficient.
|
Data analysis
The data analysis will depend on the analytical parameter of interest, and it can be performed using the methods discussed above (as appropriate). To access the performance of the novel method, the Bland-Altman plot should be employed, to describe the agreement between the two methods based on the endpoint(s) of interest [53,54]. The results obtained from the two methods should be compared within a group (i.e., holding all other conditions/parameters constant).
Measurement uncertainty
Validation results should be reported with an uncertainty measure (e.g., 95% CI), which indicates the margin of doubt that exists for the obtained results [22,44]. For example, the CV as a measure for precision can be reported together with its corresponding 95% CI.
Outlying data points
Outlying data points/outliers are extreme values in an experimental dataset [52,55]. Outliers can negatively impact results and/or the validity of fitted models by violating the normality assumption and therefore outliers should be identified and handled appropriately [18]. All extreme data points should be double checked to remove the possibility of recording error or operator error prior to outlier analysis. Outliers can be identified using visualisation, e.g., boxplots, or formal statistical tests such as the regression models or modified Grubb-test using the median and MD68-statistic, for example, this can be employed using the VCA R-package [52,56]. The proportion of outliers should not exceed 1% in the total dataset [18,52]. If outliers are identified, error estimates/analytical parameters such as precision should be calculated with and without the outliers to assess the impact of the outliers on the method’s performance results [18].
Supplementary case studies 1 and 2 (Additional files 3 and 4) show two worked examples of applying the validation framework at different stages. Case Study 1 (Additional file 3) contains a description and worked example of the commercial validation of the Insecticide Quantification Kit, a test used to quantify residues applied to wall surfaces during IRS [57, 58]. Case study 2 (Additional file 4) contains the study design for the external validation of the ViCTA Plus, an adaptation of the WHO cone test that is used to quantify mosquito behavioural responses to ITNs [59].
The evaluation process for vector control products entering the market has evolved over time. Prior to the establishment of the vector control prequalification department in 2016 [60], responsibility for product assessment lay with the WHO Pesticide Evaluation Scheme (WHOPES), who conducted evaluation studies and developed testing guidelines [4, 5]. These processes relied heavily on the WHO cone bioassay, tunnel tests, and experimental hut trials (EHTs) to provide information on the effectiveness of insecticidal products [4, 5]. Recently, some groups have assessed the reliability of the standard tests such as WHO cone and tunnel tests, and their findings suggest that the tests might not have been properly validated with unreliable results obtained when inherent stochastic variability and systematic bias are not accounted for [12, 61]. In addition, the lack of standard method validation guidelines for assessing novel methods resulted in ad hoc adoption of methods to characterise products even in the recognised absence of rigorous validation [62–66].
The new WHO ITN guidelines (currently in draft format) which are based on a “weight of evidence” approach require that quality data supporting an ITN listing should be produced using any appropriate test that the manufacturers deem suitable [27]. In this approach, appropriate method validation is paramount for building confidence in the method results and providing solid scientific evidence for its, and, by extension, the product’s, performance [17, 19, 20, 22, 44]. Ignoring the various sources of variability and/or not properly assessing bioassay and semi-field test precision could lead to misleading conclusions that inform future decisions [12, 41]. Therefore, a standardised approach to method validation that can be employed by manufacturers to ensure that the bioassays, tests and equipment used to evaluate vector control products are fit for purpose and reliable is necessary.
Unlike other fields where a method can be designed to be implemented in a single laboratory, the continuing processes of post-market testing, routine surveillance, and durability monitoring of vector control products necessitate that methods used to assess product characteristics are validated for implementation in a wide range of laboratories and testing sites. This complexity is increased by the range of product types used in vector control, e.g., ITNs, IRS treatments, repellents and larvicides, all of which might have different modes of action and applications, but which are united by the need to use entomological endpoints to characterise product characteristics [4, 5]. Following the validation framework described in this manuscript will ensure that methods are sufficiently sensitive to capture different modes of actions and can be used to assess the impact of new tools on various vector populations. Although the examples provided in this framework mainly focus on methods for evaluating product-based (ITNs) bioassays and semi-field studies, this approach can be extended to non-product-based assays with entomological endpoints, such as resistance monitoring bioassays.
It was not possible in this framework to provide standard cut-off points or acceptability criteria for various analytical parameters as are established in the chemistry or health sectors, due to the inherent variability in bioassays and the accepted variability in the specifications of the products that bioassays are used to characterise. Also, although some tests such as the WHO cone, tunnel tests and EHTs have been implemented for a long time, the focus of the experimental designs for these tests has been on satisfying the WHO pre-specified thresholds and not on assessing precision or robustness, hence historical data were judged to be unreliable for establishing standard acceptability criteria. However, methods on how best to establish suitable acceptability criteria for proposed methods have been provided.
The validation framework described in this report presents a more robust approach for assessing the performance of novel methods used to evaluate a product or tool than has been used previously, allowing a deeper understanding of the contribution of variability to each test method to be gained and delivering a more nuanced understanding of product characteristics and predicted effectiveness. Such understanding is vital given the continuing increase of pyrethroid resistance in Afro-tropical malaria vectors and the demand for novel vector control chemistries or tools to combat this [67–69]. The adoption of this framework will ensure the rigorous evaluation of methods used to characterise vector control products, or vector populations, leading to timely and accurate data, and, ultimately, the appropriate deployment of the most efficacious products and tools to at-risk communities.
AI Active Ingredient
CI Confidence Interval
CV Coefficient of Variation
CQV Coefficient of Quartile Variation
EHTs Experimental Hut Trials
GCV Geometric Coefficient of Variation
I2I Innovation to Impact
ICC Intraclass Correlation Coefficient
IRS Indoor Residual Spraying
ITN Insecticide Treated Net
SD Standard Deviation
SOP Standard Operating Procedure
VCT Video Cone Test
WHO World Health Organisation
Ethics approval and consent to participate
Not applicable
Consent for publication
Not applicable
Availability of data and materials
Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
The bioassay method validation framework was conceived by GMF, AS and RSL. AM, GMF and RSL wrote the manuscript of which the final draft was approved by all authors.
Acknowledgements
We thank Drs Mark Paine and Hanafy Ismail for permission to use the IQK case study. Professor Sarah Moore provided invaluable critical review and feedback on draft stages of the manuscript.
Funding
The work was funded by the Bill and Melinda Gates Foundation, as part of the project entitled, Developing entomological indicators to assess the public health value of next generation LLINs (INV-010445).
- WHO. Global report on insecticide resistance in malaria vectors: 2010-2016. Geneva; 2018.
- WHO. WHO malaria terminology 2021 update. 2021.
- Wilson AL, Courtenay O, Kelly-Hope LA, Scott TW, Takken W, Torr SJ, et al. The importance of vector control for the control and elimination of vector-borne diseases. PLoS Negl Trop Dis. Public Library of Science; 2020. https://doi.org/10.1371/journal.pntd.0007831.
- WHO. Guidelines for testing mosquito adulticides for indoor residual spraying and treatment of mosquito nets control of neglected tropical diseases: WHO pesticide evaluation scheme. 2006.
- WHO, WHO Pesticide Evaluation Scheme. Guidelines for laboratory and field testing of long-lasting insecticidal nets (No. WHO/HTM/NTD/WHOPES/2013.1). 2013.
- WHO. Vector Control Product Prequalification. 2023 [cited 2023 Jan 11]. Available from: https://extranet.who.int/pqweb/vector-control-products.
- Skovmand O, Dang DM, Tran TQ, Bossellman R, Moore SJ. From the factory to the field: considerations of product characteristics for insecticide-treated net (ITN) bioefficacy testing. Malar J. BioMed Central Ltd; 2021. https://doi.org/10.1186/s12936-021-03897-7.
- van den Berg H, da Silva Bezerra HS, Al-Eryani S, Chanda E, Nagpal BN, Knox TB, et al. Recent trends in global insecticide use for disease vector control and potential implications for resistance management. Scientific Reports 2021 11:1. Nature Publishing Group; 2021. https://doi.org/10.1038/s41598-021-03367-9.
- Graham K, Kayedi MH, Maxwell C, Kaur H, Rehman H, Malima R, et al. Multi-country field trials comparing wash-resistance of PermaNet and conventional insecticide-treated nets against anopheline and culicine mosquitoes. Med Vet Entomol. Med Vet Entomol; 2005. https://doi.org/10.1111/j.0269-283x.2005.00543.x.
- Fritz BK, Hoffmann WC, Bonds JAS, Haas K, Czaczyk Z. The biological effect of cage design corrected for reductions in spray penetration. J Plant Prot Res. -; 2014;54:395–400. https://doi.org/10.2478/JPPR-2014-0059.
- Owusu HF, Jančáryová D, Malone D, Müller P. Comparability between insecticide resistance bioassays for mosquito vectors: time to review current methodology? Parasit Vectors. BioMed Central; 2015. https://doi.org/10.1186/s13071-015-0971-6.
- Mbwambo SG, Bubun N, Mbuba E, Moore J, Mbina K, Kamande D, et al. Comparison of cone bioassay estimates at two laboratories with different Anopheles mosquitoes for quality assurance of pyrethroid insecticide-treated nets. Malar J. BioMed Central Ltd; 2022;21. https://doi.org/10.1186/s12936-022-04217-3.
- WHO. Manual on development and use of FAO and WHO specifications for pesticides. 2016.
- WHO. The evaluation process for vector control products. 2017.
- Huber L. Validation and Qualification in Analytical Laboratories. Second. Boca Raton: CRC Press; 2007.
- International Organization for Standardization (ISO). ISO 9000:2015 - Quality management systems — Fundamentals and vocabulary. 2015.
- NATA. Guidelines for the validation and verification of quantitative and qualitative test methods. 2012.
- CLSI. Evaluation of Precision of Quantitative Measurement Procedures; Approved Guideline-Third Edition. CLSI document EP05-A3. Wayne, PA: Clinical and Laboratory Standards Institute; 2014.
- Westgard JO. Basic Method Validation and Verification, Training in Statistical Quality Control for Medical Laboratories. 4th. ed. Madison WI 53717: Westgard QC, Inc; 2020.
- European Medicines Agency. ICH Topic Q2(R1) Validation of Analytical Procedures: Text and Methodology. 2006.
- Garcia PL, Buffoni E, Gomes FP, Quero JLV, Garcia PL, Buffoni E, et al. Analytical Method Validation. Wide Spectra of Quality Control. IntechOpen; 2011.
- Magnusson B, Örnemark U. Eurachem Guide: The Fitness for Purpose of Analytical Methods A Laboratory Guide to Method Validation and Related Topics. 2nd ed. 2014.
- Lees RS, Fornadel C, Snetselaar J, Wagman J, Spiers A. Insecticides for Mosquito Control: Improving and Validating Methods to Strengthen the Evidence Base. Insects 2023, Vol 14, Page 116. Multidisciplinary Digital Publishing Institute; 2023. https://doi.org/10.3390/insects14020116.
- United States Pharmacopeia. General Chapter, 〈1033〉 Biological Assay Validation. 2022.
- United Nations Office on Drugs and Crime. Guidance for the Validation of Analytical Methodology and Calibration of Equipment used for Testing of Illicit Drugs in Seized Materials and Biological Specimens. New York; 2009.
- International Council for Harmonisation. M10 Bioanalytical Method Validation and Study Sample Analysis: Guidance for Industry. 2022.
- WHO Prequalification of Vector Control Products. WHO Guideline for the Prequalification Assessment of Insecticide-Treated Nets - DRAFT FOR CONSULTATION. 2022.
- Lubrizol Life Science. Analytical Method Development and Validation. 2019 [cited 2023 Mar 17]. Available from: https://lubrizolcdmo.com/technical-briefs/analytical-method-development-and-validation/.
- Westgard JO. Basic method validation. 3rd ed. Madison, WI: Westgard QC. Inc.; 2008.
- LGC Limited. Method Validation Course 1125. 2015 [cited 2023 Mar 17]. Available from: https://www.lgcgroup.com/media/1504/precision.pdf.
- Sougoufara S, Yorkston-Dives H, Aklee NM, Rus AC, Zairi J, Tripet F. Standardised bioassays reveal that mosquitoes learn to avoid compounds used in chemical vector control after a single sub-lethal exposure. Scientific Reports 2022 12:1. Nature Publishing Group; 2022. https://doi.org/10.1038/s41598-022-05754-2
- Chaloner-Larsson G, Bioconsult G, Anderson OR, Antonio Da Fonseca M, Filho C, Gomez Herrera JF. A WHO guide to good manufacturing practice (GMP) requirements In collaboration with: Global Programme For Vaccines and Immunization Vaccine Supply and Quality Global Training Network 2 Good manufacturing requirements-Part 2: Validation. Geneva: World Health Organization. 1997.
- Lissenden N, Armistead JS, Gleave K, Irish SR, Martin JL, Messenger LA, et al. Developing consensus standard operating procedures (SOPs) to evaluate new types of insecticide-treated nets. Insects. MDPI; 2022. https://doi.org/10.3390/insects13010007.
- Lees RS, Armistead JS, Azizi S, Constant E, Fornadel C, Gimnig JE, et al. Strain Characterisation for Measuring Bioefficacy of ITNs Treated with Two Active Ingredients (Dual-AI ITNs): Developing a Robust Protocol by Building Consensus. Insects. MDPI; 2022;13. https://doi.org/10.3390/insects13050434.
- Mulholland M. Chapter 5 Ruggedness tests for analytical chemistry. Data Handling in Science and Technology. Elsevier; 1996;19:191–232. https://doi.org/10.1016/S0922-3487(96)80018-9.
- Hibbert DB. Systematic errors in analytical measurement results. J Chromatogr A. Elsevier; 2007;1158:25–32. https://doi.org/10.1016/j.chroma.2007.03.021.
- Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. J Eval Clin Pract; 2004. https://doi.org/10.1111/j..2002.384.doc.x.
- Sim J, Lewis M. The size of a pilot study for a clinical trial should be calculated in relation to considerations of precision and efficiency. J Clin Epidemiol. J Clin Epidemiol; 2012. https://doi.org/10.1016/j.jclinepi.2011.07.011.
- Johnson PCD, Barry SJE, Ferguson HM, Müller P. Power analysis for generalized linear mixed models in ecology and evolution. Methods Ecol Evol. Methods Ecol Evol; 2015. https://doi.org/10.1111/2041-210X.12306.
- Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Stat Med. Stat Med; 2019. https://doi.org/10.1002/sim.8086.
- Challenger JD, Nash RK, Ngufor C, Sanou A, Toé KH, Moore S, et al. Assessing the variability in experimental hut trials evaluating insecticide-treated nets against malaria vectors. Current Research in Parasitology & Vector-Borne Diseases. Elsevier; 2023. https://doi.org/10.1016/j.crpvbd.2023.100115
- Carley S, Lecky F. Statistical consideration for research. Emerg Med J. Emerg Med J; 2003. https://doi.org/10.1136/emj.20.3.258.
- Innovation to Impact (I2I). Fighting vector-borne diseases by optimizing vector control tools regulations. [cited 2023 Mar 17]. Available from: https://innovationtoimpact.org/
- ISO Committee on Conformity Assessment (CASCO). ISO/IEC 17025:2005 - General requirements for the competence of testing and calibration laboratories. 2005.
- Tholen DW. Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach, Approved Guideline. NCCLS Document EP6-A. NCCLS Document EP6-A. 2003.
- Cousineau D, Chartier S. Outliers detection and treatment: a review. Int J Psychol Res (Medellin). Universidad de San Buenaventura; 2010. https://doi.org/10.21500/20112084.844.
- Bower KM. Certain Approaches to Understanding Sources of Bioassay Variability - BioProcess International. 2018 [cited 2023 Mar 17]. Available from: https://bioprocessintl.com/upstream-processing/assays/certain-approaches-to-understanding-sources-of-bioassay-variability/.
- Arachchige CNPG, Prendergast LA, Staudte RG. Robust analogs to the coefficient of variation. J Appl Stat. 2022;49:268–90. https://doi.org/10.1080/02664763.2020.1808599.
- Ospina R, Marmolejo-Ramos F. Performance of Some Estimators of Relative Variability. Front Appl Math Stat. Frontiers Media S.A.; 2019;5. https://doi.org/10.3389/fams.2019.00043.
- Nakagawa S, Schielzeth H. Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists. Biological Reviews. John Wiley & Sons, Ltd; 2010. https://doi.org/10.1111/j.1469-185X.2010.00141.x.
- Stoffel MA, Nakagawa S, Schielzeth H. rptR: repeatability estimation and variance decomposition by generalized linear mixed-effects models. Methods Ecol Evol. John Wiley & Sons, Ltd; 2017. https://doi.org/10.1111/2041-210X.12797.
- Schuetzenmeister A, Dufey F. Package VCA: Variance Component Analysis. R package version 1.4.5. 2022 [cited 2023 Mar 17]; Available from: https://CRAN.R-project.org/package=VCA
- Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8:135–60. https://doi.org/10.2307/2987937.
- Giavarina D. Understanding Bland Altman analysis. Biochem Med (Zagreb). Biochem Med (Zagreb); 2015;25:141–51. https://doi.org/10.11613/bm.2015.015.
- Osborne JW, Overbay A. The power of outliers (and why researchers should ALWAYS check for them). Practical Assessment, Research, and Evaluation. 2019;9:6. https://doi.org/10.7275/qf69-7k43.
- Jiao X, Pretis F. Testing the Presence of Outliers in Regression Models*. Oxf Bull Econ Stat. John Wiley & Sons, Ltd; 2022;84:1452–84. https://doi.org/10.1111/obes.12511.
- Russell TL, Morgan JC, Ismail H, Kaur H, Eggelte T, Oladepo F, et al. Evaluating the feasibility of using insecticide quantification kits (IQK) for estimating cyanopyrethroid levels for indoor residual spraying in Vanuatu. Malar J. BioMed Central Ltd.; 2014;13:1–8. https://doi.org/10.1186/1475-2875-13-178.
- Ismail HM, Kumar V, Singh RP, Williams C, Shivam P, Ghosh A, et al. Development of a Simple Dipstick Assay for Operational Monitoring of DDT. PLoS Negl Trop Dis. Public Library of Science; 2016;10:e0004324. https://doi.org/10.1371/journal.pntd.0004324.
- Hughes A, Matope A, Emery M, Steen K, Murray G, Ranson H, et al. A closer look at the WHO cone bioassay: video analysis of the hidden effects of a human host on mosquito behaviour and insecticide contact. Malar J. BioMed Central Ltd; 2022;21:1–11. https://doi.org/10.1186/s12936-022-04232-4.
- WHO. WHO - Vector Control Product Prequalification. 2023 [cited 2023 Mar 17]. Available from: https://extranet.who.int/pqweb/vector-control-products.
- Praulins G, McDermott DP, Spiers A, Lees RS. Reviewing the WHO Tube Bioassay Methodology: Accurate Method Reporting and Numbers of Mosquitoes Are Key to Producing Robust Results. Insects. Insects; 2022;13. https://doi.org/10.3390/insects13060544.
- Chandre F, Darriet F, Duchon S, Finot L, Manguin S, Carnevale P, et al. Modifications of pyrethroid effects associated with kdr mutation in Anopheles gambiae. Med Vet Entomol. Med Vet Entomol; 2000;14:81–8. https://doi.org/10.1046/j.1365-2915.2000.00212.x.
- Okumu FO, Chipwaza B, Madumla EP, Mbeyela E, Lingamba G, Moore J, et al. Implications of bio-efficacy and persistence of insecticides when indoor residual spraying and long-lasting insecticide nets are combined for malaria prevention. Malar J. 2012;11. https://doi.org/10.1186/1475-2875-11-378.
- Massue DJ, Kisinza WN, Malongo BB, Mgaya CS, Bradley J, Moore JD, et al. Comparative performance of three experimental hut designs for measuring malaria vector responses to insecticides in Tanzania. Malar J. Malar J; 2016;15. https://doi.org/10.1186/s12936-016-1221-x.
- Oumbouke WA, Fongnikin A, Soukou KB, Moore SJ, N’Guessan R. Relative performance of indoor vector control interventions in the Ifakara and the West African experimental huts. Parasit Vectors. BioMed Central Ltd.; 2017;10. https://doi.org/10.1186/s13071-017-2365-4.
- Chinula D, Sikaala CH, Chanda-Kapata P, Hamainza B, Zulu R, Reimer L, et al. Wash-resistance of pirimiphos-methyl insecticide treatments of window screens and eave baffles for killing indoor-feeding malaria vector mosquitoes: An experimental hut trial, South East of Zambia. Malar J. BioMed Central Ltd.; 2018 [cited 2023 Mar 15];17:1–9. https://doi.org/10.1186/s12936-018-2309-2.
- Bhatt S, Weiss DJ, Cameron E, Bisanzio D, Mappin B, Dalrymple U, et al. The effect of malaria control on Plasmodium falciparum in Africa between 2000 and 2015. Nature. 2015;526:207–11. https://doi.org/10.1038/nature15535.
- Churcher TS, Lissenden N, Griffin JT, Worrall E, Ranson H. The impact of pyrethroid resistance on the efficacy and effectiveness of bednets for malaria control in Africa. Elife. Elife; 2016;5 https://doi.org/10.7554/eLife.16090.
- Hemingway J, Ranson H, Magill A, Kolaczinski J, Fornadel C, Gimnig J, et al. Averting a malaria disaster: will insecticide resistance derail malaria control? HHS Public Access. Lancet. 2016;387:1785–8. https://doi.org/10.1016%2FS0140-6736(15)00417-1.
Tables 1 and 3 are available in the Supplementary Files section.
No competing interests reported.
- Tables1and3.docx
- Additionalfile1MethodValidationManuscriptGlossary.docx
File name: Additional file 1 File format: Word document (.doc) Title: Glossary of standard method validation terms Description: This file contains the standard definitions for the terms used in method validation.
- Additionalfile2MethodValidationManuscriptPrecisionSummaryTable.docx
File name: Additional file 2 File format: Word document (.doc) Title: Precision estimates Description: This file contains the different methods of estimating precision, their associated formulas, pros and cons, and examples of R packages for implementing the analysis.
- Additionalfile3CaseStudy1IQKMethodValidationManuscript.docx
File name: Additional file 3 File format: Word document (.doc) Title: Case Study 1 – Commercialisation of the Insecticide Quantification Kit (IQK) Description: This file contains a detailed example on how method validation can be implemented using this proposed framework.
- Additionalfile4CaseStudy2VCTPLUSMethodValidationManuscript.docx
File name: Additional file 4 File format: Word document (.doc) Title: Case Study 2 – Video Cone Test (VCT) PLUS Laboratory Assay Description: This file contains another example demonstrating how method validation can be implemented using this current proposed framework.