Investigating the validity and reliability of the traditional checklist for assessment of specic inhalation techniques with pressurized metered-dose inhalers using the Aerosol Inhalation Monitor

Background: Correct inhalation techniques are essential for optimum delivery of drugs to the airways for patients with asthma and chronic obstructive pulmonary disorder. Checklists often are used to assess the techniques, but the validity of checklists is unknown and few studies have been conducted to date. We investigated the validity and reliability of checklists. Methods: A prospective, observational study was conducted at a tertiary care teaching hospital in Taiwan using an objective tool, the aerosol inhalation monitor (AIM). Inhalation techniques using pressurized metered-dose inhalers (pMDIs) were assessed by checklist and AIM. The κ coecient and prevalence and bias-adjusted κ (PABAK) were calculated to determine the consistency between the checklist and AIM in four specic techniques, including actuation, inspiratory ow, inhalation time, and breath hold time. Sensitivity and specicity were used to evaluate the validity of the checklist using these techniques. Results: The κ and PABAK values were −0.023 and −0.176 for actuation, 0.43 and 0.529 for inspiratory ow, −0.046 and −0.412 for inhalation time, and 0.12 and 0.686 for breath hold time, respectively. Sensitivity and specicity for the four individual steps were 0.706 versus 0.265, 0.833 versus 0.600, 1 versus 0.1, and 0.955 versus 0.143, respectively. Conclusion: Our results invalidate use of the checklist alone as a reliable tool to assess inhalation techniques. The low specicity indicates that the checklist tends to overestimate the accuracy of inhalation techniques. AIM can be added as an objective tool to assess and quantify specic steps for MDIs.

techniques. The low speci city indicates that the checklist tends to overestimate the accuracy of inhalation techniques. AIM can be added as an objective tool to assess and quantify speci c steps for MDIs.

Background
Inhalation therapy is the cornerstone of management of asthma and chronic obstructive pulmonary disorder (COPD), with advantages of direct delivery of medication, less adverse effects, and faster onset of action. However, treatment effect depends greatly on the correct inhalation techniques for optimum delivery of drug to the airways [1,2]. Unfortunately, treatment effect focuses exclusively on medication adherence and minimal attention is provided to inhaler techniques.
Many inhalation devices have emerged in recent years, each having speci c requirements to ensure the drug is delivered to the lungs [3][4][5][6][7][8][9]. Inhalers should be prescribed only after patients have been trained to use them properly [10]. Inhaler techniques must be rechecked and education must be reinforced regularly to maintain correct technique, since inhaler technique deteriorates again after education [3,11]. Improper use of inhalation devices may lead to poor control of asthma and COPD, with decreased quality of life, greater medical expenditures, increased hospitalization risk, and higher drug doses being prescribed [8,12]. Therefore, inhalation techniques must be assessed and a validated instrument for assessment is required to avoid over-or underestimation of the accuracy.
A checklist is used commonly to evaluate whether an inhalation device is being used properly [13][14][15].
Several checklists have been reported to evaluate use of different inhalers [13]. These checklists were either taken from previous studies or copied from the instructions provided with the inhaler device [16].
The inhalation steps are described in easily understood words. Steps, such as removing the mouthpiece cover and shaking the device, can be observed easily while most inhalation steps cannot be veri ed through visual observations due to subtle movements. For example, it is di cult to observe whether actuation of the metered-dose inhaler (MDI) is coordinated with inspiration or the inhalation ow rate is within the required range (20 L/min) [17,18]. Correct evaluation relies greatly on observer experience and is in uenced easily by various factors. Unfortunately, these subjectively evaluated steps often have great impact on drug delivery. However, few studies have highlighted poor consistency between the checklist and objective tool [19] and noted the potential overestimation of accuracy by the checklist in speci c inhalation techniques [20]. Usmani OS et al. [16] also stated that checklists were used without external validation. The validity of the checklist to evaluate those critical steps remains unknown.
An aerosol inhalation monitor (AIM) [21][22][23] quanti es key inhalation steps and identi es whether the actuation is coordinated with inspiration of MDIs, which can serve as an objective tool to assess inhalation techniques [24]. It can validly and reliably assess patient inhaler technique. We investigated the consistency of key inhalation steps between the checklist and AIM in MDIs. The AIM is further viewed as the gold standard to determine whether a checklist is a valid tool.

Methods
This prospective, observational study was conducted at a tertiary care teaching hospital in Taiwan between February 2018 and March 2018. Pharmacists were included as participants because the hospital initially began using the AIM to train pharmacists rather than for patient education. A total of 51 pharmacists at two hospitals with work experience for more than a year participated in the study after obtaining written informed consent. This study was approved by the Chang Gung Medical Foundation Institutional Review Board (IRB No. 201601112B0C101) and followed the Declaration of Helsinki Ethical Principles for Medical Research Involving Human Subjects. All participants were requested to demonstrate use of an MDI. The demonstration was recorded with a camera and could be viewed repeatedly afterwards. A placebo was used instead of active drug. The MDI placebo canister was xed to a single-use disposable MDI inhaler simulator, and the simulator was connected to an AIM. During the demonstration, the lights on the AIM were hidden from the participants, and the AIM measured four steps, including actuation, inhalation speed, inhalation time, and breath hold time. When they nished inhalation, participants were told to raise their hands and the researcher would press the end-of-breath button on the AIM. The technique summary was given after the demonstration was completed.

Assessment by checklist
Two independent experienced experts watched the recorded video and evaluated the whole process with the checklist, which was adapted from the National Asthma Education and Prevention Programs of America (NAEPP) step criteria [25]. These steps included shaking the contents well, removing the cap, breathing out slowly, opening the mouth with the inhaler 1 to 2 inches away or in the mouth with the lips tightly sealed around it, breathing in slowly and deeply through the mouth, and actuating the canister once and holding breath for 10 to 20 seconds. If both experts had any disagreement in an individual step, the disagreements were resolved through discussion with a third experienced expert.

Assessment by AIM
The AIM was designed by Vitalograph, Inc. (Lenexa, KS, USA) with the purpose of enabling medical professionals to objectively assess in detail how the test subject uses an inhaler. By measuring and recording four essential inhalation steps (actuation, respiratory ow, inhalation time, and breath holding time) when the subject actuated and inhaled through a mouthpiece of the simulator, AIM validated the effective use of MDIs and quanti ed poor inhaler techniques.
After completing the inhalation, the individual results lights and technique good/poor summary then would appear. Green colored individual results lights indicated correct activation, correct inhalation ow rate, inhalation time >3 seconds, and breath hold >3 seconds. Red lights implied incorrect activation, inhalation ow rate too high or low, inhalation time <3 seconds, and breath hold <3 seconds. Incorrect activation indicted that the canister was red too early or not at all. The technique summary was classi ed as fail, suboptimal, and good, and the corresponding lights were red, orange, and green.
The AIM was designed to measure the aforementioned four steps. Therefore, in our study, the consistency between the checklist and AIM focused on these four steps.
Because the κ value might be affected by the skewed data, we also calculated the percent agreement, proportion of speci c agreement, prevalence and bias indices, and prevalence and bias-adjusted κ (PABAK) to help interpret κ value. The magnitude of κ was affected by the prevalence of the attribute.
When the prevalence index was high, the magnitude of κ was reduced. In contrast, when there was a large bias index, κ was higher than when the bias index was low or absent [26]. In the aforementioned condition, PABAK adjusted the κ for imbalances caused by differences in the prevalence and bias. Since the AIM is an objective measurement tool, we further took it as a gold standard to calculate the sensitivity and speci city of speci c steps for the checklist. All analyses were performed using the Statistical Analysis System (SAS) statistical software package (v. 9.4.1; SAS Institute, Inc., Cary, NC, USA).

Page 5/13
The percentages of correct techniques assessed by the AIM and checklist are shown in Figure 1.
Overall, the accuracy evaluated by the checklist was higher than that by AIM (72.5% and 33.3% of participants had correct activation, respectively). The recommended inhalation time was >3 seconds, and the accuracy was 92.2% as evaluated by the checklist. However, only 21.6% of the subjects had adequate inhalation time when tested by the AIM.
The technique summary given by the AIM is shown in Table 1. Only 9.8% of participants had correct canister activation along with adequate ow rate, inhalation, and breath holding time allowing the medication to be brought to the deep lungs. The green light would light up in this circumstance. Of the patients, 17.6% performed a suboptimal technique, which was reported when inhalation time and/or breath hold was too short. In this situation, the medication was inhaled but only deposited at the upper lungs, which led to suboptimal effects. A total of 72.5% of participants were viewed and classi ed by AIM (Table 1) as having a failed drug delivery to the lung because the canister activation was incorrect or the inspiration ow rate was too fast.
The statistical agreements of the AIM and checklist are shown in Table 2. The percent agreements (proportion of consistent results between the checklist and AIM) were higher for inspiratory ow and The PABAK value was 0.686 for breath hold time, which indicated substantial agreement despite a low κ value (0.128).
Based on AIM as a gold standard, the sensitivity and speci city of the checklist are shown in Table 3. The checklist had high sensitivity but low speci city (0.1-0.6) in the four steps (0.706-1). The sensitivities of each step in the following order were 70.59%, 83.3%, 100%, and 95.45%, respectively, for actuation, inhalation ow, inhalation time, and breath holding time. The speci city was generally low at 26.47%, 60.00%, 10.00%, and 14.29%, respectively.

Discussion
It is easier for the evaluator to accurately evaluate whether subjects have good inhalation skills when the steps are performed in a somewhat exaggerated manner, such as an obvious chest raise when inhaling helps the evaluator know when the actuation begins. However, even if actuation is observed, a correct inspiration ow rate is di cult to measure visually. Electronic training tools are the solution to this problem [22], and the AIM has had been approved by the United States Food and Drug Administration.
Some studies have used the AIM to help determine the most suitable device for a patient [24], and others have focused on the consistency between different evaluation methods [27,28]. However, relatively few studies have been conducted on the validity of the checklist to date [29]. Even if the results between evaluators show good consistency when using a checklist, this does not mean the results are correct.
In the NAEPP step criteria, essential steps were de ned because they directly affect whether the inhalation is successful [25]. These steps can be classi ed further into two groups: device handling and inhalation techniques [30].
Device handling techniques, such as shaking the contents well and removing the cap, can be observed directly and only visually. Inhalation techniques, such as actuating the canister and breathing in slowly and deeply, are di cult to assess without assistance of a quanti cation tool. Even though the medication is inhaled successfully, the amount delivered to the lungs is unknown.
We used a checklist and the AIM to assess the accuracy of four major inhalation techniques of an MDI; that is, actuation, inhalation ow, inhalation time, and breath holding time. All results indicated a negligible to weak correlation between the two different methods.
All evaluators in our study were experienced and well-trained experts like those in other studies, and the results assessed by them with the checklist were consistent. However, we found that accuracy often was overestimated when the checklist was used especially for actuation and respiratory ow. Of the participants, 72.5% failed the inhalation because the canister activation was incorrect or the inspiration ow rate was too fast. Among these failing subjects, improper actuation was the main reason for failure. Appropriate actuation requires coordination between release of the dose by the hands and inhalation through the mouth [31]. Assessment via checklist overestimated the accuracy of coordination, since it was di cult to distinguish between good and bad coordination by visual observation alone.
When assessed by the AIM, only 33.3% of participants could actuate the inhaler correctly. The agreement was poor between the checklist and AIM. In the study of Taylor et al. [32], the fair-to-moderate agreement (according to the Cohen's κ statistic) between the subjective visual checklist assessment and the Inhaler Compliance Assessment audio recording device (κ = 0.49 for actuation coordination assessment and κ = 0.36 for PIFR assessment) highlighted the potential inaccuracy of the checklist method in assessing patient inhaler user technique.
In our results, the checklist and AIM demonstrated relatively high consistency in inspiration ow among the four techniques, even only with moderate agreement. Our result was similar to that of Taylor et al. [32] (κ = 0.36 in their study, 0.43 in our study) in inspiration ow rate. Though the objective tool used in our study was different from that used by Taylor et al. [32], both studies showed that the checklist and objective tools lacked consistency.
When the bias index or prevalence index is high, the κ (coe cient) may be under-or overestimated. The value of 0.913 and 0.804 for the proportion of positive agreement and prevalence indices, respectively, indicated an imbalance in the prevalence. Therefore, κ showed a downward bias in the breath hold time step in our results. In this circumstance, we may refer to the PABAK, which was 0.686. Most participants could achieve adequate breath holding time since many participants helped themselves count seconds with gestures. This act provided the evaluators with an absolute timing of when the breath hold began and ended. The other three steps were consecutive. In our study, we recorded the process so that the evaluators could watch it repeatedly afterwards to achieve a more close-to-reality evaluation. However, recording the inhalation in clinical daily practice is infeasible; therefore, a larger error is expected.
The study of Shealy et al. [20] also used a checklist and the AIM as assessment tools, but they only determined the prevalence of MDI misuse in both assessments, and concluded that the incidence of misuse was similar when assessed by the checklist and AIM. The study did not assess the accuracy of individual steps, so the consistency of the checklist and AIM in each step was unknown. It is worth noting that their result indicated that the checklist might overestimate the accuracy of actuation.
The checklist demonstrated relatively high sensitivity and low speci city when AIM was viewed as a gold standard. More attention must be paid to the low speci city, since the clinical purpose of patient education is to determine the wrong techniques and improve them. High sensitivity means that when the step is evaluated as correct by the AIM, the checklist also views them as correct. Low speci city indicates that the checklist is invalid to identify the wrong techniques and the right techniques evaluated by the checklist are not trustworthy. This is especially signi cant for actuation, inhalation time, and breath hold time. Inadequate inhalation time and breath hold time are the main reasons that cause suboptimal drug delivery, in which case the drug does not reach the deep lungs.
The advantage of the AIM is that a summary of whether the medication is inhaled successfully to the deep lungs or only deposited at the oropharyngeal site is given so that the user can know de nitely the result of each inhalation. The inhalation techniques are considered optimal only if there is no error in all essential stages. A failed or suboptimal inhalation may lead to not receiving the required doses of medication and as a result not achieving the desired treatment effect. Overestimating the accuracy of the inhalation techniques by the checklist may further lead to overlooking the actual reason for poor disease control and higher drug dosages may be prescribed.
We focused only on pressurized MDIs. However, different inhaler devices require different techniques, and our results could not extrapolate to the checklists used to evaluate other types of inhalers. One potential limitation of the study is not to assess the inter-rater reliability. Another limitation was the small number of participants and restriction of participant recruitment to pharmacists. This may not be representative of the general population.

Conclusion
While the checklist is invalid for assessment of inhalation techniques, including actuation, inhalation ow, inhalation time, and breath hold time, the AIM only assesses inhalation techniques instead of device handling techniques. AIM can be added as an objective tool to assess and quantify speci c steps for MDIs. Moreover, with the help of the technique summary lights given by the AIM, health care practitioners can further distinguish whether the medication is inhaled to the deep lungs or is deposited only at the upper airways. They then can improve the wrong techniques to make all inhalations optimal.

Consent for publication
Not applicable.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests
None to declare Funding This research was supported by a grant from Chang Gung Memorial Hospital (No. CDRPG1G0021). The funding source had no in uence on the design of the study, data collection and interpretation, or the content of this paper.
Authors' contributions DY, TL, and TC conceived and designed the analysis. TL, and YL performed data collection. TC, and DY contributed to data/analysis tools. TL Performed the analysis. YW, TL wrote the paper. All authors read and approved the nal manuscript.