A Study to Evaluate the Accuracy, Usability, and Acceptance of a Blood-Based HIV Self-Test Performed by Observed Intended Users in Canada

Introduction: Self-Testing for HIV (“HIVST”) is an emerging targeted intervention with the potential to increase the access, uptake and frequency of HIV testing and more effectively reach the undiagnosed especially in priority populations. The study objectives of this study were to (1) evaluate the INSTI HIV-Self-Test (HIV-ST) performance compared with laboratory reference testing, (2) document if intended users can perform the steps to use the HIV-ST device, and (3) document if intended users can successfully interpret contrived positive, negative, and invalid results. Study was intended to be submitted to Health Canada for review for license purposes. Methods: The study used a cross-sectional design and recruited consenting adults from four community sites across Ontario, Québec, and Manitoba between August 2019 and March 2020. The results of the observed HIV-ST were compared with results of the Abbott Architect HIV Ag/Ab Combo test. Results: Primary ecacy analysis on 678 completed HIV-ST revealed a positive percent agreement of 100% and a negative percent agreement of 99.5% with the comparator method. The overall percent agreement of results interpretation between participant and observer was 93.5%. Of the 708 participants who took part in the usability study, 92.4% of participants successfully performed the steps determined to be “critical” for successful completion of the test, 96.7% of participants found the instructions easy to follow, and 95% of participants indicated that they would use the test again. Of the 404 participants who interpreted the strong positive, weak positive, negative and invalid contrived results, successful interpretation ranged from 90.6% (for weak positive) to 99.3% (for negative). Conclusions: The addition of this very rst self-test into the Canadian HIV testing landscape could signicantly increase HIV testing rates. An approved blood-based HIV self-test in Canada can offer an accurate, acceptable and simple alternative to facility-based HIV testing, particularly when impacted by Coronavirus pandemic restrictions.


Introduction
Unlike other G7 countries, Canada is not seeing a reduction in the number of new people being diagnosed with HIV. Recent data from the Public Health Agency of Canada (PHAC) indicate that in 2016 an estimated 2,165 people became infected with HIV in Canada, an increase of 10% over 2014, and it is estimated that there are 63,110 people living with HIV in Canada as of 2016 [1]. Although Canada has endorsed the initial UNAIDS 90-90-90 target (90% diagnosed, 90% of those on treatment and in care, and 90% of those achieving viral suppression) [2], only 86% of those living with HIV are diagnosed (1st 90 target) -this represents over 9,000 individuals who have undiagnosed HIV infection across the country who are not adequately served by or connected to our health care system. For those people diagnosed with HIV, 81% are now on antiretroviral treatment (2nd 90 target), and of those, 91% have suppressed viral load (3rd 90 target) [1]. While we have reached one of three key UNAIDS targets, there are over 23,000 people who have not yet been diagnosed or who are not on antiretroviral therapy in Canada, which represents nearly a third of the total people living with HIV in Canada. Targeted interventions for testing, reaching the undiagnosed, and supporting more people to manage and adhere to treatment, and achieve viral suppression are needed to achieve Canada's UNAIDS commitment to all three of the 90-90-90 targets and move successfully towards ending its HIV epidemic in the next ve years. Indeed, diagnosing HIV is critically important for reaching targets aimed at controlling the HIV epidemic [3].
In 2016, WHO recommended HIV Self testing as an alternative to conventional facility-based testing [4].
By 2018, 59 countries had developed national policies on HIV self testing [5].
Self-Testing for HIV ("HIVST") is an emerging targeted intervention with the potential to increase the access, uptake and frequency of HIV testing, and could potentially become a high impact, low cost, and empowering alternative for those who may not otherwise test, notably for populations at high risk for HIV infection [6]. In a review on modern diagnostic technologies for HIV, the authors conclude that these novel technologies, including blood-based HIV self tests, show promise as they are associated with ease of use, high diagnostic accuracy, rapid detection, and the ability to detect HIV-speci c markers [7]. There is a growing body of supporting evidence showing the acceptance and usability of HIVST in various global key populations and groups [6,8,9], however similar study data for blood-based HIV self testing within the Canadian population had not existed prior to this study.
Three earlier studies were conducted in Montreal, Quebec with oral HIV self tests in key populations and low risk populations. In these studies, HIV self testing was demonstrated to be an acceptable and feasible testing solution, with data on technology innovations that can help increase testing and expedite linkages to care [10,11]. A survey of Canadian stakeholders demonstrated that HIV self testing is desired by Canadians yet, operational models and linkage data need to be thought through [12].
This study aims to provide independent data on the blood-based INSTI HIV Self-Test (HIV-ST) (BioLytical Laboratories, Richmond, BC, Canada) performance, acceptance, and usability in the hands of intended users in Canada The study objectives were to (1) evaluate the device performance, i.e., sensitivity, speci city, and percent agreement compared with laboratory reference testing, (2) document if intended users (non-professional and inexperienced in HIV self-testing) can successfully perform the steps to use the HIV-ST device without product demonstration, and (3) document if intended users can successfully interpret contrived strong positive, weak positive, negative, and invalid results.

Study Design
The study used a cross-sectional design and recruited consenting adults from four community-based sexual health and medical clinics or health centres across Ontario, Québec, and Manitoba between August 2019 and March 2020. Due to the emergence of the COVID-19 infection in populations throughout Canada and related restrictions concerning clinical practice, study sites were forced to discontinue study enrolment by March 18, 2020. The study was open to the English and French-speaking general public over the age of 18 who met the inclusion criteria and did not meet any of the exclusion criteria (see Appendix A for full list of inclusion and exclusion criteria). No participant was excluded based on race, gender, ethnicity or sexual orientation. Participants were compensated with $50 at their initial study site enrollment and an additional $25 when they returned for their laboratory testing results. Self-reported HIV risk activity for the year prior to the study enrolment was collected for each participant. Recruitment occurred during routine clinic visits through use of brochures, waiting room posters, and word of mouth. Sample size for the study was determined through published Health Canada guidelines (Health Canada, 2017).
The results of the HIV-ST performed and interpreted by intended users were compared with results of licensed laboratory-based Comparator Methods (CM). The CM used for all sites was the HIV-1/HIV-2 combination antigen/antibody test (Abbott Architect HIV Ag/Ab Combo test, Abbott Laboratories, Abbott Park, IL) which is licensed by Health Canada and is in routine use at the central public health laboratories of each of the participating provinces. The Geenius™ HIV 1/2 Con rmatory Assay (Bio-Rad Laboratories, Marnes-la-Coquette, France) was used for positive con rmation where necessary.

Study Protocol
After written informed consent was obtained, one venous blood sample was collected from each participant for testing at a central laboratory by the CM. Each participant then started the self-test -they self-collected a ngerstick blood specimen and performed the HIV-ST, including result interpretation, according to only the manufacturer's instructions for use. Each self-test performance was directly observed by a trained healthcare professional (Observer). The Observer did not tutor or interact with the participant conducting the HIV-ST but noted errors and other observations about the participant's test performance. There were 24 items developed, similar to those used in a usability assessment of multiple HIV self-test devices in South Africa [13] to capture usability information including instructions for use, pre-ST preparation, procedure for self-collecting blood sample, self-test procedure, and follow-up procedures. Half of the items (12 in total) were considered "critical" with respect to correct usability. The Observer also interpreted the HIV-ST result immediately after the participant and recorded the result separately. Lastly, a sub-set of participants who volunteered were provided the contrived "mock" result membrane cartridges individually and asked to interpret the results.
The provincial public health laboratories conducted CM testing on serum from venous blood samples collected from study participants and the results were provided back to the sites and subsequently to the central data coordination centre at St. Michael's Hospital, Toronto. Each central laboratory test result was used for study device performance evaluation with the HIV-ST, however study participants were able to return to the clinic site at a later date to receive their laboratory testing results if desired. All clinical care decisions were based solely on the results of the standard of care HIV testing in place at each site.

Electronic Data Capture
All participant data were collected by the study observers through the use of pre-loaded tablets at the study sites and stored using Qualtrics (Qualtrics, Provo, UT, USA https://www.qualtrics.com), a survey collection tool that encrypts data using Hypertext Transfer Protocol Secure (HTTPS) and enforces HTTP Strict Transport Security (HSTS). Participant data were stored on local encrypted servers in Toronto at St. Michael's Hospital for further data analysis. Only designated research team members accessed the password-protected data from the Qualtrics servers. The survey data for each participant were coded with a corresponding unique study ID to maintain privacy and con dentiality.

Data Analysis
All data analysis was conducted using IBM SPSS Statistics version 24. The Primary E cacy Analysis (performance) includes calculation of positive and negative percent agreement between study participants' self-interpreted results of the HIV-ST versus the 4th generation Abbott Architect test results. The proportion of study participants' interpretation of their self-test which is con rmed by the con rmatory test algorithm, and thereby considered "true" will be con rmed following determination of study participant's data inclusion in either the positive percent agreement (PPA) population or the negative percent agreement (NPA) population. Invalid self-test results and results indicated as "do not know" (n=56) were excluded from the calculation of PPA and NPA.
The overall 95% con dence interval will be determined where: As a secondary analysis, study participants' self-interpreted results were compared with the self-test results interpreted by the observer.

Study Participants
A total of 767 participants were recruited over the study period. Of the 767 participants that started the process, 89 participants were excluded from the Primary E cacy Analysis according to the exclusion list (see Appendix B for full list of excluded participants). Recruited participant numbers varied across the performance, usability and mock results interpretation arms of the study as not all recruited participants completed all elements. Table 1 shows the demographic pro le of participants included in the primary e cacy analysis (n=678). Over 75% of the sample was between the ages of 18 and 45 years of age. Over 60% identi ed as "male" and 37% as "female" (for the 21 who chose "other": 8 were "Trans" people, 8 identi ed as "non-binary", 1 as "agender", 3 as "queer", and 1 as "Two Spirit"). There were two participants who identi ed that they were pregnant. In terms of ethno-racial composition, approximately 44% identi ed as "White", 17% as "Black", 14% as "First Nations, Métis or Inuit", 16% as "Asian", 7% as "Latin American", and the remaining 2% as "other". In terms of educational achievement, approximately 27% of the sample completed High School, 23% completed College, and 48% a University degree or higher. Less than 5% indicated having any kind of reading or writing impairment. In the sample, 89% identi ed as "right-handed" and 55% indicated use of reading glasses. Approximately 45% were employed, 23% were unemployed, 13% were students, 2% were retired, and 17% preferred not to answer.
In terms of self-reported HIV status, 20% indicated that their status was unknown or had never been tested, while 80% indicated that they were HIV-negative. When asked about HIV testing, 74% indicated having prior experience with testing. When evaluating self-reported risk category for HIV, 600 participants (89%) were considered "high risk" for HIV infection. For speci c self-reported sexual behaviours in the year prior to survey, 30% identi ed as having condomless sex with men while 18% had condomless sex with women. Approximately 30% of participants indicated having multiple sexual partners in the past year, 26 individuals (4%) indicated that their sexual partner was living with HIV, and 51 participants (8%) indicated having ever injected drugs.

Primary E cacy (Performance) Analysis
After removing the 89 subjects who met the exclusion criteria from the 767 recruited participants, the primary e cacy analysis on the 678 who completed HIV-ST revealed a positive percent agreement of 100% and a negative percent agreement of 99.5% when comparing the valid self-tester results to the Abbott Architect (see Table 2 for full comparison of participant and laboratory test results and Table 3 for PPA and NPA calculations). The overall percent agreement between participant and observer for the selftest results was 93.5% (see Table 4 for full comparison of participant and observer interpretations of HIV-ST results). The overall "invalid" rate for the HIV Self-Test was 5.6%.

Usability Assessment Analysis
From the total of 767 study participants, 59 subjects withdrew after consenting, did not have a venous blood sample collected or did not complete the self-test, leaving 708 available for analysis with the usability assessment. Overall, the average for all desired outcomes was 91.8% and for the 12 critical items, the average was 92.4% (see Table 5 for full breakdown of usability questions). Despite as instructed by the kit instructions for use, only 47.2% of participants were observed to have washed and dried their hands. Just over 92% of participants were able to lance their nger correctly, and 88.7% were able to form a blood droplet, but there was a considerably lower number of participants (81.2%) who were able to get the blood droplet to fall directly into Bottle 1.
Following the ngerstick blood collection, there were three types of procedures carried out as part of the test process: (1) shaking bottles (accomplished with range of 88.8% to 90.2% of time), (2) pouring contents into the test device (97.3% to 99.0% of participants did so correctly), and (3) waiting for liquids to disappear before adding next liquid into device (participants did this procedure 98.0% to 98.2% of the time). Over 89% of participants performed the procedural steps of the self-test in the correct order, while less than 10% missed any steps and yet continued the process despite this, and only 1.0% quit the process before the test was completed. Table 6 shows the results from the self-test questionnaire given to each participant. Regarding the instructions for use, 96.7% of participants found them easy to follow. In terms of device use, over 90% of participants indicated that they were con dent with performing the test on their own, and 96.2% found the device easy to use. Approximately 95% of participants indicated that they would use the test again and would also recommend this test to a sexual partner or friend. In terms of where participants would prefer using the self-test, 58.6% indicated that they would do the test at home, while 41.1% indicated that they would prefer to do the test at a clinic.

Mock Test Interpretation Results
A total of 404 participants volunteered to complete the mock test interpretation (see Table 7 for a full breakdown of mock test results). Overall, 97.8% of participants correctly identi ed a "strong positive", 90.6% a "weak positive", 99.3% a "negative" result, 97.8% an "invalid" (with no control, no test) result, and 93.6% an "invalid" (with no control, with test) result.

Discussion
Since data from this study was intended to be submitted by the manufacturer to Health Canada for evidence of the INSTI HIV Self-Test safety and effectiveness as part of the device license application, it was important to determine if the self-test performance met published performance targets. The Health Canada Guidance for Manufacturers of Human Immunode ciency Virus (HIV) Rapid Diagnostic Tests (RDTs) for use at the Point of Care (POC) or for Self-Testing provides performance targets that must be met for clinical sensitivity and speci city: "Evidence that the RDT intended for use at the POC or for Self-Testing has a minimum sensitivity and speci city of ≥99% for HIV antibody detection should be provided " [14]. Primary e cacy analysis with 678 participants who completed the HIV self-testing study revealed a positive percent agreement of 100% and a negative percent agreement of 99.5% when comparing the valid self-tester results to the Abbott Architect. The overall speci city, indicated as negative percent agreement of 99.5%, with the lower bound of the 95% CI at 98.6% meets the required performance criteria.
Of the 6 HIV previously undiagnosed participants living with HIV that were con rmed by the Abbott Architect and Geenius comparator methods, 5 were positive with the self-test and 1 was invalid due to a low volume of ngerstick blood that was used in the INSTI self-test procedure. There was a 93.5% concordance in the self-test results interpretation between the untrained self-test study participants and the trained observers for the subjects in the primary e cacy analysis. Most discordant results were in the invalid or "do not know/not sure" (uncertainty) response categories.
A total of 38 participants (5.6% of sample) interpreted their self-test results as invalid, including one participant whose venous blood tested positive with the Abbott Architect. Of these, 22 were also interpreted as invalid by the observers, including the one participant living with HIV; however, 15 were interpreted as negative by the observers indicating there was a visible control dot present in the self-test which was either not observed by the self-tester, or not interpreted correctly. In some of these cases, the control dot intensity was indicated to be faint. A faint, but visible control dot may also have led to "do not know/not sure" interpretations recorded by both the self-testers and observers. Mitigations may be needed to improve the ngerstick blood collection process since usability results indicated that nearly 1 in 5 participants had di culties in getting a free-owing blood drop to fall into the INSTI sample diluent (Bottle 1). These mitigations could include revisions to the package insert to provide more clear instructions on the proper use of the lancet and subsequent blood drop collection, as well alerting the self-tester that even a faint dot intensity is considered valid. Consequently, self-test users should be instructed to conduct self-testing in a well-lit area.
There was an observed tendency for some self-testers to wipe their ngertip on the rim of the INSTI Bottle 1 (sample diluent) instead of allowing the blood drop to fall freely into the sample diluent. This was indicated for 60% (n=22/37) of invalid self-test results and for several self-test results that were still valid. A further mitigation to address the critical factor for the success in getting the blood drop into Bottle 1 could be to revise the self-test package insert to indicate, with illustration, that one should not scrape or wipe the blood into Bottle 1.
The mock results interpretations for the 404 participants who volunteered to participate in this portion of the study show a high concordance with the expected results. The percent agreement with expected results ranged from 90.6% for the weak positive result to 99.3% for the negative result, indicating that untrained self-testers can successfully interpret a range of self-test results including strong positive, weak positive, negative and invalid test results. However, mitigations to the package insert to call attention to the possibility of faint dot intensity could improve the lower success rate in interpreting weak positive results.
In general, participants expressed a high level of satisfaction with the HIV-ST experience as shown in Table 6: 96.7% of participants found the instructions for use easy to follow and 96.2% found the self-test device easy to use. 94.7% of participants would use the HIV-ST again, and 95.5% would recommend its use to a sexual partner or friend. Of interest and for consideration -approximately 60% of participants indicated a preference for using the self-test in a home environment while 40% would prefer to use the test at a clinic or health care facility, suggesting that both assisted (supervised) and unassisted selftesting strategies will likely be needed for "real world" access and uptake.
Overall, acceptance of the HIV-ST by intended users in this study was high, and similar to studies that reported that oral uid-based HIV self-testing is highly acceptable among a variety of populations [6,15,16]. Although data on use of blood-based HIV self-tests remains scarce [17][18][19], performance, usability and acceptance in this study were similar to an INSTI HIV Self-Test study conducted in Kenya [20]. This suggests that populations in both settings were able to conduct and perform the blood-based test as per instructions and agreement and performance metrics were aligned. This attests to the broad use of blood based self-tests as an alternative to oral self-tests.

Conclusions
The impact of the COVID-19 pandemic and its related public health control measures on HIV treatment and prevention, including access to facility-based testing, has had profound effects on individuals, communities, and societies across the world [21]. Implementation of HIV self-testing has been shown to increase HIV testing rates in men who have sex with men [22], and we found high acceptability amongst the diverse sample of participants in the current study. So, the addition of this HIV self-test into the Canadian HIV testing landscape, once approved by Health Canada, could have similar outcomes. An approved blood-based HIV self-test in Canada can offer an accurate, acceptable and simple alternative to facility-based HIV testing for key populations, particularly when impacted by COVID-19 pandemic restrictions.

Declarations Competing interests [Mandatory]
Richard A. Galli was a paid external contractor for the INSTI HIV Self-Test manufacturer, bioLytical Laboratories, during the early phase of the study conduct.