The samples consisted of serum specimens sent to the Virology Laboratory, Hospital de Clínicas, Universidade Federal do Parana (HC-UFPR), Brazil, a tertiary care academic center for diagnostic purposes. Immunological and RT-qPCR for SARS-CoV-2 assays were performed in the HC-UFPR virology laboratory, which is certified by the Health Secretary of Paraná, Brazil.
This study was approved under a waiver of informed consent by the HC-UFPR institutional review board, Brazil.
Patients admitted to the HC-UFPR between March 1st and August 7th 2020 were considered eligible if they had respiratory symptoms that were suspicious of COVID-19. The performances of the four different LFAs were evaluated in serum samples obtained on corresponding dates that respiratory samples were collected for the performance of NAAT. A total of 119 serum or plasma samples were randomly selected for convenience, and distributed in the following groups (Figure 1):
Group with SARS-CoV-2 (COVID-19, n=82)
Definite SARS-CoV-2 infection, RT-qPCR positive (n=70)
Serum samples from COVID-19 patients, who tested positive for SARS-CoV-2 on RT-qPCR from nasopharyngeal samples. Participants admitted to the hospital COVID-19 unit or intensive care unit (ICU), n=60 (86%); outpatients (n=10; 14%); male 38 (54%); median (IQR) of age was 50 (38; 58.5) years old; time after symptoms onset 17 (12; 23) days. Two participants (2.9%) were asymptomatic.
Probable SARS-CoV-2 infection (n= 12)
Serum samples from participants who tested RT-qPCR negative for SARS-CoV-2 on nasopharyngeal samples, but fulfilled the World Health Organization (WHO) clinical diagnostic case definitions for SARS-CoV-2 [6] (Suppl table 1). All participants admitted in the hospital COVID-19 unit or intensive care unit (ICU); male 6 (50%); median (IQR) of age was 61.5(47.5, 74.5) years old; time after symptoms onset 11 (7.5; 19) days (definite vs. probable, p=0.046 and 0.118, respectively).
Group with other diseases (RT-qPCR for SARS-CoV-2 negative, n= 27)
Other viruses were identified in patients (n=8); RT-qPCR positive on a nasopharyngeal swab on a respiratory panel RT-qPCR test for other virus infections, Rhinovirus (n=6) and Coronavirus 229e/NL63 (1case); Epstain-Barr virus and Cytomegalovirus (EBV/CMV, n=1).
Severe acute respiratory syndromes (SARS), RT-qPCR for SARS-CoV-2 on respiratory samples were either negative and did not fulfill the WHO case definitions for SARS-CoV-2 [6] or other etiologies were identified (n= 19); the etiologies were: asthma, chronic obstructive pulmonary disease, endocarditis, pneumonia (2 cases), respiratory insufficiency, heart failure, hypoxia, pulmonary thromboembolism, stroke, cystic fibrosis (2 cases), viral infection, tuberculosis (2 cases), and bronchiectasis.
Median (IQR) of age was 54 (41, 76) years old; male 13 (48%) (definite vs. probable vs. other diseases, p=0.078 and 0.851, respectively); time after symptoms onset 4 (3; 8.5) days.
Group without disease (Healthy control group, n=10)
Ten serum samples from blood donors were collected in 2015 (HIV, HCV, HBV, HTLV I/II, syphilis, and Chagas disease negative). This group was not tested for SARS-CoV-2 by RT-qPCR as the samples were taken before the emergence of the virus in China [6].
Lateral flow immunochromatographic assays (LFAs)
All essays were based on the colloidal gold-labeled immunochromatography (GICA) principle and the one-step method with results obtained within 15 minutes, using serum or plasma samples. The kits use capture reaction to detect SARS-CoV-2 IgM/IgG or total antibody in the samples.
IgG, IgM, and Total antibodies for SARS-CoV-2.
The MedTest Coronavírus (COVID-19) IgG/IgM (MedLevensohn, Brazil), the COVID-19 IgG/IgM ECO Test (Ecodiagnóstica, Brazil), and the Camtech COVID-19 IgM/IgG Rapid Test Kit (Camtech Diagnostics Pte Ltd, Singapore) detect IgM and IgG. According to the manufacturer, the diagnostic characteristics of the tests are as follows: Eco test IgG (sensitivity, 95%; specificity, 99%; PPV 95%; NPV 100%); IgM (sensitivity, 90%; specificity,94%; PPV 89%; NPV 94%); MedTest IgG (sensitivity, 95.74%; specificity, 99.3%; PPV 97%; NPV 99%); IgM (sensitivity, 86.8%; specificity, 98.6%; PPV 94%; NPV 97%). The One Step COVID-19 Test (Guangzhou Wondfo Biotech Co.,China) kit detects antibodies of both IgM and IgG isotypes, without distinction, reactive towards SARS-CoV-2 antigens not specified by the manufacturer or in the literature [7]. According to the manufacturer, that test has 86.4% (95% confidence interval (CI) = 82.4–89.6%) sensitivity and 99.6% (95% CI = 97.6–99.9%) specificity.
Samples were tested in parallel in the four assays. The tests were performed at room temperature according to the manufacturer’s instructions. For all tests, the recommended sample volume of 10μl serum was added to the specimen well on the individual test cassettes followed by the supplied buffer. The result was visually read after 10 minutes, by two researchers; in case of doubt, a third one checked it. For combination of the IgM and the IgG kit, the test card or cassette has two test lines (M and G lines) and a quality control line (C line). The M line was fixed with a monoclonal anti-human IgM antibody for detecting SARS-CoV-2 antibody; the G line was fixed with a reagent for detecting SARS-CoV-2 antibody; the C line was fixed with a quality control antibody. Any visible band for IgG, IgM or unspecified immunoglobulin was an indicative for a positive result. The test card of the Wondfo kit has only one test line (T line) and a quality control line (C line). This kit does not differentiate IgM or IgG, thus, results were interpreted as positive or negative for SARS-CoV-2 antibody.
RT-qPCR for SARS-CoV-2
The RT-qPCR for SARS-CoV-2 was composed of dual positive results from a single NAAT targeting two different SARS-CoV-2 genes. Samples were collected with a rayon swab, and transported immediately to the virology laboratory in a viral transport medium (VTM). Samples were taken from the oral cavity and subsequently from the nasal cavity using a nasopharyngeal rayon swab. We performed RT-qPCR using the XGEN-Master COVID-19 (XGEN) for qualitative detection of nucleic acid in RT-qPCR format-reverse transcription, followed by amplification of a conserved region of the ORF1ab and N genes for SARS-Cov-2 [8], using specific primers and a fluorescence-labeled probe in respiratory samples. Specificity: 100% for SARS-CoV-2 (ORF1ab gene), 10 copies/reaction, with probability ≥ 95%. Sensitivity – SARS-CoV-2 (ORF1ab gene): 10 copies/reaction, with probability ≥ 95%. Sensitivity – SARS-CoV-2 (N gene): 50 copies/reaction, with probability ≥ 95%.
Statistical Analyses
The results were presented as the median (interquartile, IQR), number (n) and percentage, as appropriate. Categorical variables were compared between groups using the Fisher’s exact test and continuous variables were compared using the Mann–Whitney or Kruskal–Wallis test for non-parametric data, as appropriate. We performed the comparison of concordance and discordance proportions of the different kits with the McNemar test for paired nominal data. Results were considered significant at the 5% alpha level.
Clinical Performance characteristics of the tests
We evaluated, for each kit, the clinical performance of the LFA for SARS-CoV-2 IgM/IgG or total antibody (index test) in predicting the SARS-CoV-2 infection. The RT-qPCR for SARS-CoV-2 was the reference method. We analyzed separately the diagnostic performance for the detection of IgM and IgG antibodies in each test. For the calculation of SARS-CoV-2 IgG clinical performance, we included samples from patients who presented symptoms onset ≥ 8 days. If an IgG result was positive before the eighth day, we considered it true positive (n= 75). For IgM, we included all the samples collected.
The following clinical performance measures were calculated: sensitivity; specificity; accuracy (efficiency); positive and negative predictive values (PPV, NPV); Youden index [9]; positive and negative clinical utility index (CUI+, CUI−). The CUI values were classified as follows: excellent, ≥0.81; good, ≥0.64; fair, ≥0.49; poor, ≤0.49; and very poor, ≤0.36 [10, 11]. We calculated the positive and negative likelihood ratio (LR+, LR−) and diagnostic odds ratio (DOR), in which a LR+ value ≥10.0 indicates that a positive test almost confirmed the disease, a value of ~6.0 indicates that the disease was present, and a value of ~1.0 indicates that the test was not able to show whether the disease was present or not. A LR+ value ≤0.1 indicates that the disease was practically absent [12, 13]. The higher the DOR value the better the test is. The Matthews correlation coefficient (MCC) is a value between −1 and +1. A coefficient of +1 represents a perfect prediction, 0, no better than random prediction, and −1 indicates total disagreement between prediction and observation.
As the Wondfo test detects total antibodies, in order to compare its results with other tests, we evaluated the clinical performance results of all kits considering any positive results (IgM or IgG), in a subsequent analysis, we evaluated them separately (IgM and IgG).
Positive rates and levels of agreement between the kits were assessed using Cohen’s Kappa coefficients of agreement, which may be interpreted as follows: values ≤ 0 as indicating no agreement (i.e., purely random ), 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement [14].