Samples
The samples used to assess the performance of the antibody tests were 278 serum or plasma samples from SARS-CoV-2 seronegative persons, 220 serum samples from COVID-19 patients, and 86 samples from 38 COVID-19 patients who were sampled at least twice after symptom onset (Fig. 1). The diagnosis of all COVID-19 patients was confirmed by NAAT.
The 278 negative samples were collected before 1 December 2019 and were from 35 healthy donors, 164 persons seeking medical care, and 79 patients with infectious diseases, out of which 32 were caused by bacteria, 7 by parasites, and 40 were caused by viruses (Supplementary text). The latter set of samples included 16 samples from patients infected by human coronaviruses: NL63 (n=6) and 229E (n=3), and patients co-infected with OC43 and HKU1 (n=7). Of the 278 negative sample donors, 102 (37%) were men and 132 (47%) were women, and information on sex was missing for 44 persons (16%). The median age was 31 years (range 2–83 years).
The 220 COVID-19 positive samples (each sample representing one person) were categorized into two subsets based on the time interval between symptom onset and sampling: 1–21 and ≥22 days post-symptom onset. A third subset consisted of samples for which information on symptom onset, sampling date, or both were missing. With few exceptions, the samples were collected within 3 months post-symptom onset. Of the 220 positive sample donors 54 (25%) were men and 41 (19%) were women, and information on sex was missing for 125 (57%) patients. The median age of the COVID-19 patients was 54 years (range 15–90 years). For 63 patients, information on disease severity was available; 20 were outpatients and 43 were hospitalized.
The 86 consecutively collected samples were categorized into seven subsets: samples collected week 1, 2, 3, 4, 5, 6, and week 7 after symptom onset. The samples were stored at
-20°C pending analysis and working aliquots at +4°C.
Studies have demonstrated a loss in infectivity of coronaviruses after heating21. We implemented a precautionary safety protocol and all samples were subjected to heat treatment at 56°C for 30 min before use. Initial testing revealed that one RDT had difficulties processing heat-inactivated samples. For this reason, heat-inactivated samples were subjected to a short centrifugation (1 min at 1000 × g) before being applied to RDT cassettes. Centrifugation markedly reduced the number of invalid tests and was adopted for testing of RDTs.
Immunofluorescence assay
The anti-SARS-CoV-2 IgG antibody titer was determined for all samples from COVID-19 patients by using an in-house IFA as previously described22. The IFA was evaluated by using samples from 45 persons with no prior history of COVID-19 and all samples were found negative in IFA at a dilution of 1:40. We therefore decided to analyze samples from COVID-19 patients in two-fold dilutions starting at a dilution of 1:80. The intensity of the fluorescence was determined independently (blinded reading) by two laboratory technicians experienced in both performing and interpreting IFA results and was graded as none (no or unspecific fluorescence) or present (rated “+”, “++”, or “+++”). The titer was expressed as the reciprocal of the highest dilution resulting in “+”. No fluorescence at dilution 1:80 are referred to as a titer <80. A fluorescence intensity of “++” or “+++” at a dilution of 1:320 was referred to as a titer of >320.
Rapid diagnostic tests
Five RDTs were included in the evaluation; 2019-nCoV IgG/IgM Rapid Test (Acro Biotech Inc., Rancho Cucamonga, CA, USA), Anti-SARS-CoV-2 Rapid Test (Autobio Diagnostics Co. Ltd, Zhengzhou, China), COVID-19 IgG/IgM Rapid Test (Healgen Scientific Limited Liability Company, Houston, TX, USA), NADAL COVID-19 IgG/IgM Test (Nal von Minden GmbH, Moers, Germany), and OnSite COVID-19 IgG/IgM Rapid Test (CTK Biotech Inc., Poway, CA, USA). The RDTs are here referred to as Acro, Autobio, Healgen, Nadal, and OnSite and their characteristics are summarized in Table S1. Because of limited sample volume, their performances were evaluated using a subset of the panel, which included 96 negative and 87 positive samples. The criteria for inclusion of positive samples was an IFA IgG titer of ≥80.
The RDTs were performed according to the manufacturers’ instructions. A test result was classified as negative (no detectable band), positive (clear band), or inconclusive (shade of colored line in the test-line-region) independently by two laboratory technicians using blinded reading. Discordant results between the technicians were handled as follows; an inconclusive and a positive result were interpreted as a positive result, and an inconclusive and a negative result were interpreted as an inconclusive result.
Platform-based tests
Six platform-based tests were evaluated; Architect SARS-CoV-2 IgG (Abbott, Chicago, IL, USA), EDI Novel coronavirus COVID-19 IgG ELISA (Epitope Diagnostics Inc., San Diego, CA, USA), Anti-SARS-CoV-2 ELISA (IgG) (Euroimmun, Lübeck, Germany), SARS-CoV-2 IgG S-ELISA (in-house Region Västerbotten (in-house RV))23, SARS-CoV-2 Spike S1-RBD Ig Bridge ELISA (MabTech AB, Stockholm, Sweden), and Wantai SARS-CoV-2 Ab ELISA (Beijing Wantai Biological Pharmacy Enterprise Co., Ltd., Beijing, China). The assays are here referred to as Abbot, Epitope, Euroimmun, in-house RV, Mabtech, and Wantai and their characteristics are shown in Table S1.
Commercial antibody tests were performed according to the manufacturers’ instructions with the exception of the use of heat-inactivated samples. For all commercial tests, the cutoff was calculated according to the package insert. In-house RV was used with a cutoff of 0.723. Samples were tested in duplicates in Euroimmun, Epitope, and in-house RV and the average absorbance reading of each duplicate was used for subsequent calculations according to the manufacturers’ instructions. Samples were tested in single wells in Abbott, Mabtech, and Wantai as they required an input volume of 300 µL, 25 µL, and 100 µL, respectively. Epitope, Euroimmun, and In-house RV were evaluated using the full set of negative and positive samples and Abbott, Mabtech, and Wantai with a subset of the samples. Samples were only tested once if not stated otherwise. The difference in the number of samples used and the number of replicates per assay were due to the available sample volume.
Statistical analysis
For the calculation of sensitivity and specificity, an inconclusive RDT result for a known negative sample was considered a positive result, while a known positive sample (IFA titer ≥80) with an inconclusive RDT was classified as a negative result. A borderline outcome in a platform-based test were considered as a negative result. Sensitivities and specificities with Wilson-score 95% confidence interval (CI) and interrater agreement (Cohen’s kappa, κ) with standard error (SE) were calculated by using STATA version 15.1. Student t-test was performed and graphs were made by using GraphPad Prism version 8.
Ethical statement
The evaluation was performed as part of the Public Health Agency of Sweden’s assignment to monitor communicable diseases, evaluate infection control measures, and support laboratories in diagnostic development and quality assessments in accordance with §§ 3.6 and 3.8 of the ordinance (2013:1020) from the Swedish Parliament. The leftover samples used in the evaluation were either obtained from the biobank repository at the Public Health Agency of Sweden or provided by clinical laboratories. These samples were used as stipulated in the regulations for use of such material in diagnostic quality assessment.