To evaluate the performance of the Simplexa™ COVID-19 Direct assay—a quasi-point-of-care SARS–CoV-2 detection assay—we compared the results from 125 cases that first tested either positive (n = 54) or negative (n = 71) with the Allplex™ 2019-nCoV assay—the first implemented SARS–CoV-2 detection assay in our laboratory. As shown in Table 1 (for details, see Table S1 in the supplemental material), CT values of Allplex™ 2019-nCoV positive samples ranged from 17.9 to 39.3 for E, RdRP, and N genes (33 samples), 28.4 to 39.3 for RdRP and N genes (9 samples), 33.7 to 39.4 for the N gene (11 samples), and 35.6 to 37.1 for E and N genes (1 sample). In particular, the mean (± SD) CT value for the E gene (26.4 ± 3.9) was lower than the values for RdRP (28.0 ± 3.6; P = 0.09) or N (28.9 ± 4.4; P = 0.02) genes in 33 samples and the value for the N gene (37.1) in 1 sample. Forty-seven of 54 positive samples by the Allplex™ 2019-nCoV assay had also positive results with the Simplexa™ COVID-19 Direct assay. The CT values of positive Simplexa™ COVID-19 samples ranged from 17.5 to 39.7 for S and ORF1ab genes (40 samples), 21.0 to 35.6 for the ORF1ab gene (4 samples), and 29.3 to 34.9 for the S gene (4 samples). In particular, the mean (± SD) CT value for the S gene (27.9 ± 5.1) was lower than the value for the ORF1ab gene (27.9 ± 3.9; P = 0.99) in 40 samples.
The agreement between the Allplex™ 2019-nCoV assay and the Simplexa™ COVID-19 Direct assay results was 93.6% (Table 1). Of eight samples with discordant results, seven samples tested positive with the Allplex™ 2019-nCoV assay (the N gene was detected alone or in combination with E and/or RdRP genes) but negative with the Simplexa™ COVID-19 Direct assay. The remaining one sample tested negative with the Allplex™ 2019-nCoV assay but positive with the Simplexa™ COVID-19 Direct assay (both S and ORF1ab genes were detected). As detailed in Table S1, the mean (± SD) CT value of the N gene in the seven samples with discordant results was 34.7 ± 5.9, and this value differed from that of the 47 remaining Allplex™ 2019-nCoV positive samples (31.2 ± 5.0; P = 0.09).
Table 1
Overall results of 125 NOS samples tested by two molecular SARS–CoV-2 detection assays.a
|
Value for the following assays expressed as number (CT range):
|
Between-assay value
|
|
Allplex™ 2019-nCoV
|
Simplexa™ COVID-19 Direct
|
Positive results
|
|
|
|
All
|
54 (17.9–39.4)
|
48 (17.5–39.7)
|
|
By target(s)
|
|
|
|
E, RdRP, and N genes
|
33 (17.9–39.3)
|
|
|
E and N genes
|
1 (35.6–37.1)
|
|
|
RdRP and N genes
|
9 (28.4–39.3)
|
|
|
N gene
|
11 (33.7–39.4)
|
|
|
S and ORF1ab genes
|
|
40 (17.5–39.7)
|
|
S gene
|
|
4 (21.0–35.6)
|
|
ORF1ab gene
|
|
4 (29.3–34.9)
|
|
Negative results
|
71 (0.0–0.0)
|
77 (0.0–0.0)
|
|
|
|
|
|
No. of concordant results
|
|
|
117
|
No. of discordant results
|
|
|
8
|
% agreement
|
|
|
93.6
|
Cohen’s kappa (95% CI)
|
|
|
0.86 (0.78–0.96)
|
aNOS, nasal/oropharyngeal swab; CT, threshold cycle; E, envelope; RdRP, RNA-dependent RNA polymerase; N, nucleocapsid; S, spike; ORF, open reading frame.
Analytic performance of molecular assays
We used the Quanty COVID-19 assay to arbitrate the results of the two molecular assays under comparison. Table S1 provides an overview of testing results. Fifty-five samples, including Allplex™ 2019-nCoV (n =54) and Simplexa™ COVID-19 (n = 48) positive samples, tested positive and the remaining 70 samples tested negative for all the N gene regions targeted by Quanty COVID-19 assay. In summary, the Simplexa™ COVID-19 Direct assay failed to detect seven positive samples and the Allplex™ 2019-nCoV assay failed for one positive sample. Interestingly, this one sample tested positive also with the Simplexa™ COVID-19 Direct assay.
As shown in Tables 2 and 3, sensitivity and negative predictive value (NPV) of the Allplex™ 2019-nCoV assay were 98.2% and 97.2%, respectively, and those of the Simplexa™ COVID-19 Direct assay were 87.3% and 90.9%, respectively. When analyzing the results according to single assay’s targets, we found lower sensitivities and NPVs for RdRP (76.4% and 84.3%, respectively) and E (61.8% and 76.9%, respectively) genes in one assay (Table 2) and for both S and ORF1ab (80.0% and 86.4%, respectively) genes in the other assay (Table 3).
Table 2
Performance of the Allplex™ 2019–nCoV assay according to the Quanty COVID-19 assay used as comparator assessed by target.a
|
Allplex 2019–nCoV results by target
|
E gene
|
RdRP gene
|
N gene
|
Total
|
No. matched positives
|
34
|
42
|
54
|
54
|
No. matched negatives
|
70
|
70
|
70
|
70
|
No. Allplex 2019–nCoV misses
|
21
|
13
|
1
|
1
|
% agreement
|
83.2
|
89.6
|
99.2
|
99.2
|
% sensitivity (95% CI)
|
61.8 (47.7–74.6)
|
76.4 (63.0–86.8)
|
98.2 (90.3–100.0)
|
98.2 (90.3–100.0)
|
% specificity (95% CI)
|
100.0 (94.9–100.0)
|
100.0 (94.9–100.0)
|
100.0 (94.9–100.0)
|
100.0 (94.9–100.0)
|
% PPV (95% CI)
|
100.0 (89.7–100.0)
|
100.0 (91.6–100.0)
|
100.0 (93.3–100.0)
|
100.0 (93.3–100.0)
|
% NPV (95% CI)
|
76.9 (66.9–85.1)
|
84.3 (74.7–91.4)
|
97.2 (90.3–99.7)
|
97.2 (90.3–99.7)
|
aE, envelope; RdRP, RNA-dependent RNA polymerase; N, nucleocapsid; CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value.
Table 3
Performance of the Simplexa™ COVID-19 Direct assay according to the Quanty COVID-19 assay used as comparator assessed by target.a
|
Simplexa COVID-19 results by target
|
S gene
|
ORF1ab gene
|
Total
|
No. matched positives
|
44
|
44
|
48
|
No. matched negatives
|
70
|
70
|
70
|
No. Simplexa COVID-19 misses
|
11
|
11
|
7
|
% agreement
|
91.2
|
91.2
|
94.4
|
% sensitivity (95% CI)
|
80.0 (67.0–89.6)
|
80.0 (67.0–89.6)
|
87.3 (75.5–94.7)
|
% specificity (95% CI)
|
100.0 (94.9–100.0)
|
100.0 (94.9–100.0)
|
100.0 (94.9–100.0)
|
% PPV (95% CI)
|
100.0 (92.0–100.0)
|
100.0 (92.0–100.0)
|
100.0 (92.6–100.0)
|
% NPV (95% CI)
|
86.4 (77.0–93.0)
|
86.4 (77.0–93.0)
|
90.9 (82.2–96.3)
|
aS, spike; ORF, open reading frame; CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value.
Relationship between samples’ CT values and viral loads
Table 4 shows the viral loads determined by the Quanty COVID-19 assay (expressed as log10 N1 copies per ml) in positive samples, which were stratified by the Allplex™ 2019–nCoV (E, RdRP, and N) or the Simplexa™ COVID-19 Direct (S and ORF1ab) assays’ targets. We found highest proportions of E (29.4% and 26.5%, respectively), RdRP (23.8% and 26.2%, respectively), and N (22.2% and 20.4%, respectively) gene detections, as well as S (27.3% and 22.7%, respectively) and ORF1ab (25.0% and 22.7%, respectively) gene detections in samples with viral load levels ranging from >3.0 to ≤4.0 or >4.0 to ≤5.0 log10 copies per ml.
To determine if there was relationship between viral load and CT value, we performed a Spearman’s correlation analysis. Before that, samples with CT values ≥40 by the Allplex™ 2019-nCoV assay or the Simplexa™ COVID-19 Direct assay were assigned a value of 40. Analyzing all 55 samples that tested positive or negative by the assays, we found a strong (negative) association between the CT values of N (Spearman’s ρ = −0.92; P <0.001) and RdRP (ρ = −0.91; P <0.001) genes—detected by the Allplex™ 2019-nCoV assay—and viral loads (Fig. 1). Conversely, we found a less strong (negative) association between the CT values of ORF1ab (ρ = −0.65; P <0.001) and S (ρ = −0.80; P <0.001) genes—detected by the Simplexa™ COVID-19 Direct assay—and viral loads (Fig. 2).
Table 4
Detection results of Allplex™ 2019–nCoV and Simplexa™ COVID-19 Direct assays’ targets according to viral load levels in positive NOS samples.a
Viral load levels (log10 copies/ml)
|
No. (%) of detections by Allplex 2019–nCoV targetsb
|
No. (%) of detections by Simplexa COVID-19 targetsc
|
E gene
|
RdRP gene
|
N gene
|
S gene
|
ORF1ab gene
|
|
n = 34
|
n = 42
|
n = 54
|
n = 44
|
n = 44
|
≤1.0
|
0 (0.0)
|
0 (0.0)
|
1 (1.9)
|
0 (0.0)
|
0 (0.0)
|
>1.0–≤2.0
|
3 (8.8)
|
5 (11.9)
|
12 (22.2)
|
7 (16.0)
|
10 (22.7)
|
>2.0–≤3.0
|
2 (5.9)
|
6 (14.3)
|
8 (14.8)
|
6 (13.6)
|
5 (11.4)
|
>3.0–≤4.0
|
10 (29.4)
|
10 (23.8)
|
12 (22.2)
|
12 (27.3)
|
11 (25.0)
|
>4.0–≤5.0
|
9 (26.5)
|
11 (26.2)
|
11 (20.4)
|
10 (22.7)
|
10 (22.7)
|
>5.0–≤6.0
|
7 (20.6)
|
7 (16.7)
|
7 (13.0)
|
6 (13.6)
|
5 (11.4)
|
>6.0–≤7.0
|
3 (8.8)
|
3 (7.1)
|
3 (5.5)
|
3 (6.8)
|
3 (6.8)
|
aThe Quanty COVID-19 assay was used to determine the SARS–CoV-2 RNA level in nasal/oropharyngeal swab (NOS) samples.
bThe Allplex 2019–nCoV targets the E (envelope), RdRP (RNA-dependent RNA polymerase), and N (nucleocapsid) genes of SARS–CoV-2.
cThe Simplexa COVID-19 targets the S (spike) and ORF1ab (open reading frame 1ab) genes of SARS–CoV-2.