A Method to evaluate analytical sensitivity of qRT-PCR kits for SARS-CoV-2 detection

Background: The ongoing Coronavirus disease 2019 (COVID-19) pandemic has spread across the globe and is representing a huge challenge for all human population. Many commercial qRT-PCR assays have been developed to detect SARS-CoV-2, but related method validation especially the sensitivity evaluation has been insu�cient, resulting in some false-negative cases have been reported. Methods: The analytical sensitivity of nine brands of qRT-PCR kits for detecting SARS-CoV-2 was evaluated in parallel based on a newly developed certi�ed reference material, which was derived from genomic RNA of SARS-CoV-2 from clinical positive specimens. After validation of the the reference material by digital PCR, the detection sensitivity of these kits was preliminarily tested using the serially diluted reference material, resulting in three kits with two signi�cantly different sensitivity levels were selected for further evaluation. We sequenced the qRT-PCR products for assay speci�city evaluation, and used serial dilutions of the reference material to calculate ampli�cation e�ciency and estimate the limit of quanti�cation as well as 95% limit of detection.. Results: The results indicated that the analytical sensitivity varied markedly among these kits. For the three types of qRT-PCR kits (Kit-1, Kit-2 and Kit-7), speci�city of the PCR products was con�rmed by sequence alignment, in which the target amplicons completely matched the corresponding parts of the genome of SARS-CoV-2. The resulting limit of detection from replicate tests for the Kit-1 and Kit-2 was 5.6 copies (N), 3.5 copies (ORF 1ab), and 6.4 copies (N), 4.6 copies (ORF 1ab), respectively, at 95% probability. Compared with Kit-7, the limit of detection as well as limit of quanti�cation of Kit-1 and Kit-2 were signi�cantly lower, further supporting that the both kits worked well to detect low abundance of SARS-CoV-2. Conclusions: Considering that most of the tested kits have been approved for in vitro diagnostics (IVD) in China, the established method here provides a reliable tool to evaluate the sensitivity performance of various qRT-PCR


Introduction
The ongoing pandemic of the novel coronavirus named severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), identi ed in Wuhan City, Hubei Province 1 , has now infected over seven million people in more than 200 countries [2][3] .This enveloped positive-strand RNA virus is genetically similar to a bat virus in the subgenus Sarbecovirus [4][5] .The genome sequence appears to be closely related to the member of the viral species termed severe acute respiratory syndrome (SARS)-related CoV, the species responsible for the 2003 outbreak of SARS in humans 6 .
The COVID-19 pandemic represents an enormous challenge for molecular diagnostics.Reliable laboratory diagnosis is among the foremost priorities to facilitate public health interventions including con rming suspected cases and surveilling the virus.Real-time qRT-PCR detection of SARS-CoV-2, regarded as the gold standard of COVID-19 diagnostics, has been widely applied in the war against the virus 7-9 .Among the various strategies selected by diagnostic laboratories, ORF 1ab, N and E genes have been most frequently chosen as targets for the qRT-PCR assays [9][10][11] .Many in-house and commercial assays have been developed or are being developed to detect SARS-CoV-2 using e cient reaction systems [9][10][11] , but method validation for the assays has been insu cient, and low positive rates have been reported [12][13][14] .Hence, an appropriate method to evaluate sensitivity is crucial to improve detection e ciency of molecular diagnostics for SARS-CoV-2.
We here established a method for sensitivity evaluation of qRT-PCR kits for SARS-CoV-2, using reliable certi ed reference material derived from genomic RNA of SARS-CoV-2.Based on the preliminary test of nine types of commercial qRT-PCR kits produced in China, 3 kits with two sensitivity levels were selected to indicate the complete analytical work ow.

qRT-PCR kits
A total of nine brands of qRT-PCR kits for SARS-CoV-2 detection produced in China were tested preliminarily, and among them 6 kits had been approved for IVD by National Medical Products Administration of China (NMPA), while the rest 3 were research used only.The related information of these kits was listed in Table 1.While for all these assays, the exact working concentrations of main reaction components, such as those of primers and probes were unknown due to no related information provided by producers.These nine kits were named from Kit-1 to Kit-9 randomly in the following parts.

Certi ed Reference material
The certi ed reference material (GBW(E)091098-091099) we chose was approved by National Medical Products Administration of China (NMPA) and originated from genomic RNA of the SARS-CoV-2 from infected patients.The copy number concentration of the Open Reading Frame 1ab (ORF 1ab), nucleoprotein N gene, and the envelope protein E genes were quanti ed by digital PCR as the the reference value.Information on the reference material is given in Table 2.

Primers and probes used for digital PCR
Primers and probes that used (Table 3) were synthesized by Shanghai Sangon Biotech Co., LTD (Shanghai, China).The three sequence regions for ORF 1ab, E and N gene were recommended by the National Health Commission of China and the World Health Organization (WHO) 15 .

Digital PCR assay
The concentrations of ORF 1ab, N and E gene in the reference material were con rmed by digital PCR using the One-Step RT-ddPCR Advanced Kit for Probes based on the QX 200 platform (Bio-Rad Laboratories, USA) according to the manufacturer's instructions.

qRT-PCR assays
The serial dilutions of the reference material were tested by the qRT-PCR using the qRT-PCR kits for SARS-CoV-2 detection in parallel.The reactions were run using the Roche Light Cycler 480 II platform (Roche, Germany) and ABI QuantStudio 12K Flex (Thermo Fisher Scienti c, USA), strictly following the protocol in the instructions for each kit.Each dilution and the negative control were analyzed in triplicate.These ten kits were named from Kit-1 to Kit-9 randomly.

Sequence alignment
Sequence alignment was conducted by DNAMAN software to identify the ampli ed regions of ORF 1ab and N gene in the qRT-PCR assays.The reference sequence was retrieved from the NCBI (http://www.ncbi.nlm.nih.gov/genbank/)databases.The GenBank accession number of full-length of SARS-CoV-2 is MN908947.3,and that of SARS-CoV is AY394997.1.The sequences of ORF 1a and ORF 1ab were referred to the previous publication 16 .

Standard curve generation and PCR e ciency calculation
A standard curve for each target for each kit was constructed using the quanti cation cycle (Cq) values of serial three-fold dilutions of the CRM plotted against the logarithm of their concentrations in copies per reaction.Then PCR e ciency (E) was calculated as E=10 -1/k -1, where k is the slope of the linear regression equation produced using the standard curve.

Determination of limit of quanti cation (LOQ)
The limit of quanti cation was assessed using the standard curve.It was determined by the lowest template concentration that the assay could accurately quantify based on the linear portion of the standard curve, and the relative standard deviation (RSD) of copies number≤25%.

Probit analysis for limit of detection (LOD)
The LOD value for each assay was calculated by the probit analysis as the measured concentration at 95% probability (95% con dence intervals) [17][18][19] .Probit analysis was constructed using SPSS 17.0 (SPSS, Chicago, IL, USA).In the generated dot plot, the y-axis shows fraction positive of all parallel assays performed, and the x-axis shows the RNA copies per reaction of each given concentration (8 replicates per sample).

Validation of certi ed reference material
The sensitivity of the qRT-PCR kits for SARS-CoV-2 detection was evaluated using a recently developed certi ed reference material (CRM) that was derived from genomic RNA of the SARS-CoV-2 as the standard.The CRM has known concentrations of ORF 1ab, N and E gene, the three major targets used to detect the virus in qRT-PCR systems, although the speci c ampli cation regions may differ among the various diagnostic laboratories.
Digital PCR was performed to validate the concentrations of the three targets in the reference material.In the dot plots, all positive reactions were obviously separated from the negative, and only a few smears appeared in the middle part, which indicated that the reaction systems were well optimized for quanti cation (Figure 1 A-C).The resulting RNA inputs for the three target genes quanti ed by digital PCR was 921 copies/μL for ORF 1ab, 1689 copies/μL for N and 1098 copies/μL for E gene, respectively, which was consistent with the reference values of concentrations with the measurement uncertainty of the CRM (Figure 1D).And the no reverse transcriptase control and no template control were negative.Thus, the reference material was applied for subsequent sensitivity analysis of the commercial qRT-PCR kits produced for SARS-CoV-2 detection.

Preliminary detection by qRT-PCR assays
In our preliminary test, the certi ed reference material was 10-fold serially diluted four times (S1-S4) with RNA storage solution and carrier RNA (4.5 to 4.5 x 10 3 copies per reaction for ORF 1ab and 8.7 to 8.7 x 10 3 copies per reaction for N gene).Dilutions S1-S4 were then used as the templates for qRT-PCR assays of the nine types of commercial qRT-PCR kits according to the manufacturer's instructions.
The results revealed that all these test kits were capable to amplify the ORF 1ab and N gene fragments using samples from S1 to S3 (Figure 2, Additional le, while variations in detection rate between kits was found when testing lowest concentration of the dilutions.Among them, 5 of 9 kits were highly sensitive to detect the S4 sample in all replicates, and the assays using other 3 kits were partially positive.For the rest Kit-7, all tested replicates targeting ORF 1ab and N were negative (Table 4).This indicated that the analytical sensitivity varied markedly among these kits for SARS-CoV-2 detection.Based on the above result, three types of kits including two kits with high sensitivity level (Kit-1 and Kit-2) and one less sensitive Kit-7, were selected as the representative for further analysis.The ampli cation curves of assays generated by the three kits were shown in Figure 2.

Sequences alignment for target amplicons
To determine if the amplicons were matched to ORF 1ab and N sequences of SARS-CoV-2 sequence deposited in the GenBank, PCR products of the selected three kit targeting ORF 1ab and N genes were puri ed and sequenced.It demonstrated that for Kit-2 and Kit-7, the PCR products were quite consistent with amplicons ampli ed by the speci c primers and probes for SARS-CoV-2 detection suggested by the Chinese Center for Disease Control and Prevention (CCDC).For Kit-1, both target fragments matched another region of the genomic sequence, without overlapping with those of other two kits.The ampli cation product targets for ORF 1ab were located at the 3' end of ORF 1a and ORF 1ab, respectively (Figure 3A).Additionally, alignments of these amplicons with SARS-CoV-2 and SARS-CoV were complemented.The low divergence of sequences among primer binding domains (5' and 3' end of the ampli cation products) and the two virus (Figure 3B), suggested that these assays were di cult to discriminate the two types of coronavirus, as was the case in other reported qRT-PCR assays [9][10] .

Standard curve and limit of quanti cation (LOQ) analysis
The Kit-1, Kit-2 and Kit-7 were further evaluated by standard curve generation based on quanti cation cycle (Cq) values of a new serial dilutions of the reference material.From the standard curves, the ampli cation e ciencies (E) of both targets using the three kits were ranged from 90% to 105% (R 2 > 0.98), revealing that all these reactions had been well optimized to balance both targets detection (Figure 4).The LOQ of the the three kits for ORF 1ab gene was 23, 23 and 61 copies per reaction, respectively.While for the N gene, it was 26,26 and 78 copies per reaction, respectively.Thus, compared with other two kits, the kit-7 displayed poor linearity range for quantifying the target genes, which was in concordance with the above preliminary test results for analytical sensitivity.

Limit of detection (LOD) analysis
The reference material of the SARS-CoV-2 genomic RNA was initially diluted into the concentrations around S4 (4.5 copies per reaction for ORF 1ab and 8.7 copies per reaction for N gene) for the Kit-1 and Kit-2, as well as those between S3 and S4 for the Kit-7, which were estimated to be nearly at the detection end point.Probit analysis revealed a LOD of 3.5 copies for the ORF 1ab gene, and 5.6 copies for the N gene at 95% probability using kit-1 (Figure 5A).For kit-2, the LOD at 95% hit rate was 4.6 copies for ORF 1ab and 6.4 copies for N gene (Figure 5B).These results were roughly in line with the theoretical LOD of qRT-PCR reaction (3 molecules per reaction) when only sampling noise would contribute to replicate variation at 95% con dence according to the Poisson distribution [20][21][22] .The discrepancy was probably caused by other factors in the real reaction conditions.While the resulting LOD for the Kit-7 was much higher, which was 14.3 copies for ORF 1ab and 20.4 for N assays (Figure 5C).Furthermore, for Kit-1 and Kit-2, the LOD values for both targets at 95% probability were less than 10 copies/reaction, demonstrating a sensitivity similar to that of the digital PCR.

Discussion
The SARS-CoV-2 pandemic represents a huge global challenge for diagnostic detective work; various types of qRT-PCR kits for SARS-CoV-2 detection have been developed and deployed by different manufacturers to keep pace with demands.Though many factors including sample collection, RNA extraction and instrument calibration can affect the diagnostic results 17 , qRT-PCR assay performance is of particular importance.Compared with some in-vitro transcribed RNA standards, the reference material we selected has an advantage in that it is derived from genomic RNA of SARS-CoV-2, which should theoretically re ect the true features of clinical specimens and allow us to detect all genes of the SARS-CoV-2 genome 11 .Thus as a standard, it could provide better validation and quality control for the qRT-PCR assays deployed for SARS-CoV-2 molecular diagnostics.
Interestingly, the number of RNA copies differs between the three targets in the reference material according to the digital PCR results, similar to what was found by Chu et al. 9 .Several factors may be responsible, but differences in speci c transcript abundances might be the major contributor.The reference material is derived from clinical positive specimens, so it probably contains infected cells from the donor that express speci c subgenomic mRNA (sgRNA), with an abundance that is commonly signi cantly higher than that of genomic RNA.For instance, the sequence of the N gene universally exists in the 3' end of the sgRNA, resulting in more N gene copies in the reference material, an assumption that is validated by our data.In addition, variation in reverse-transcription (RT) e ciency for the targets in the one-step RT-qPCR assays could be another reason.
To study analytical sensitivity of the kits, in this work serial dilutions of the reference material were used as templates to perform qRT-PCR assays, and standard curves were generated to calculate ampli cation e ciency, followed by the LOD determination for ORF 1ab and N gene.The Kit-1 and Kit-2 appeared to provide much better sensitivity performance compared with that of the Kit-7.Both of the former two kits worked well to detect low abundance of the reference material, and the resulting LOD of these assays was roughly in line with previous reports by Corman et al 10 .In addition, the better two kits achieved a sensitivity level similar to that of the digital PCR, although the sensitivity may be reduced when actual swab specimens are used due to the inferior quality of the RNA samples.Based on the above results, the work ow here enabled us to evaluate and compare the analytical sensitivity of these assays effectively.
Considering sensitivity performance, digital PCR provides ultrasensitive and absolute nucleic acid quanti cation without using a standard curve generated by calibrators.This method enables the direct quantifying of nucleic acids molecules by separating the sample into numerous partitions for individual reactions.Compared with qRT-PCR, the digital PCR allows a more sensitive and precise measurement, especially for low viral loads.And it is more tolerant of potential PCR inhibitors in the reaction mixture, which is especially desirable for clinical detection [23][24][25] .Despite these advantages, digital PCR has not been widely applied for SARS-CoV-2 diagnostics due to its higher cost and more complicated operation, which is more prone to error when used by the inexperienced 26 .Thus, it may not now be suitable for large-scale screening of suspected cases, but it is a powerful complementary tool for con rming infection in those patients who are highly likely to be infected based on clinical presentation and exposure history, but tested negative by qRT-PCR.

Conclusion
Using a newly developed certi ed reference material, a method to evaluate the sensitivity of qRT-PCR diagnostic kits for SARS-CoV-2 was established.Compared with other reference material derived from invitro transcribed RNA published previously [10][11]27 , the reference material used here was derived from genomic RNA of SARS-CoV-2. Deection sensitivity differed between the three tested kits, stressing the importance of quality control for qRT-PCR assays in the ongoing pandemic.LOD values for these assays revealed that the well-optimized qRT-PCR assays could be as highly sensitive as the digital PCR, although sensitivity may be reduced when clinical specimens are assayed.
qRT-PCR is globally the gold standard for detecting SARS-CoV-2, but different molecular diagnostic strategies might be adopted for some speci c circumstances.The ultra-sensitive digital PCR can be a powerful complementary tool for those specimens highly suspected to be positive but tested negative in qRT-PCR.The qRT-PCR combined with digital PCR when necessary will undoubtedly enhance the diagnostic capability for combating the SARS-CoV-2 pandemic.

Figures Figure 1
Figures

Figure 2 Real
Figure 2

Figure 3 A
Figure 3

Figure 4 Standard
Figure 4