Evaluating the value proposition of DIA to replace DDA for co-IP requires data sets where both types of experiments were performed. At the time of writing, the ProteomeXchange repository (Deutsch et al. 2023) offers a host of co-IP data sets for Thermo instruments, but a majority of these PXD entries contain DDA experiments only. Seeking ProteomeXchange Bruker and SCIEX QqTOF co-IP experiments where both DDA and DIA were included produced even fewer options. The sets included here will begin with two complex samples: mouse digestive tissue lysates intended as an input to immunoprecipitation and nanoparticle-enriched lysosomal proteins (“Thermo SPION”). It will move from there to a variety of immunoprecipitations: co-IP experiments in human cell lines (“Thermo LINE-1”), fruit fly Schneider 2 cells (“SCIEX RACK1”), murine basal cells (“SCIEX ID4”), and human MCF-7 mammary epithelium cells (“Bruker DUBs”). Having both DDA and DIA data for these experiments enables a direct comparison of spectral libraries derived from DDA database search (via FragPipe, MaxQuant, or Spectronaut) or DIA database search (via FragPipe, DIA-NN, or Spectronaut). Quantifying the peptides of the spectral libraries can then be carried out in the DIA experiments (via all of these algorithms).
Spectrum library diversity from DDA or DIA database search
We framed two hypotheses before examining the spectral libraries derived from these experiments:
A) Subsets of peptides are most likely to be identified by a particular identification algorithm.
B) Subsets of peptides are more likely to be identified in DDA than in DIA, or vice versa.
A given peptide might have a greater chance of identification by one search engine rather than another because the software embeds a fragmentation model that better predicts the fragments to be seen in an MS/MS of the peptide (C. Silva et al. 2019). If the same fragmentation model is in operation whether that search engine is operating on DIA or DDA experiments, the advantage in identifying this peptide would apply in both cases. This phenomenon can be evaluated in FragPipe, since MSFragger can manage either DDA or DIA identification, and in Spectronaut, where the Pulsar search engine can manage either experiment type. Only these two software workflows of the four we examined are designed to identify directly both DIA and DDA experiments.
Since the creation of DIA instrument methods, many in the proteomics community have assumed that the best spectral library for a sample type must be created from DDA experiments. Because DDA produces an MS/MS of fragments from an isolated peptide, recognition of that peptide by a search engine is more likely. This assumption, however, ignores advantages of DIA for identification. First, DIA produces redundant fragment measurements for each peptide; multiple MS/MS scans enumerate fragments for each peptide, increasing the information upon which identification may be based. Second, DIA provides fragment information from a greater diversity of peptides by multiplexing the MS/MS process. In DIA, a peptide is no longer dependent on producing an intense MS signal for its fragments to be measured. In the last several years, search engines designed to take advantage of these features have greatly improved the identification yield from DIA experiments (Pino et al. 2020).
Spectral Libraries from IP Inputs and SPION lysosome enrichment
The IP Inputs data set represented mouse gut extracts without antibody enrichment, and each subset experiment represented many replicates (21 Thermo RAWs for “A”, 24 for “C”, and 26 for “G”). The DDA sets held the advantage of LC-MS/MS experiments collecting spectra for 120 minutes, while the DIA experiments each collected spectra for 75 minutes (Quality metrics for all raw data appear in “Basic Quality Metrics,” Supplementary File 1). All six of the searches (DIANN-DIA, FragPipe-DDA, FragPipe-DIA, MaxQuant-DDA, Spectronaut-DDA, and Spectronaut-DIA) yielded substantial collections of peptides in their spectral libraries, from a low of 20108 distinct peptide sequences in MaxQuant on the “C” experiments to a high of 57677 distinct peptide sequences in Spectronaut using DIA data from the “G” experiments (see Supplementary Table 2 for a comparison of all spectral libraries for all algorithms in each experiment). We used the number of distinct peptide sequences in a library as its “diversity” because these values could be compared directly from the different search engine spectral library formats (represented as green cylinders in Fig. 1).
The peptide diversities of DIA-derived libraries mount a challenge to the assumption that DDA experiments are necessary to create spectral libraries for quantitation of DIA experiments. FragPipe produced more diverse spectral libraries for DDA than DIA in IP Inputs experiments “A” and “C”, but its DIA library was more diverse than its DDA library in experiment “G”. For Spectronaut, the DIA experiments led to more diverse libraries in “C” and “G”, but the DDA experiments for “A” gave more diverse libraries in Spectronaut. This seems like an equivocal result until instrument time is factored into the assessment. The DDA experiments required 60% more instrument acquisition time per LC-MS/MS run than did the DIA experiments, but they did not yield more diverse spectral libraries.
The intersections of these spectral libraries indicate which peptide sequences were shared among different combinations of these six spectral libraries for each of the three IP Input experiments. As shown in Fig. 2 for experiment “C”, the largest intersection set of peptides among these spectral libraries (10,855 peptide sequences) was universally identified among all six searches. Figure 2 illustrates a common trend among the experiments in that peptide sequences frequently are associated with only DDA libraries (4091 peptide sequences) or only DIA libraries (3868 peptide sequences). This common feature among the UpSet plots reinforces hypothesis B from the head of this section: some sets of peptide sequences have a propensity to being identified in either DIA or DDA experiments. Although the project did not investigate peptide-level intensities, it is likely that consistently identified peptides are also among the more intense signals in these experiments (Tabb et al. 2010). The UpSet plots for all experiments can be found in Supplementary Fig. 1.
The SPION experiments enriched lysosomes from mouse tissue using nanoparticles; as a result, the enriched samples contain far greater peptide diversity than would be typical of a co-IP. The DIA methods employed in SPION varied in duration from 120 to 300 minutes, and they emphasized MS/MS data for quantitation rather than relying on MS scans as does the “High-Resolution MS1” method employed for the IP Inputs. The long SPION LC gradients acquired nearly 200,000 MS/MS scans per RAW. DDA was only lightly used in SPION, collecting only a single 240-minute RAW for each of the three sub-experiments: Lysosome-Enriched Fraction, Liver Tissue Lysates, and Mouse Whole Cell Lysates.
The IP Inputs and SPION sets both show DIA-NN and Spectronaut producing very diverse spectral libraries from DIA experiments. Spectronaut libraries were more diverse than DIA-NN libraries in the IP Inputs, while SPION showed DIA-NN having the advantage over Spectronaut. Despite coming from a nano-particle enrichment rather than a lysate, the SPION experiments produced more diverse libraries than the IP Inputs, reaching a high of 135,775 distinct peptide sequences (DIA-NN in the SPION Mouse Whole Cell Lysates). Comparing the sizes of these libraries between different algorithms may ignore subtly different search spaces and different strategies for downstream peptide-spectrum match filtering. At early stages of this project, the very large DIA libraries produced by Spectronaut were viewed with some doubt by the authors, but subsequent confirmation of many of these additional peptides by use of DIA-NN alleviated much of this skepticism. The peptide diversities of spectral libraries for IP Inputs and SPION from MaxQuant were consistently lower than for other algorithms, even considering only those produced from DDA data.
Spectral libraries from co-immunoprecipitation
After the highly diverse spectral libraries of IP Inputs and SPION, the antibody-enriched spectral libraries seem very compact, with only the Bruker timsTOF DUBs set yielding any libraries above 10,000 distinct peptide sequences. This reduction of scale brought with it larger variability and greater prominence to contaminant proteins.
Handling of contaminant proteins is uneven among these four search engines (Frankenfield et al. 2022). For FragPipe (whether operated in DDA or DIA data), the FASTA database provided should already include contaminants, and each target sequence should be matched by a decoy. The FragPipe “Add decoys” button handled both tasks for this project. MaxQuant also adds contaminants and decoys, but these actions take place internally based on an uncontaminated target-only database provided by the user; its contaminants overlap with those of FragPipe, but many sequences are specific to each contaminant set. Spectronaut and DIA-NN do not appear to add contaminants, and they handle FDR filtering internally without explicit decoys.
The low peptide mass resulting from co-IP may cause bait and prey proteins to be supported by less peptide evidence than mass spectrometry-friendly contaminant proteins (here “contaminant” implies the protein is likely to have resulted from sample handling rather than to have non-specifically interacted with the antibody (Mellacheruvu et al. 2013)). For the SCIEX RACK1 D. melanogaster data set, for example, the protein with the largest number of distinct peptides in all FragPipe and MaxQuant libraries was P04264 (human keratin, type II cytoskeletal 1). Because Spectronaut and DIA-NN lacked contaminant sequences, they typically reported C7LA75 / P11147 (heat shock 70 kDa protein cognate 4) or P08736 (elongation factor 1-alpha 1) from D. melanogaster as the top hit instead because the D. melanogaster reference proteome does not contain human keratins. As in DDA database search, if the FASTA does not contain a protein sequence, DIA will not be able to identify or quantify it.
The RACK1 set was not alone in having prominent contaminant peptides. In the SCIEX ID4 M. musculus set, the first batch of DIA experiments delivered lower sensitivity than the second batch; as a result, P13645 (human keratin, type I cytoskeletal 10) accounted for the largest number of peptides in the ID4 pull-down spectral libraries for this batch. The top hits of the Bruker DUBs H. sapiens set appear to be the deubiquitylating enzymes targeted by their experiments. The co-IP data for LINE-1 ORF1p in H. sapiens cell lines introduced with this project, however, identified a range of keratins in both the mock co-IP controls and genuine co-IPs of the target protein: P04264, P05783, P05787, P13645, P35527, and P35908. The contaminants accompany rather than mask the abundant proteins of interest that interact with the LINE-1 ORF1p target. If researchers perform searches of non-human databases, they will want to ensure that the sequence database provided to Spectronaut and DIA-NN contain common human contaminants, but the tools do not also require decoy sequences to be added.
The RACK1 and ID4 experiments represent the performance of SCIEX “TripleTOF” instruments for co-IP. RACK1 (collected on a TripleTOF 5600) started with a positive outcome for MaxQuant, producing diverse spectral libraries for the “C-term” and “Ctrl” subsets, but attempts to use those libraries to quantify the DIA WIFF files in MaxDIA resulted in errors and no output tables. MaxDIA produced similar errors from the ID4 experiment, which employed the newer TripleTOF 6600. Because WIFF files are not supported natively in FragPipe, the DDA WIFFs were changed to mzMLs with the AB Sciex MS Data Converter [http://www.absciex.com/downloads/software-downloads] while SWATH WIFFs were converted to mzMLs in ProteoWizard msConvert. The SCIEX ID4 set is notable for including DIA experiments collected in two distinct batches, about three months apart, leading to a prominent batch effect (Čuklina et al. 2021). The second set of experiments produced far more diverse spectral libraries than did the first, with inventories growing by a factor of 3.8x to 7.1x (Fig. 3 shows overlaps within the second batch). This identification impact may reflect the variability of immunoprecipitation and/or instrument performance.
While most experiments appeared to offer similar sensitivity of identification between DIA and DDA experiments, the Bruker timsTOF DUBs experiments were a substantial exception. In all four inhibitors as well as the negative control and titration experiments, the DIA-derived spectral libraries were far more diverse in peptides than were the DDA-derived libraries (see Supplementary Table 2F and Supplementary Fig. 1F). FragPipe 21.1, used for this project, does not yet support direct identification of diaPASEF experiments (Meier et al. 2020), and so only DIA-NN and Spectronaut were able to take advantage of the high-quality DIA experiments for much larger spectral libraries. The set demonstrates that being able to identify DIA spectra directly can offer a considerable advantage in the set of peptides available for quantitation.
Sensitivity, missingness, and reproducibility for quantity tables
It is certainly possible that software could identify a peptide from MS/MS data without being able to quantify it. Most DIA software workflows distinguish between no viable chromatogram (resulting in a missing value or a reported 0 for a given precursor ion) and detected chromatograms (positive intensity is recorded). The tables that are most valued from DIA searches typically are those that report quantities for proteins, which requires some type of summarization for the precursor ion chromatograms that often take into account which peptide sequences are specific to a particular protein.
This project evaluated the protein quantity tables on a variety of bases. First, we compared the number of proteins identified from a set of experiments to the number of proteins with any quantity reported. Second we compared the number of proteins quantified in any experiment to the number of proteins identified in every experiment (no missing values). Because the intensity recorded for a protein bears a strong relationship to the variability of measurement (Oberg and Mahoney 2012), we separated the proteins with no missing values to five categories based on summed intensity. We could then compare the coefficient of variations for high-intensity, middle-intensity, and low-intensity proteins. With this analysis in place, it becomes possible to compare the missingness and reproducibility of quantitation between DIA and DDA experiments for selected experiments.
The nanoparticle-enriched (SPION) lysosome data illustrate the disparity between identified and quantified proteins. DIA-NN identified a spectral library of 10396 distinct proteins among the nine DIA RAWs for the LEF (lysosome-enriched fraction) cohort. The software reported quantities for 9023 of these proteins (87% of identified proteins). Of the 9023 quantified proteins, only 6718 (74%) had quantities reported for all nine experiments. This fraction is driven downward by the fact that three of the RAWs spanned 300 minutes, three spanned 180 minutes, and three spanned 120 minutes (labeled 240, 120, and 60 in filenames to represent the shallow gradient duration). As expected, the number of missing values for the shortest duration experiments (mean of 1615) is higher than for the middle duration experiments (mean of 568), and the middle duration experiments have more missing values than the longest duration experiments (mean of 168).
Co-IP for protein-protein interactions typically yields hundreds of proteins rather than thousands, and these diversities dip lower for negative controls. Mock immunoprecipitations and other types of negative controls may vary substantially from positive pull-downs in the sets of proteins they sample (Moresco et al. 2010) (see Supplementary Fig. 4). These sparse LC-MS/MS data create quite a challenge for these software workflows. In LINE-1 experiments, FragPipe failed to quantify any proteins in its output tables for the IgG control HRMS1 data, perhaps due to the low peptide concentration of these controls. DIANN occasionally warned that its machine learning models for retention times had too few peptides for proper training. Spectronaut frequently warned of too few peptides when quantifying in negative control samples. MaxDIA depends heavily upon MS2 chromatograms, damaging its quantitation performance in the HRMS1 experiments of the IP Inputs and of the LINE-1 co-IPs (see Supplementary Table 3 and Supplementary Text 2). The MaxQuant pipeline was unable to quantify the RACK1 and ID4 experiments because MaxDIA produced no output when presented with WIFF files (or their mzML equivalents). Co-IP experiments often rely upon comparing intensities of proteins in a positive pull-down to their intensities in a negative pull-down, but the latter category of sample is exactly where quantification software struggles most.
Coefficients of Variation (CVs) are computed by dividing the standard deviation of expression by the mean of expression. The R scripts created for this project winnowed out sets of “unanimous” proteins for which no missing values were reported among all experiments of a cohort. These sets were then separated into quintiles based on the sum of intensity reported for each protein. We expected that the quintile of highest-intensity proteins would have a smaller median CV of quantity than the mid-intensity proteins, and that the mid-intensity proteins would have a smaller median CV of quantity than the lowest-intensity proteins.
The protein quantity tables created by MaxDIA contained substantially higher CVs than the tables reported by other quantitation engines (see Supplementary Table 3 and Supplementary Fig. 3), so they were omitted from consideration of the intensity-CV relationship. Each quantitation of each cohort in each data set “voted” a TRUE if the CV values conformed to the hypothesis that highest-intensity proteins would have a lower CV than mid-intensity proteins and mid-intensity proteins had lower CVs than lowest-intensity proteins, or a FALSE if both these conditions were not met. If the CV values had no relationship to intensity, we would expect one in six data sets to randomly vote “TRUE.” The three Thermo data sets (IP Inputs, SPION, and LINE-1) produced 45 cases where the CVs were ranked as expected and 12 cases where they did not. The two SCIEX data sets (RACK1 and ID4) were also generally in agreement, giving 25 cases where the CVs were ranked as expected and 10 cases where they did not. Finally, the Bruker data set (DUBs) produced 15 cases where the CVs were ranked as expected and 9 where they were not. The IP Inputs and SPION data sets were considerably more diverse proteomes than the others, and only one quantitation effort of 27 (FragPipe using a DIA library on the “C” cohort of the IP Inputs) did not yield CVs in the rank order expected. It is possible that the greatly decreased proteomic diversity of co-IP experiments can also disrupt expected relationships between intensity and variance.
Spectronaut software offers a “library-free” quantification method named “directDIA.” In the IP Inputs G set, we tested the directDIA method from Spectronaut 19 versus the DIA-derived spectral library method in Spectronaut 18. The directDIA method quantified 6% more proteins in IP Inputs G, and the number of proteins quantified unanimously across all RAW files climbed by 4%. The CV values were slightly lower for directDIA high-intensity, mid-intensity, and low-intensity proteins. These minor differences may have resulted from the use of directDIA rather than a DIA-derived spectral library, or they may have resulted from changes in the more recent version of Spectronaut. In either case, it appears that the characterization of DIA-derived spectral libraries in Spectronaut can approximate expected directDIA performance.
Adjusting Lab Practices in Response
For the past few years, most DIA experiments on Thermo instruments at the Interfaculty Mass Spectrometry Center of UMCG have made use of the High-Resolution MS1 technique (Xuan et al. 2020), introduced in Thermo training workshops and evaluated through a variety of tests at UMCG. HRMS1 infers peptide intensities from peptide ion chromatograms rather than fragment ion chromatograms, ensuring a high rate of MS acquisition interspersed among the cycles of MS/MS windows. The specific variant of HRMS1 employed in the IP Inputs set included two FAIMS compensation voltages (CVs) and 88 MS/MS windows, and so software that incorporates fragment ion chromatograms alongside peptide ion chromatograms would find relatively few samplings of the fragments for a given peptide for this data set.
We sought to develop the “Variabele Vensters” (variable windows) method to boost the chromatographic resolution for fragment ions, using the 21 LC-MS/MS experiments of the IP Inputs Set “A” to represent a diverse proteome. We decided to include 30 windows in each cycle based on an estimated chromatographic peak width of 30 seconds and an MS/MS acquisition rate of 10 Hz. If each cycle of windows is collected in three seconds, a typical fragment chromatogram would be sampled in MS/MS ten times. We computed theoretical window boundaries that separated the peptide precursor m/z values in the IP Inputs Set “A” spectral library to 30 equal parts. Each of these theoretical window boundaries were rounded down (to give the next window start value) or rounded up (to give the previous window stop value) (See Table 2). This rounding gave two advantages: entering the method into the instrument control software was less error-prone, and successive windows overlapped by one m/z.
Table 2: The Variabele Vensters DIA method attempts to make each window the same number of identifiable peptide ions in width, using IP Inputs A DDA as a guide. After 1000 m/z, the number of identifiable peptides grows progressively less dense.
The LINE-1 data enabled comparison of spectral library diversity and CVs directly between DDA, HRMS1, and “VV” experiments. As Fig.
5shows, the co-IPs for LINE-1 in both HEK293T and N2102Ep cell lines provided the lowest CVs for high-intensity proteins and the highest CVs for low-intensity proteins, whether HRMS1 or VV methods were used (in this figure, spectral libraries were always derived from the data where they were applied for quantitation). Evaluating the missingness of HRMS1 and VV for the positive LINE-1 co-IPs reveals no clear “winner:” Spectronaut and FragPipe identified more proteins with no missing values in VV experiments for both cell lines, while DIA-NN identified more proteins without missing values in HRMS1 experiments for both cell lines. For the N2102Ep cell experiments, the VV experiments produced lower median CV values, while the HEK293T experiments were a mixed bag between VV and HRMS1 methods. The VV experiments were configured to use the same 60 minutes of instrument time as the DDA experiments, giving it a throughput advantage. VV and HRMS1 come from very different DIA design paradigms, and yet both seem to pair well with co-IP.The availability of DDA, HRMS1 (MS1-based DIA), and VV (MS2-based DIA) for the same samples of the LINE-1 experiment make it possible to return to the question that animated this study:
does switching from DDA to DIA measurement improve the information yield from co-IP experiments? FragPipe was able to perform a Match Between Runs analysis in IonQuant for the DDA experiments in its “LFQ-MBR” workflow and performing both identification and quantitation in the HRMS1 and VV experiments in its “DIA-SpecLib-Quant” workflow. The combined_protein.tsv file reported protein intensities in both “Intensity” and “MaxLFQ Intensity” columns (Yu et al.
2021). Because the MaxLFQ Intensity columns contained more missing values, we employed Intensity columns instead.
The DDA experiments quantified fewer proteins without missing values and produced higher CVs than the DIA experiments. The DDA experiments for LINE-1 pull-downs quantified 260 proteins with no missing values in HEK293T cells and quantified 293 proteins with no missing values in N2102Ep cells. The HRMS1 values were 348 and 378, respectively, while the VV experiments quantified 356 and 398 proteins unanimously. For the most intense quintile of proteins in the LINE-1 pull-downs, the CV values for all three types of instrument methods were excellent, ranging from 0.077 to 0.126. For middle intensity proteins the CVs ranged from 0.088 to 0.187. For the least intense quintile of proteins in the LINE-1 pull-downs, the CV values were consistently higher, ranging from 0.146 to 0.218. In every case, the highest CV value came from the DDA experiment for the LINE-1 pull down in HEK293T, with the DDA experiment for N2102Ep cells being next highest. The negative control IgG experiments in these two cell types are where the problematic performance takes place. Median CV values as high as 0.714 were produced from DDA experiments in HEK293T with the IgG pull-down. FragPipe DIA analysis failed in the negative control HRMS1 experiments for both cell lines, producing tables containing zero proteins. The low signal-to-noise environment of negative control IPs is a challenging one for both DDA and DIA quantitation, and research teams should gain familiarity with at least two different software workflows to have a fallback when one algorithm fails on a set.