Conﬂuence of Artiﬁcial Intelligence and High Performance Computing for Accelerated, Scalable and Reproducible Gravitational Wave Detection

Finding new ways to use artiﬁcial intelligence (AI) to accelerate the analysis of gravitational wave data, and ensuring the developed models are easily reusable promises to unlock new opportunities in multi-messenger astrophysics (MMA), and to enable wider use, rigorous validation, and sharing of developed models by the community. In this work, we demonstrate how connecting recently deployed DOE and NSF-sponsored cyberinfrastructure allows for new ways to publish models


Introduction
Gravitational waves were added to the growing set of detectable cosmic messengers in the Fall of 2015 when the advanced LIGO detectors reported the observation of gravitational waves consistent with the collision of two massive, stellar-mass black holes [1,2].Over the last five years, the advanced LIGO and advanced Virgo detectors have completed three observing runs, reporting over 50 gravitational wave sources [3,4].
Significant improvements in the sensitivity of the advanced LIGO and advanced Virgo detectors during the last three observing runs have increased the observable volume they can probe, thereby increasing the number of gravitational wave observations [4].As these observatories continue to enhance their detection capabilities, and other detectors join the international array of gravitational wave detectors, it is expected that gravitational wave sources will be observed at a rate of several per day [4,5].
An ever-increasing catalog of gravitational wave sources will enable systematic studies that will refine and advance our understanding of stellar evolution, cosmology, alternative theories and gravity, among others [6][7][8][9][10][11].The combination of gravitational and electromagnetic waves, and cosmic neutrinos, will shed revolutionary insights into the nature of supranuclear matter in neutron stars [12][13][14] and the formation and evolution of black holes and neutron stars, providing new and detailed information about their astrophysical environments [15][16][17][18].
While all of these science goals are feasible in principle given the proven detection capabilities of astronomical observatories, it is equally true that established algorithms for the observation of multi-messenger sources, such as template matching and nearest neighbors, are compute-intensive and poorly scalable [19][20][21][22][23]. Furthermore, available computational resources will remain oversubscribed, and planned enhancements will be rapidly outstripped with the advent of next-generation detectors within the next couple of years [24,25].Thus, an urgent re-thinking is critical if we are to realize the Multi-Messenger Astrophysics program in the big-data era [26][27][28].
To contend with these challenges, a number of researchers have been exploring the application of deep learning and GPU-accelerated computing.Coauthors of this article pioneered the use of deep learning and high performance computing to accelerate the detection of gravitational waves [29][30][31].While the first generation of these algorithms targeted a shallow signal manifold (the masses of the binary components), and only required tens of thousands of modeled waveforms for training, these models served the purpose of demonstrating that an alternative method for gravitational wave detection was indeed possible, since fully trained neural networks are both as sensitive as template matching and significantly faster, at a fraction of the computational cost.
Research and development in deep learning is moving at an incredible pace .Specific milestones in the development of AI tools for gravitational wave astrophysics include the construction of neural networks that describe the same 4-D signal manifold of established gravitational wave detection pipelines, i.e., the masses of the binary components and the z-component of the 3-D spin vector: (m 1 , m 2 , s z 1 , s z 2 ).This requires the design of sophisticated neural network models, and the use of distributed training algorithms on high performance computing platforms to train neural networks with millions of modeled waveforms in a reasonable amount of time [44].Another milestone concerns the creation of neural networks that enable gravitational wave searches over hour-long datasets, keeping the number of misclassifications at a minimum [53].
In this article, we introduce a deep learning ensemble, designed to cover the 4-D signal manifold (m 1 , m 2 , s z 1 , s z 2 ), to search for and find binary black hole mergers over the entire month of August 2017 advanced LIGO data [54].
Our findings indicate that this approach clearly identifies all black hole mergers contained in that data batch with no misclassifications.
To conduct the aforementioned analysis, we used the Hardware Accelerated Learning (HAL) cluster deployed and operated by the Innovative Systems Lab (ISL) at the National Center for Supercomputing Applications (NCSA).This cluster consists of 16 IBM SC922 POWER9 nodes, with four NVIDIA V100 GPUs per node [55].The nodes are interconnected with EDR InfiniBand network and the storage system is made of two DDN all-flash arrays with SpectrumScale file system, providing 250 TB of usable space.Job scheduling and resources allocation are managed by SLURM.
As we show below, we can process the entire month of August 2017 with our deep learning ensemble in just seven minutes using all 64 of HAL's NVIDIA V100 GPUs.In addition to taking this significant step forward in the use of deep learning for accelerated gravitational wave searches, we also demonstrate that we can share these models with the broader community by leveraging the Data and Learning Hub for Science (DLHub) [56,57].We believe that this approach will accelerate the adoption and further development of deep learning for gravitational wave astrophysics.
Given that DLHub has the ability to both archivally store and actively run trained models, it provides a means to address reproducibility, reuse, and credit.With sufficient computational resources (HAL in this case) and a connection made via funcX, a function-as-a-service platform, having a model on DLHub allows the inference or analysis done in a published paper to be reproduced given the original data is also available (as described in §4), and if applied to new data instead, the model can then be reused, with DLHub's registration providing a means for the developers of the model to receive credit by the users citing the archived model.This also allows users to easily experiment with trained models, even from one discipline to another.
This paper brings together several key elements to accelerate deep learning research.We showcase how to combine NSF-and DOE-funded cyberinfrastructure to release state-of-the-art, production scale, neural network models for gravitational wave detection.The framework DLHub → funcX → HAL provides the means to enable open source, accelerated deep learning gravitational wave data analysis.This approach will empower the broader community to readily process open source LIGO data with minimal computational resources.Going forward, this approach may be readily adapted to demonstrate interoperability, replacing HAL with any other compute resource.
At a glance, the novel developments for AI-driven gravitational wave detection introduced in this article encompass: • Open Source The AI models we introduce in this study are shared with the broader community through DLHub.This approach will streamline and accelerate the development of AI algorithms for gravitational wave astrophysics • Reproducible We present results of two independent analyses to test the predictions of our AI models, and confirm that the output of these studies is consistent and reproducible • Accelerated We use 64 NVIDIA GPUs to process the entire month of August 2017 advanced LIGO data in just sevent minutes, finding all expected gravitational wave events in that dataset, and reporting no misclassifications • Scalabe We establish that AI-driven gravitational wave detection scales strongly as we increase the number of GPUs used for inference This is the first analysis of its kind in the literature that combines all these elements.This article is organized as follows.§2 describes the AI ensemble used for this analysis, including the modeled waveforms and open source advanced LIGO noise used to train these models.It also provides a brief description of the approach used to combine the output of the AI models in the ensemble to identity real events, or otherwise noise triggers associated with loud noise anomalies.§3 presents results for the analysis of open source advanced LIGO data using our AI ensemble in HAL.In §4 we describe how we repeated the analysis described in §3 using the AI ensemble deployed in DLHub and using funcX to access HAL computational resources.This study marks an important milestone in the development of open source gravitational wave detection with accelerated and reproducible AI models.We summarize our findings and outline future directions in §5.

AI Ensembles for Accelerated Gravitational Wave Detection
Each model in the ensemble consists of two independent modified WaveNets [58] processing Livingston and Hanford strain data sampled at 4096Hz.The two outputs are then concatenated and jointly fed into a final set of two convolutional layers which output a classification (classes: noise or waveform) probability for each time step.
For training we use the final second of 4096 Hz modeled waveforms generated with SEOBNRv3 [59] covering the parameter space with total masses M ∈ [5M , 100M ], mass-ratios q ≤ 5, and individual spins s z {1,2} ∈ [−0.8, 0.8].These modeled waveforms are then whitened and linearly mixed with advanced LIGO noise obtained from the Gravitational Wave Open Science Center [54].Specifically, we use the three 4096s long noise data segments starting at GPS times 1186725888, 1187151872, and 1187569664.The ground-truth labels for training are encoded such that each time-step after the merger is classified as noise, and all preceding time-steps in the 1 second long window are classified as waveform strain.Hence, the transition in classification from waveform to noise identifies the location of the merger.
At test time on advanced LIGO strain data, we employ a post-processing function to precisely locate such transitions in the model's output.Specifically, we use a 1 second window on the strain data, with a step size of 0.5 seconds, and use off-the-shelf peak detection algorithm find peaks, provided by SciPy, on the models probability output.In the find peaks algorithm, we specify the thresholds so that only peaks with a width within [0.5, 2] seconds.Since the timestep for the sliding window is 0.5 seconds, we merge any repeated detections, i.e., peaks within 0.5 seconds of each other are counted as repeated detection.
Once we have the predicted locations of the peaks/mergers from each of the four (randomly initialized and trained) models in the ensemble, we combine them in one final post-processing step so that all peaks that are within 1/128 seconds of each other are flagged as detection of true gravitational-wave events, while the rest are discarded as random False Alarms.This workflow is shown in Figure 1.

Accelerated AI Gravitational Wave Detection in HAL
We used the AI ensemble described in the previous section to process advanced LIGO open source data, both from Livingston and Hanford, that cover the entire month of August 2017.We chose this month since it contains several gravitational wave sources, and thus it provides a test bed to quantify the sensitivity of our AI model to identify real events, and to estimate the number of false positives over extended periods of time.
The results we present below were obtained by directly running our AI ensemble in HAL. Figure 2 presents two panels that summarize the speed and sensitivity of our approach.The top panel shows that when we distribute the inference over all 64 NVIDIA V100 GPUs in HAL, we can complete the search over the entire month of August 2017, including post-processing of the noise triggers identified by each model in the ensemble, within just seven minutes.The bottom panel shows that our AI ensemble identifies all four binary black hole mergers contained in this dataset.We follow up two of these gravitational wave sources in Figure 3.The top and middle rows present spectrograms (left  column) and the response of one of the AI models in the ensemble to these real events (right column).This is the first time an AI approach achieves this level of sensitivity and computational performance over long stretches of real advanced LIGO data.This is also the first accelerated, reproducible AI gravitational wave search at scale that covers the 4-D signal manifold that describes quasi-circular, spinning, non-precessing binary black hole mergers.

Connecting DLHub to HAL through funcX
We exercise the ensembles through DLHub [57], in order that the models be widely available and our results reproducible.DLHub is a system that provides model repository and model serving facilities, in particular for machine learning models with science applications.DLHub itself leverages funcX [60], a functionas-a-service platform that enables high performance remote function execution in a flexible, scalable, and distributed manner.A funcX endpoint is deployed to HAL and registered with the funcX service; an endpoint consists of a funcX agent, a Manager, and a Worker, and abstracts the underlying computing resource.The endpoint dynamically provisions HAL nodes and deploys Workers to  perform inference requests.This computational infrastructure is schematically shown in Figure 4.
DLHub packages the ensemble of models and the inference function as a servable, i.e., builds a container that includes the models and their dependencies and registers the inference function with the funcX registry.An inference run corresponds to a funcX function invocation, which triggers the dispatch of the task to the HAL funcX endpoint.On every such run (i.e., per collection of data, not per sample) the funcX agent deployed to HAL allocates the task to multiple nodes and managers (one manager per node), each of which then forwards the task to a single worker, which finally distributes the task across the GPUs in a MISD fashion (i.e., one model per GPU, with all models operating on the same  data).The models then perform their individual forward passes and the workers aggregate their results.Note that distinct managers, and therefore workers on distinct nodes, operate on non-overlapping chunks of data.Figure 5 shows that scaling the inference analysis through the DLHub architecture is equivalent to directly running the analysis on the HAL cluster through distributed inference.
In both instances, we succeeded at using the entire HAL cluster optimally.
Using this computational infrastructure, we reproduced the results presented in the previous section.Not only that, we also found that the computational performance of the DLHub architecture provides the same throughput as running the AI ensemble directly in HAL.The throughput of the DLHub+HAL architecture is similar to the speedup obtained by directly distributing the inference analysis in the HAL cluster.An AI ensemble of four neural networks processes entire month of advanced LIGO data, August 2017, in 7 minutes using 64 NVIDIA V100 GPUs that are evenly distributed in 16 nodes in the HAL cluster.
Currently the system is set up only for post-processing of data, which provides the required framework for accelerated, off-line analyses.We are working to extend DLHub and funcX to support streaming data by using Globus [61].This approach is laying the foundations for future scenarios in which advanced LIGO and other astronomical observatories broadcast real-time data to the broader community.Such an approach would enable researchers to carry out analyses with open source data that are beyond the scope of scientific collaborations, but that are essential to push the frontiers of Multi-Messenger Astrophysics.

Conclusions
Research and development of AI models for gravitational wave astrophysics is evolving at a rapid pace.In less than four years, this area of research has evolved from disruptive prototypes [29][30][31] into sophisticated AI algorithms that describe the same 4-D signal manifold as traditional gravitational wave detection pipelines for binary black hole mergers, namely, quasi-circular, spinning, nonprecessing, binary systems [53]; have the same sensitivity as template matching algorithms; and are orders of magnitude faster, at a fraction of the computational cost.
There is also a vigorous program to apply AI to accelerate the detection of binary neutron stars [41,42], and to forecast the merger of multi-messenger sources, such as binary neutron stars and neutron star-black hole systems [51,52].
At this time, AI models have been proven to effectively identify real gravitational wave signals in advanced LIGO data, including binary black hole and neutron stars mergers [41,42,51,53].The current pace of progress makes it clear that the broader community will continue to advance the development of AI tools to realize the science goals of Multi-Messenger Astrophysics.
Furthermore, mirroring the successful approach of corporations leading AI innovation in industry and technology, we are releasing our AI models to enable the broader community to use and perfect them.This approach is also helpful to address healthy and constructive skepticism from members of the community who do not feel at ease using AI algorithms.
This article also demonstrates how complementary communities can work together to harness DOE-and NSF-funded cyberinfrastructure to enable open source, accelerated, and reproducible AI-driven compute-intensive analysis in record time.This approach will facilitate a plethora of new studies beyond gravitational wave astrophysics, since DLHub and funcX are discipline and hardware agnostic.

Figure 1 :
Figure 1: Gravitational wave detection workflow with artificial intelligence ensembles.Four independent neural network models process concurrently Livingston and Hanford data faster than real-time.Their output is then post-processed to identify significant triggers that are present in the output of all neural network models.

Figure 2 :
Figure2: Top panel: Speedup in inference using our AI ensemble in the HAL cluster.The search for gravitational waves in the Hanford and Livingston data channels over the entire month of August 2017 is completed within seven minutes.Bottom panel: Our AI ensemble clearly identified all four binary black hole mergers previously reported in these data, with no false positives.

Figure 3 :
Figure 3: Left panels Normalized L-channel spectrograms around positives identified by our AI ensemble.Right panels Detection output of one of our deep learning models in the ensemble.

Figure 4 :
Figure 4: Architecture of the DLHub architecture used to conduct accelerated and reproducible gravitational wave detection on open source advanced LIGO data.This architecture provides a command line interface (CLI), a Python Software Development Kit (SDK) and a REST API to publish, manage and invoke AI models.

Figure 5 :
Figure5: The throughput of the DLHub+HAL architecture is similar to the speedup obtained by directly distributing the inference analysis in the HAL cluster.An AI ensemble of four neural networks processes entire month of advanced LIGO data, August 2017, in 7 minutes using 64 NVIDIA V100 GPUs that are evenly distributed in 16 nodes in the HAL cluster.