Real-Time 3D Analysis During Tomographic Experiments using tomviz

The demand for high-throughput electron tomography is rapidly increasing in biological and material sciences. How-ever, this 3D imaging technique is computationally bottlenecked by alignment and reconstruction which runs from hours to days. We demonstrate real-time tomography with dynamic 3D tomographic visualization to enable rapid interpretation of specimen structure immediately as data is collected on an electron microscope. We show volumetric interpretation can begin in less than 10 minutes and a high-quality tomogram is available within 30 minutes. Real-time tomography is integrated into tomviz , an open-source and cross-platform 3D data analysis tool that contains intuitive graphical user interfaces (GUI), to enable any scientist to characterize biological and material structure in 3D.

The demand for high-throughput electron tomography is rapidly increasing in biological and material sciences. However, this 3D imaging technique is computationally bottlenecked by alignment and reconstruction which runs from hours to days. We demonstrate real-time tomography with dynamic 3D tomographic visualization to enable rapid interpretation of specimen structure immediately as data is collected on an electron microscope. We show volumetric interpretation can begin in less than 10 minutes and a high-quality tomogram is available within 30 minutes. Real-time tomography is integrated into tomviz, an open-source and cross-platform 3D data analysis tool that contains intuitive graphical user interfaces (GUI), to enable any scientist to characterize biological and material structure in 3D.

Main
Three-dimensional (3D) characterization across the nanoscale is now possible using scanning / transmission electron microscopes (S/TEM) [1][2][3][4][5]. In an electron tomography experiment, volumetric structure of biological or materials specimens are reconstructed from high-resolution projection images acquired across many viewing angles [6,7]. Unfortunately, tomographic reconstructions can take one to several days to complete depending upon the dataset size or algorithm(s) employed. Even worse, the reconstruction occurs offline, long after all the data has been collected, preventing immediate interpretation during an ongoing experiment. While advancements in detector hardware have boosted throughput with digital data collection [8], substantial human effort and computational resources are still required to process the raw data before visualization. It has been a longstanding goal to begin 3D analysis of specimens in real-time to allow immediate assessment of nanoscale structure and data quality [9].
Here we present facile 3D visualization of specimens during an electron or cryo-electron tomography experiment using the tomviz platform (tomviz.org). Our platform now provides interactive 3D material or biological structure in real-time to enhance high-throughput specimen interpretation. Tomviz offers multiple real-time reconstruction algorithms integrated into a fully graphical interface that presents the user with immediate visualization during data collection. Achieving high-throughput electron tomography requires an integrated pipeline that links the microscope hardware to optimized reconstruction algorithms and efficient 3D volumetric visualization. A multi-threaded data analysis pipeline runs dynamic visualizations that update as new data is collected or reconstruction algorithms proceed. Scientists can interactively analyze 3D specimen structure concurrent with a tomographic reconstruction after or during an experiment. The robust graphical interface allows for 3D specimens to be rendered as shaded contours or translucent volumes that can be rotated, cropped, or sliced as the reconstruction occurs. In favorable cases, structural interpretation can begin as early as 10 minutes and a high-resolution volume is available after only 60% of data is acquired (∼30 minutes). The latest tomviz release (v 2.0), is now packaged with real-time 3D analysis for electron tomography, is available as an open-source cross-platform tool with compiled binaries certified for Linux, Mac, and Windows.

Results
Real-Time Tomography Workflow. The real-time tomography workflow is illustrated in Figure 1: electron micrographs are collected, passed to tomviz for reconstruction, and visualized as an interactive 3D rendering. This process runs simultaneously and continuously while the electron microscope is being operated. During experimental acquisition, tomviz monitors when new projections are collected (Fig. 1a) and appends new data into the reconstruction process. Importantly, tomograms are reconstructed in parallel with data acquisition. The intermediate reconstructions are rendered in 3D and immediately presented to the scientist (Fig. 1b). Thus, the tomogram dynamically improves with time as both the reconstruction algorithm converges and additional specimen information arrives. High-quality 3D reconstructions are available before the end of the experiment (Fig.  1c).
Early Insight into 3D Structure. Direct visualization of a specimen's 3D structure enables immediate identification of morphological and internal information shortly after a tomography experiment begins. We demonstrate real-time tomography on a helical nanoparticle comprised of a chiral dipeptide Cysteine amino-acid coordinated with Cadmium (Cys/Cd) (Supplemental Video S1, S2). These semiconducting nanoparticles contain strong tunable chiroptical properties due to a twisted geometry [10]. As shown in Figure 1b, the overall morphology for the Cd/Cys nanoparticle can be observed in as early as 10 minutes and fine details are visible after 20-30 minutes of the experiment (roughly half-completion). The specimen's right handed chirality cannot be determined from a single projection image and requires 3D imaging (Fig. 1c). With real-time tomography the material's chirality and symmetry were identified within the first third of data acquisition (∼15 minutes). This immediate feedback can save researchers days of effort as reconstructions are no longer processed offline. Moreover, real-time visualization allows quick adjustment and optimization of reconstruction parameters that can greatly influence the reconstruction quality. Ultimately, scientists can efficiently investigate 3D nano-structure during imaging to guide experiments and redefine scientific objectives while simultaneously operating the microscope.

Real-time 3D Visualization During Reconstruction.
Currently, the best tomographic reconstructions are obtained from algorithms that are slow and iterative. In practice, electron tomography experiments are limited by a finite and restricted angular range (e.g. < ±70 • ) resulting in incomplete information that degrades resolution in 3D [11]. Iterative algorithms can recover tomograms with high spatial resolution and minimal reconstruction error [12]. While these algorithms better estimate 3D structure from under-determined measurements, they come at the expense of computational time [13]. Fortunately, using the tomviz tool, iterative reconstructions can be visualized in realtime throughout the arduous computation (Supplemental Video S2).
Real-time tomography greatly alleviates the wait-time by visualizing the intermediate 3D structure between algorithm iterations-beneficial during an experiment or analysis. Figure 2 demonstrates interactive visualizations of the Simultaneous Iterative Reconstruction Technique (SIRT) [14] for a cobalt phosphide (Co 2 P) hyperbranched nanoparticle [15] (512 3 pixels volume reconstructed across the 363.52 nm full field of view). SIRT tomograms begin with a loose estimate [16] (Fig. 2a) and develop sharper, high frequency information with each increasing iteration ( Fig. 2b-c, Supplementary Video S4). Compressed sensing algorithms such as total variation minimization (TVmin) seek maximally sparse solutions to recover high-resolution, low-noise structure using fewer projections than conventional methods [17,18]. Figure 2e-f and Supplementary Video S5 demonstrates an interactive 3D visualization using TVmin reconstruction of a iron platinum (FePt) nanoparticle at atomic resolution (256 3 pixels volume reconstructed across the 9.536 nm full field of view)data provided and pre-processed by Yang et. al. [19]. Recent developments in dynamic compressive sensing [20] have also been incorporated into tomviz to accommodate the arrival of new projections during an experiment.
In addition to early estimates of specimen structure, real-time tomography allows assessment of the reconstruction convergence. This is observed qualitatively in the 3D visualization (Fig 2e-f)

Fig.2 Demonstration of iterative reconstruction algorithms. a-c,Visualization
of the Co 2 P nanoparticle early, mid, and at the end of the reconstruction process. At the beginning, the underlying structure can partially be seen behind the excess of background intensity. In the middle of the process, sharp features begin to form. The final iteration converges to a tomogram visually similar to the input tilt series. Scale bar, 50 nm. e-g, Visualization of an atomic resolution FePt nanoparticle. The atoms in the TV nanoparticle are resolved with increasing iteration and its periodicity demonstrated with the fast Fourier transform (FFT). Scale bar, 1 nm. d, h, A plot of the normalized residual to demonstrate convergence. and quantitatively plotted in the residuals (Figure 2d,h). Realtime assessment of convergence can further save researchers time by providing early feedback to determine optimal reconstruction parameters. Coupled with our live visualization tools, users can adaptively optimize the reconstruction accuracy.
Alternatively, weighted back projection (WBP) reconstructions are ideal for quick assessment of specimen morphology due to their fast, non-iterative computation [21,22]. Figure 3 shows screenshots taken from a live WBP reconstruction visualized using tomviz-time proceeds from left to right. Figure 3a is a tomogram of gold (Au) nanoparticles on a strontium titanate (STO) nanocubes. Figure 3b shows platinum (Pt) nanoparticles on a carbon (C) support with the rotation axis along the x-direction. For WBP of single-axis tomography, partial volumetric updates are provided slice by slice along the direction parallel to the rotation axis. In the software, the 3D visualizations dynamically grow along one direction throughout the computation. Supplemental Video S6 shows the user experience of a WBP reconstruction emerging over just a few minutes.
The Live Tomography Software. The latest tomviz release (v. 2.0) includes real-time tomography capabilities, is entirely opensource (BSD License), runs on all operating systems (OSX, Windows, Linux) with certified installers, and can be implemented on rudimentary TEMs available at most institutions. A user manual with step-by-step instructions for implementing real-time tomography is provided as a Supplementary Protocol along with Supplemental Video demonstrations.
The tomviz graphical user interface (GUI) (Fig. 4) provides an intuitive tomography tool that allows scientists to focus on 3D specimen interpretation. Tomviz monitors data directories for the arrival of new projections during an experiment (Fig. 4 a) and visualizes the 3D reconstruction as it dynamically updates (Fig.  4 c). During a real-time tomographic reconstruction users can zoom, rotate, slice, and segment the object to highlight regions of interest as the algorithm runs independently. Each voxel in the 3D render (i.e. volumes or isometric contours) are assigned a color and opacity controlled by the color-opacity transfer function overlayed on the histogram visible at the top of the GUI (Fig. 4, top-right). Users can intuitively define voxel transparency by selecting points on the curve and dragging it between transparent or opaque. The data 'Pipeline' retains all transformations and modules performed to produce visualizations, all of which can be saved in a state file for sharing and reproducibility.
A seamless user experience is enabled by an underlying multithreaded framework of Python / C++ interactions. As the reconstruction occurs, algorithms written in Python trigger signals to notify the application that a new volume is available (Methods). Tomographic reconstructions can either run on basic computer infrastructure found on any laptop or scaled across multiple GPUs to process large volumes (> 1024 3 voxels). Live reconstructions without performance degradation requires a tripling of memory requirements. One data copy resides on the visualization (GUI) thread, another on the reconstruction (Python) thread, plus a temporary copy for efficient staging and handoff. The temporary copy allows the reconstruction to run unhindered during the handoff process. The total memory usage for real-time reconstruction is usually well within a consumer grade computer (c.a. 0.4 -16 GB). After the reconstruction is complete, all copies are released from memory and only the final reconstruction remains. Analytical reconstruction methods such as WBP can run sliceby-slice with new reconstructed slices appended along a single reconstruction direction. For iterative methods, we recommend updating the entire volume either every iteration or every few (depending on the speed of computation)-especially for complex sampling schemes such as dual or multiple-axis tomography which lacks a single rotation axes. Enhancements to the underlying 3D rendering (VTK) within tomviz were made to improve interactive visualization and analysis throughout the reconstruction process [23].

Discussion
We demonstrate real-time visualization of electron tomography reconstructions as they proceed during or after an experiment using tomviz, an open-source cross-platform tool compatible with all electron microscopes. In the actual software, the 3D visualizations are dynamically updated in parallel with computation. This means that scientists need not wait for a reconstruction to complete, or all data to be collected before beginning the interpretation of results. Continuous feedback provides high-throughput and early diagnoses of 3D specimens, opportunities to optimize experimental parameters, or investigate multiple regions of interest. Optimized, threaded pipelines and the iterative nature of tomographic methods allows tomviz to show intermediate results with minimal impact on performance. This enables interactive 3D analysis of the current reconstruction state while the reconstruction proceeds on a separate thread. A robust graphical interface allows objects to be rendered as shaded contours or volumetric projections and these objects can be rotated, cropped, or sliced. This capability opens radically new possibilities for developing high-throughput, real-time tomographic reconstruction algorithms for materials or biological applications. Ultimately, interactive real-time visualization goes beyond high-throughput and allows researchers to make early judgments to answer or identify new scientific questions during experimentation.

Methods
Installing Tomviz: Tomviz binary installers are available at tomviz.org for macOS, Linux, or Windows [24]. A user manual for real-time tomography using tomviz is provided as Supplemental Material.

Source-Code Availability:
In addition to stable binary releases, the latest experimental builds and source code is available at github.com/OpenChemistry/tomviz. The application is primar-ily developed in C++ using CMake to coordinate the build process. Algorithms for electron tomography are primarily written using Python to offer facile in-app readability and modification.
License: Tomviz and its underlying tomography algorithms are developed openly and freely as open source software under the 3-clause BSD License [25]-an Open Source Initiative approved license. This allows for unrestricted academic, commercial, and government use with no obligation on the part of the licensee to distribute the source code. This license encourages the widest possible re-use of the source code.

Specimen Synthesis and Preparation:
Cys / Cd helical nanoparticles are self-assembled via electrostatic and coordination interactions between positively charged cadmium ions and negatively charged chiral dipeptide cysteine. The helical nanoparticles were mixed in an aqueous solution and drop cast using a micropipet onto a 3mm copper TEM grid dried at room temperature. The TEM grid was an ultrathin (3 nm) carbon film with a large hexagonal mesh (100) to provide high specimen tilts without beam shadowing (Electron Microscopy Sciences, Hatfield, PA, USA). Specimen preparation for the Co 2 P, C/Pt [26], FePt, [19], and Au/STO [27] datasets are available in the cited manuscripts with data descriptors.
Electron Tomography Acquisition: Real-time electron tomography of the Cys/Cd helical nanoparticle (Fig. 1) was performed during experimental image acquisition on a Talos F200X (Thermo Fisher) operated at 200kV with a 10.5 mrad semiconvergence angle using an annular dark field detector with an inner collection angle of 36 mrad. The projections were recorded from -64°to +71°with a +1°angular increment using a Model 2021 Fischione Analytical Tomography Holder. At each tilt angle, a STEM image with a 4 µs dwell time at each pixel of lateral dimension 2.47 nm. The tilt series for Figures 2-4 were collected and aligned in advance of the real-time reconstruction. The FePt nanoparticle (Fig. 2f) was collected on a FEI Titan at 300 keV with a 30 mrad convergence angle and 1.5°tilt increment. [19]. The Co 2 P (Fig. 2c), Au/STO (Fig. 3a) and C/Pt (Fig. 3b) [26] nanoparticles [27] were acquired on a FEI Tecnai F20 at 200 keV and 2°, 2°and 1°tilt intervals, respectively. Additional experimental information are available in the corresponding references for each dataset.
A user manual with step-by-step instructions for implementing real-time tomography is provided as a Supplementary Protocol along with supplemental video demonstrations. As with any tomography experiment, microscope alignment is critical. In particular, the sample should be eucentric to alleviate specimen drift and the need for any stage refinement during acquisition. After the microscope is aligned, a user defines the data directory tomviz will monitor to bridge the pipeline from data acquisition to the 3D reconstruction and visualization. Because tomviz operates independently from the microscope acquisition control, this realtime tomography tool can run on any TEM and users can choose their preferred acquisition programs (e.g. Nion Swift [28], Digital Micrograph, FEI Velox, SerialEM [29]).
File Formats. Data stored as raw binaries, XDMF, HDF5, text, png, SER, DM3/4, or TIFF can be read into tomviz. This includes 32-bit IEEE floats. Users can save visualizations and computations as state files (.tvsm) to reproduce results and be shared among colleagues. Reconstructions can be exported into file formats compatible with dedicated 3D rendering software (e.g. Blender).

Data Processing.
Tomographic experiments require identifica-tion of the center of rotation in the projection tilt series, otherwise artifacts will be introduced into the tomogram [30]. Even after aligning the stage to eucentricity, the rotation axis can be offset from center and often require additional processing. When an object is tilted around the rotation axis, the object's center of mass (CoM) forms a circle and and coincides with the origin of the perpendicular axis. To determine the CoM we projected each projection onto the perpendicular axis and calculated its shift: is the Coulomb potential at position x i [4]. This method is known to be sensitive to noise, so prior to aligning the projections we performed a background subtraction to account for the sample support (lacey carbon) and increasing thickness from high tilts.
Successful tilt axis identification with center of mass alignment requires the total projected volume to be fixed for each projection [31] and objects to be isolated. In cases where either of this requirements are not met (e.g. slab specimen geometries or fields of view where multiple particles are visible), alternative alignment routines should be considered. Further tilt axis refinement can be selected with our automated identification script. More advanced iterative projection matching alignment routines can be utilized near the end of data collection to improve the tomogram resolution [32].

Real-time Reconstruction Algorithms during Experimental Acquisition.
Modifications to the common implementation for SIRT and TVmin were made to account for the dynamic addition of input projections throughout an experiment. SIRT seeks the minimal error between the reconstruction and experimental data: arg min x Ax − b 2 where A is the measurement matrix, b are the experimental projections and x is the tomogram. We can further regularize the process through the assumption that our volumes should be piece-wise smooth and minimize its total variation x T V . Iterative algorithms require rescaling of the descent parameter based on the number of projections sampled. SIRT can easily estimate the descent parameter through calculation of the Lipschitz constant (L = A T A 2 ). The Lispchitz constant can be estimated by using the power method [33]. The descent parameter for TVmin is scaled by a dampening envelope that ensures its magnitude decays linearly [20]. Non-iterative algorithms such as WBP do not require rescaling of descent parameters and simply needs to reinitialize the computation with the new projection images collected.
GUI Parallelization for Real-Time Visualization. The multithreaded pipeline within the tomviz application executes longrunning jobs while simultaneously offering real-time visualization of the progress. As the reconstruction occurs, algorithms written in Python can trigger signals to notify the application that a new volume is available. A slot on the C++ side listens for this signal, using a mutual exclusive lock (mutex) on the image data to secure access to the updated volume. The new data is copied into the foreground thread (main GUI), and once it is available the mutex is released. Once the application receives a signal indicating that the output has been updated, downstream data operations can then be re-executed and any connected visualization modules will also be notified. As an effect, the histogram is recalculated in another background thread while all the current visualization modules display the rendered representation. In the case of the contour module, this will necessitate the recalculation of the surface mesh or the update will be uploaded to the GPU for volume rendering. Figure 1 and the user manual use exemplary data from Supplemental Dataset 1. The aligned and raw tilt series for the FePt dataset in Figure 2 can be accessed through physics.ucla.edu/research/imaging/FePt. In addition, the projection images for the Co 2 P and C-Pt nanoparticles displayed in Fig.  2 and Fig. 3 are available through doi.org/10.6084 [26].

Code Availability
Tomviz binaries are available as Supplementary Software and the source code can be downloaded from GitHub (github.com/OpenChemistry/tomviz).