Multicolor multifocal 3D microscopy using in-situ optimization of a spatial light modulator

Multifocal microscopy enables high-speed three-dimensional (3D) volume imaging by using a multifocal grating in the emission path. This grating is typically designed to afford a uniform illumination of multifocal subimages for a single emission wavelength. Using the same grating for multicolor imaging results in non-uniform subimage intensities in emission wavelengths for which the grating is not designed. This has restricted multifocal microscopy applications for samples having multicolored fluorophores. In this paper, we present a multicolor multifocal microscope implementation which uses a Spatial Light Modulator (SLM) as a single multifocal grating to realize near-uniform multifocal subimage intensities across multiple wavelength emission bands. Using real-time control of an in-situ-optimized SLM implemented as a multifocal grating, we demonstrate multicolor multifocal 3D imaging over three emission bands by imaging multicolored particles as well as Escherichia coli (E. coli) interacting with human liver cancer cells, at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sim 2.5$$\end{document}∼2.5 multicolor 3D volumes per second acquisition speed. Our multicolor multifocal method is adaptable across SLM hardware, emission wavelength band locations and number of emission bands, making it particularly suited for researchers investigating fast processes occurring across a volume where multiple species are involved.

In this paper, we present such a multicolor multifocal microscope platform concept which allows uniform subimage intensities across multiple tunable emission bands using the same grating optics. This is accomplished by using in-situ optimization of an SLM deployed as the multifocal grating 4 and generalizing the framework for an in-situ multicolor implementation. In principle, our approach is adaptable to any c and number of emission channels. These characteristics make this multicolor multifocal microscope a useful tool for various imaging applications involving multicolor species. In the rest of the paper, we first describe the optical setup, followed by a description of single-color multifocal microscopy. We then describe our approach towards multicolor multifocal operations, before presenting the experimental demonstrations. This is followed by a discussion section before a brief conclusion paragraph which summarizes the work.

Multicolor multifocal microscope
Optical diagram. The multicolor multifocal microscope design is shown in Fig. 2. Excitation light from multiple laser sources (three lasers are shown in Fig. 2) are directed through dichroic optics towards a sample via a microscope objective. Lens 1 focuses the excitation light onto the back focal plane of the objective to illuminate the full sample imaging field of view. Emission light from fluorophores in the sample is collected using the same objective and reflected off Dichroic 3 towards Lens 2 which forms an image at the Rectangular Aperture plane. This Rectangular Aperture limits the imaging field of view to prevent multifocal subimages from overlapping. The emission light is then directed onto the SLM placed at the Fourier plane via Lens 3. A linear polarizer is placed in the emission path to ensure phase-only operation of the SLM. A grayscale grating pattern displayed on the reflective SLM results in multiple diffraction orders of the emission light which are imaged onto a camera by Lens 4 through a multi-bandpass emission filter. The inset in Fig. 2 shows the camera placement of the subim- Figure 1. Example images showing the effect deploying an SLM-displayed multifocal grating pattern for an emission filter for which the pattern is not designed for. Here, an SLM pattern is displayed which was optimized for a 3 × 3 subimage array using an emission filter having c = 671 nm and SLM calibration settings giving 0 − 2π relative phase control for = 696 nm. Using this pattern, corresponding bright-field mode multifocal images are obtained using emission filters having (a) c = 671 nm, which is the pattern design wavelength giving M = 0.81 , and (b) c = 512 nm, which is far from the design wavelength giving M = 0.07 . The orange rectangle indicates the camera region occupied by the central 3 × 3 subimage array.  Multifocal imaging over a single-wavelength emission band. Before presenting the multicolor multifocal pipeline, we first describe conventional single-wavelength emission band multifocal imaging. To begin, using the in-situ iterative algorithm 4 , an optimized SLM multifocal grating pattern specific to the emission band is acquired. Briefly, the SLM is calibrated using the calibration settings specific to the emission wavelength c to achieve 0-2π relative phase control corresponding to 0-255 grayscale patterns. The optimized pattern is displayed on the SLM and can readily be edited to vary z without affecting the subimage intensities 4 . Once a desired z is chosen, the pattern is displayed on the SLM and is kept unchanged throughout an imaging experiment until it finishes or a different z is desired.
Multifocal imaging across multiple emission bands. By and large, there are two steps involved in generalizing multifocal imaging for arbitrary multiple emission bands using an SLM-based technology. One is the calibration of the input control voltage to the phase of each SLM pixel whereas the other is the generation of SLM-generated grating pattern for multifocal image projection. In our implementation, the former is accomplished through a separate control experiment and the latter is carried out n situ. To begin, optimized SLM patterns are obtained separately for each of the emission bands. For multicolor multifocal imaging, the same SLM calibration settings are used for operations across multiple emission bands. This is possible because of the adaptability the in-situ iterative algorithm 4 which provides grating patterns giving near-uniform subimage intensities regardless of SLM calibration settings. Once optimized SLM patterns are obtained for the different emission bands using the same SLM calibration settings, these patterns are stored and synchronously displayed on the SLM with a multicolor excitation and image acquisition routine. A timing diagram of an example two-colored multifocal microscope illustrating this concept is shown in Fig. 3. In Fig. 3, the two-colored timing diagram assumes excitation lasers of wavelength 488 nm and 647 nm, while a multi-bandpass emission filter has emission bands located at c values of 512 nm and 671 nm. To begin the multicolor multifocal imaging routine, the SLM pattern optimized for c = 512 nm is first sent to the SLM for display. A wait period of time T W is subsequently needed to account for the ∼16.67 ms (60 Hz) addressing rate of typical SLMs as well as the response time of the liquid crystals in the SLM 9 . After T W has passed, the 488 nm laser is turned on while the camera acquires an image having an exposure time T E1 . Once the image corresponding to 488 nm excitation is acquired, the 671 nm centered emission band optimized grayscale is sent to the SLM. A wait time of T W passes before the 647 nm laser turns on and the camera acquires an image of exposure time T E2 corresponding to the emission band centered at 671 nm. This cycle is repeated until the desired number of images are acquired. Note that this sequence can accommodate additional wavelength emission bands, each having a corresponding wait time T W as well as acquisition time. These features make this method universally adaptable for a large variety of imaging requirements. This concludes the multicolor multifocal pipeline.

Multifocal image uniformity metric M. For a measured subimage intensities {
where N is the number of subimages, the metric M is used to quantify multifocal image intensity uniformity 4 :

Results
The multicolor multifocal microscope is implemented on a home-built aluminum block as the microscope base and controlled using an in-house developed LabVIEW 10 code. The SLM used is Holoeye Pluto-VIS-056 having a pixel size of 8 µm , which is calibrated using a manufacturer-provided file to give 2.3π relative phase control at 532 nm. This is meant to provide a linear 2π relative-phase control range for a wavelength of 611.8 nm using 8-bit grayscale patterns. We have empirically chosen this particular setting because it is near the mid-point of the emission band c 's used in the experiments. These SLM calibration settings were kept fixed throughout the duration of the experiments, an important step of the multicolor multifocal design. The z is set to 500 nm, and a grating period of 4 × 4 SLM pixels is used. The remaining experimental components used in the setup are discussed in Methods. Furthermore, details describing how z is controlled can be found in previous works 4,5 .
Using brightfield illumination, optimal SLM patterns providing near-uniform intensities across multifocal subimages are obtained using the in-situ iterative algorithm 4,11 for c values 512 nm, 610 nm and 671 nm. The corresponding brightfield-mode multifocal images are displayed in Fig. 4a, b, and c, exhibiting high M values and showing the near-uniform subimage intensities across the three empirically chosen emission bands. In principle, the method is applicable for any c location in the wavelength spectrum. Fig. 4d is a snapshot of a 200 nm fixed bead sample acquired using z = 900 nm and c = 671 nm, indicating the lateral point spread function (PSF) of the system. Supplementary video 1 shows a collection of images acquired in a z-stack of the 200 nm fixed bead sampleshowing how the axial PSFs vary as the beads come in and out of focus in the subimages. We observe the PSF axial resolution across the 9 planes to be near-uniform (see Supplementary information). Note that the shape of the PSF is different across the subimages. This is due to the finite bandwidth of the emission filter which causes chromatic spreading arising from the grating pattern displayed on the SLM. The impact of emission filter bandwidth on the shape of the PSF in such systems is discussed thoroughly in previous work 12 .
The following experiments were performed following the Fig. 3 sequence, with T W set to 135 ms. This T W value was empirically found to afford high-performance multicolor imaging with our existing hardware, given the 16.67 ms addressing time and the >70 ms response time (10% to 90%) quoted in the SLM manual provided by the manufacturer.
Using the obtained optimized SLM patterns, we first demonstrate multicolor multifocal imaging of twocolored 100 nm diameter particles freely diffusing in solution. A diluted mixture of yellow-green (505/515) and red (580/605) µm-sized particles (FluoroSphere, ThermoFisher Scientific) is sandwiched between two coverslips (ThermoFisher), and imaged using the microscope (see Methods for detailed sample preparation). The different colored particles were imaged using the sequence shown in Fig. 3 and the SLM patterns optimized for 512 nm and 610 nm, which were illuminated by Laser 3 and Laser 2, respectively (cf. Fig. 2). Both T E1 and T E2 were set to  Fig. 5c. This 2.5 volumes per second multicolored particle imaging demonstration shows the utility of our system for potentially tracking multiple types of species moving over a large volume. Next, we imaged a sample containing the bacteria E. coli and human liver cells (see Methods for sample preparation details) using the multicolor multifocal imaging platform. It has been reported that the engineered bacteria can deliver the drug to the trageted tumor to inhibit the growth of the tumor 14,15 . The proof-of-principle experiment in this section is expected to show the capability of our multicolor multifocal imaging platform that can unveil the underlying mechanism of such interactions between bacteria and mammalian cells. Fluorescent E. coli excited by Laser 3 were imaged in the 512 nm channel while human liver cells were excited by Laser 1 and imaged in the 671 nm emission channel. Both T E1 and T E2 were set to 100 ms for this demonstration. Supplementary video 5 shows the 9-plane multifocal images of E. coli, while Supplementary video 6 displays the acquisition of human liver cells from the same multicolor dataset. Selected frames from the multicolor data acquisition are shown in Fig. 6. We also overlayed the central subimages (corresponding to only the z = 0 plane) from both  www.nature.com/scientificreports/ channels to show the spatial relationship of the two species ( Fig. 6c and Supplementary video 7), where the E. coli is colored green and the human liver cells are colored red. In particular, Supplementary video 7 shows the E. coli pushing and pulling on the human liver cells. Note that the remaining 8 planes from each channel were unused in Fig. 6c as part of the overlay process, showing the large volume of information available to visualize and observe such dynamics. Fig. 6d shows a snapshot of a 3D rendered movie (Supplementary video 8) computed using the combined multicolor multifocal data as input to an in-house MATLAB rendering code. The large red structures in Fig. 6d represent the human liver cells while the blue structures around them represent the E. coli. The ability to acquire 3D volumetric multicolor data such as shown in Fig. 6 makes this technique suitable for investigating inter-species dynamics across a variety of systems.

Discussion
The multicolor multifocal imaging routine presented here overcomes prior-art issues of non-uniform subimage intensities during multicolor image acquisitions. Our in-situ optimization method for SLM as the multifocal grating, which allows high performance multicolor multifocal imaging, is adaptable across multiple microscope types and SLM hardware manufacturers, allowing access to many researchers. Our design allows the use of the same SLM calibration settings for different wavelengths of operations. Empirically, the exact choice of the 0 − 2π calibration wavelength is not important as long as the calibration wavelength for 2π relative phase control is near the desired emission c values. An important aspect of the SLM-based multifocal microscope is the efficient detection of signal photons arising from the sample. The in-situ iterative method used here gives ∼75% diffraction efficiency, which is the normalized sum of the intensities in the non-zeroth order subimages 4 . The light efficiency of our method, in which we include the zeroth order intensities into our calculations, is ∼85% , i.e., 85% of the emission light reflected by the SLM is contained in the 9 subimages captured by the camera. Note that this calculation does not include transmission through the bandpass filter as well as the lens forming the image on the camera. Assuming a ∼95% reflectivity of the SLM quoted by the manufacturer, we get roughly 80% of the emitted light striking the camera to form a multifocal image. Splitting into 9 subimages gives approximately ∼9% of the emitted light per subimage. Although this reduction in subimage brightness is not an issue for bright samples, this can impact imaging of weakly fluorescent samples. In the latter case, the exposure time of the camera and/or the sample illumination will have to be increased in case the multifocal images appear to have low a signal-to-noise ratio. The losses in the emission path include light leaking into higher diffraction orders which are not used/captured by the camera as well as the imperfect SLM reflectivity. As this technology evolves, we anticipate this efficiency to increase further as new optimization algorithms are developed.
The multicolor imaging speeds depend significantly on the hardware used. In our current setup, we used T W = 135 ms. This wait time was obtained experimentally by measuring how long it took for a new SLM grating pattern to stabilize on the camera by evaluating the resulting M values. Different SLMs have different response times which depends on a range of factors including the liquid crystal pixel characteristics as well as the drive voltage settings. Deploying SLMs with faster response times will increase the volume imaging speed of the proposed multicolor multifocal routine. Furthermore, note that the location of subimages in the multifocal images changes shown for different c values as expected according to the grating equation 16 .
Finally, the large volumes of sample information provided by the multicolor multifocal microscope as well as their variation across camera regions due to different emission wavelength bands indicate a need for specialized post-processing techniques. Although researchers have implemented various reconstruction methods for multifocal imaging volumes 17,18 , the extension to multicolor imaging presents unique computational challenges yet opportunities for future research, such as deciphering 3D shapes from multiple focal plane images of different shaped objects interacting with each other (e.g., Fig. 6). Note that in this manuscript, we extend the utility of the SLM-based multifocal microscope to realize multicolor multifocal imaging. In addition to having no objective or sample motion necessary for volumetric imaging, the advantages of having a spatial light modulator perform this task are numerous including customizable emission bands and focal spacing between the planes being imaged.

Conclusion
In conclusion, an SLM-based multicolor multifocal microscopy platform is presented. Using in-situ optimization of an SLM used as the multifocal grating, we show that it is possible to provide near-uniform multifocal subimage intensities for three emission wavelength bands in the visible range using a single multifocal grating. Using this idea, as demonstrated by our platform, researchers can optimize multifocal experiments according to desired dye emission wavelengths as well as the number of emission channels. This promises to be a useful tool for researchers probing volume dynamics across a variety of biological and material systems. Aperture was controlled such that the total field of view per subimage was set to ∼27µm 2 . Bacteria/liver cell sample preparation. A pipette tip was used to scrape a stock of frozen E. coli MG1655

Multi-particle sample preparation.
KanR having the Green Fluorescent Protein (GFP) plasmid pWR20. The pipette tip was then dipped into 3 mL of Lysogeny Broth (LB) media in a tube. 15 µL of Kanamycin was added to this mixture to neutralize the non-GFP plasmid carrying bacteria and incubated at 37 • C for 18 hours. After the incubation period, 15 µL of the bacteria solution was added to 50 µL of the liver-cell solution (see description below). This volume was dropped on a coverslip which was then placed on the microscope stage for imaging. HEP G2 human liver cancer cells (HB-8065) were purchased from ATCC and cultured in Eagle's Minimum Essential Medium (ATCC 30-2003) with 10% fetal bovine serum supplement (Gibco) using Nunclon Delta dish (60 mm×15 mm). Cells were passaged every 5-7 days at ∼80% confluence. SYTO 62 red nucleic acid stain was purchased from Invitrogen and used as directed. The culture medium of cells which had ∼80% confluence were removed and the cells were further rinsed by 1 mL DPBS (Dulbecco's phosphate-buffered saline without Mg 2+ and Ca 2+ ). 1 µ L of the 5 mM SYTO 62 stock solution was added to 1 mL of DPBS (without Mg 2+ and Ca 2+ ) to make a 5 µM staining solution. The cells were stained in the culturing dish by adding 1 mL of the staining solution. The cells were incubated for 20 minutes, followed by 1 mL DPBS (without Mg 2+ and Ca 2+ ) rinsing. The cells were then dissociated from the dish surface by adding 1 mL trypsin. After 10 minutes of incubation, the cells were transferred to solution phase for centrifuging for 5 minutes at 1000 rpm. The supernatant was removed and 2 mL DPBS (without Mg 2+ and Ca 2+ ) was added to remove the unbonded dye molecule residuals. This cleaning process was repeated twice. The stained HEP G2 cells were redissolved back to 2 mL DPBS (without Mg 2+ and Ca 2+ ) for multicolor multifocal experiments. Image processing. All processing is done using self-written code in MALAB. Images from both color channels are cropped, normalized, and smoothed using a Gaussian filter. For processing of the images of mammalian cells in the 671 nm channel, image deconvolution is performed using the deconvlucy() function with the measured PSF for this color channel acting as an input to this process. After deconvolution, The 9 axial planes are interpolated to 27 planes. Next, the transparency values for each pixel are scaled exponentially based on the intensity value of each 3D voxel. The purpose of this step is to make the low intensity voxels more transparent to allow visualization of the brighter intensity voxels spanning the extent of the cells. These transparency values are then thresholded empirically and input into an open source function vold3D() 19 which creates surface plots in all 3 dimensions resulting in the 3D volume of the cells. For the images of bacteria, deconvolution was first performed. Then, for every image in the acquisition sequence, end points of each bacteria were manually annotated in x,y and z. Once these are acquired, cylinders were plotted using the function cylinderModel() in MATLAB from these locations having a diameter of 500 nm.