Extending resolution of structured illumination microscopy with sparse deconvolution

Molecular Medicine, School of Future Technology, Peking University, Beijing 100871, China. 9 PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China. 10 School of Mathematical Sciences, Peking University, Beijing 100871, China. 11 Biomedical Engineering Department, Peking University, Beijing 100191, China. 12 Shenzhen Bay Laboratory, Shenzhen 518055, China. 13 School of Software and Microelectronics, Peking University, Beijing 100871, China. 14 Institute for Brain Research and Rehabilitation (IBRR), Guangdong Key Laboratory of Mental Health and Cognitive Science, 15

Despite the theoretically unlimited resolution, the spatial resolution of super-resolution (SR) microscopy in 39 live-cell imaging is still limited. Because multiple raw images are usually taken to reconstruct one super-40 resolved image, any increase in the spatial resolution must be matched with an increase in temporal resolution 41 to avoid resolution degradation due to motion artifacts of fast moving subcellular structures in live cells, such 42 as tubular endoplasmic reticulum (ER) 1 , lipid droplets, mitochondria and lysosomes 2 . Therefore, the highest 43 resolution of current live-SR microscopy is limited to ~60 nm, irrespective of the modalities used 3-7 . To 44 achieve that resolution, excessive illumination power (kW~MW/mm 2 ) and long exposures (> 2 s) are usually 45 required 8 , which may compromise the integrity of the holistic fluorescent structure and degrade the achievable 46 resolution. 47 5 However, the application of continuity a priori also obscures the images and reduces the resolution. Therefore, 95 we propose introducing the sparsity as another prior knowledge to antagonize resolution degradation and 96 extract the high-frequency information. This is because an increase in spatial resolution always leads to smaller 97 PSF for any given fluorescence microscopes. As compared with a conventional microscope, the convolution 98 of the object with the smaller PSF in SR imaging always confers a relative increase in sparsity 99 (Supplementary Fig. 3). Therefore, we believe that continuity and sparsity are general features of the 100 fluorescence microscope, which could be used as the prior knowledge to suppress noise and facilitate high-101 frequency information extraction (detailed in Supplementary Note 3). 102 Overall, we have proposed the following loss function containing these two priors, in which the Hessian 103 matrix continuity to reduce artifacts and increase robustness at the price of reduced resolution, and the sparsity 104 to balance the extraction of high-frequency information, which gives: First, we tested the functionality of different steps in our deconvolution pipeline on synthetic filament 118 structures. While filaments closer than ~100 nm could hardly be resolved by Wiener inverse filtering, 119 reconstruction with the sparsity a priori created only a small difference in fluorescence of the middle part 120 between the two filaments, while the final deconvolution resulted in the clear separation of two filaments 121 down to ~81 nm apart (Extended Data Fig. 2a, 2b). However, the contrast for two filaments ~65 nm apart 122 was low, which could be further improved after the pixel upsampling (labeled as ×2) procedure (Extended 123 Data Fig. 2c, 2d). Regarding synthetic filament structures corrupted with different levels of noise, 124 deconvolution without the addition of the sparsity a prior was unable to retrieve the high-frequency 125 information reliably, while deconvolution without the addition of the continuity a priori led to reconstruction 126 artifacts that manifested particularly in raw images with low SNR (Extended Data Fig. 3). Only the 127 combination of the continuity and sparsity enabled robust and high-fidelity extrapolation of the high-frequency 128 information inaccessible to SIM, even under situations with considerable noise (Extended Data Fig. 3,

129
Supplementary Table 1, 2). 130 In addition, on deconvolution of synthesized punctated-or ring-shaped structures with diameters of 60 ~ 131 120 nm, previous RL deconvolution was only able to resolve rings with diameters larger than 110 nm, while 132 more iterations led to over-shrink artifacts; in contrast, sparse deconvolution was able to resolve rings with 133 diameter down to 60 nm and produced smaller puncta closely resembling the ground truth (Extended Data 134 Fig. 4). Unlike content-dependent SR via deep learning algorithms 30, 31 , the sparse deconvolution could resolve 135 erratic synthetic structures in the same field-of-view (FOV, Supplementary Fig. 6). 136 Finally, the poor SNR condition may limit the sparse deconvolution in improving resolution. Under  Fig. 7). According to a line restoration 139 quality criterion we proposed (detailed in Supplementary Note 5), the sparse deconvolution could not 140 7 faithfully extract information beyond the OTF in images under 150% Gaussian noise condition 141 (Supplementary Fig. 8) Fig. 7), or better visualization of weakly-labeled microtubules under the SD-SIM (Supplementary Fig. 9).

144
Benchmarking of sparse deconvolution against samples with known structures 145 We benchmarked the performance of sparse deconvolution on imaging structures with known ground-truth.

146
By low-pass filtering the image obtained by the 1.4 NA objective with a synthetic PSF from a 0.8 NA objective 147 in the Fourier domain, we created a blurred version of actin filaments (Extended Data Fig. 5a, 5b). 148 Interestingly, two blurry opposing actin filaments under the low NA objective became separable after sparse 149 deconvolution, along with an extended Fourier frequency domain (Extended Data Fig. 5c, 5d). Likewise, 150 two neighboring filaments ~120 nm apart (confirmed by 2D-SIM) were resolved by the sparse deconvolution 151 of wide-field images (Extended Data Fig. 5e, Supplementary Video 15). In addition, a CCP of ~135 nm in 152 diameter under the TIRF-SIM manifested as a blurred punctum in wide-field images that had undergone 153 conventional deconvolution or been deconvolved with the Hessian continuity a priori. Only after sparse 154 deconvolution did the ring-shaped structure emerge (Extended Data Fig. 5f). Similarly, sparse deconvolution 155 but not the RL deconvolution resolved pairs of horizontal lines 150 nm apart (Extended Data Fig. 6), and 156 extended OTF of the wide-field microscope (Extended Data Fig. 7). 157 Next, we have designed and synthesized rod-like origami with two sites fluorescently labeled, each 158 labeled with 4~5 Cy5 molecules (Extended Data Fig. 8). When these molecules were 60, 80, and 100 nm 159 apart, they were barely distinguishable under the TIRF-SIM but well-separated after the up-sampling followed 160 by the sparse deconvolution (Sparse-SIM ×2, Fig. 1). Similarly, one obscure line in the commercial Argo-SIM 161 slide under the 2D-SIM could be resolved as two parallel lines 60 nm apart after the sparse deconvolution 162 only (Fig. 2a, 2b). This resolution enhancement was maintained in processing variable SNR images captured  Fig. 7, 8). 165 Sparse-SIM also resolved ring-shaped nuclear pores labeled with different nucleoporins (Nup35, Nup93,

166
Nup98, or Nup107), while they were similar in sizes to 100-nm fluorescent beads in the same FOV under the 167 2D-SIM (Fig. 2c, 2d, Extended Data Fig. 10, Supplementary Video 1). After corrected for narrower fitted 168 diameters of nuclear pores due to camera pixel sizes and pore diameters comparable to the resolution of 169 Sparse-SIM (Supplementary Fig. 23, Supplementary Note 9.1), Nup35 and Nup107 pores were ~66 ± 3 nm 170 and ~97 ± 5 nm in diameters, while Nup98 and Nup93 pores were of intermediate sizes (Fig. 2e, 2f). These 171 estimations nicely agreed with previous results obtained with different SR methods in fixed cells 32-34 .

172
Interestingly, 12-min SR imaging enables visualization of the vigorous reshaping of nuclear pores in live cells, 173 possibly reflecting reoriented individual nuclear pore complex on the nuclear membrane to or away from the 174 imaging plane (Fig. 2g, Extended Data Fig. 11), which would be difficult for other SR methods.

175
Finally, we tested the reliability of sparse deconvolution in resolving immunofluorescent-labeled 176 complicated structures after expansion microscopy 26 (Fig. 3). Compared to those obtained by the 2D-SIM, 177 tubulin filaments from the 4.5× expanded cell after the sparse deconvolution were comparable in resolution 178 but better contrasted ( Fig. 3a-3c). Similarly, while sparse deconvolution of the same expanded complex ER 179 tubules yielded similar overall shapes to that obtained by SIM reconstruction, it was much less affected by 180 artifacts (Fig. 3d-3f). Taken together, these data demonstrate a bona fide increase in spatial resolution by our 181 sparse deconvolution. 2D-SIM or Hessian-SIM, were resolved by Sparse-SIM ( Fig. 4a-c, Supplementary Video 2 Fig. 25a-25c). In our sparse deconvolution, better separation of dense actin meshes resulted from both the 190 enhanced contrast (Fig. 4d) and the increased spatial resolution, as shown both by the full-width-at-half-191 maximum (FWHM) analysis of actin filaments and Fourier ring correlation (FRC) mapping analysis 37, 38 ( Fig.   192 4e, 4f). This increase in resolution was stable during time-lapse SR imaging of actin dynamics (Fig. 4g), which 193 led to more frequent observations of small pores within the actin mesh. The mean diameter of pores within 194 the cortical actin mesh was ~160 nm from the Sparse-SIM data ( Fig. 4i), similar to those measured by the 195 STORM method in fixed cells 35 . 196 An increase in the spatial resolution also helped resolve the ring-shaped caveolae (fitted diameter ~60 nm) diameter) manifested at the later stage of the exocytosis and sustained for ~47 ms (Fig. 4q). For the TIRF-209 10 SIM, the opening time of the initial pores and the stationary pores were indistinguishable (~37 ms, Fig. 4q), 210 indicating that the early small pore stage was invisible. Nevertheless, although this exocytosis intermediate 211 was not observed by other SR methods, our data agreed with the much lower probability of observing small 212 fusion pores than the larger ones by the rapid-freezing electron microscope reported more than three decades 213 ago 39 .

214
Relative movements between sub-organellar structures observed by dual-color Sparse-SIM 215 Sparse-SIM could be readily used in a dual-color imaging mode to improve the resolution at both wavelengths.  Dual-color Sparse-SIM imaging also resolved more details regarding organelle contacts, including those 225 formed between the mitochondria and ER. We found that ER tubules (Sec61β-mCherry) randomly contacted

231
Because the continuity and sparsity a priori are general features of SR microscopy, we tested our algorithm on  Fig. 14), a nearly twofold increase of spatial resolution in all three axes compared to the SD-SIM.

236
In live COS7-cells labeled with clathrin-EGFP, Sparse SD-SIM enables a previously blurred fluorescent 237 punctum to be resolved as a ring-shaped structure with a fitted diameter of ~97 nm (Fig. 5a,  5e) and the disintegration of another CCP into two smaller rings nearby could be seen (Fig. 5f). Because 243 photon budget allowed by the Sparse SD-SIM could be as small as was ~0.9 W/cm 2 (Supplementary Table   244 3), both actin filaments and CCPs within a large FOV of 44 μm × 44 μm could be monitored for more than 15 the pixel size ~94 nm (Fig. 6a, 6b, Supplementary Video 12). We artificially upsampled the image on a finer 266 grid (labeled as × 2, ~47 nm pixel size) before subsequent sparse deconvolution. Along with an increase in the 267 FRC resolution to ~102 nm and the expanded system OTF (Fig. 6b, 6d), previously blurred ring-shaped ER 268 tubules became distinguishable (pointed by white arrows in Fig. 6c).

269
In HeLa cells, we used Sparse SD-SIM to follow dynamic interactions among lysosomes, peroxisomes, closely to the intersection of two tubulin filaments (Fig. 6f), or the co-migration of a lysosome and a 274 peroxisome along a microtubule for some time before separation and departure (Fig. 6g). These contacts may 275 mediate lipid and cholesterol transport, as reported previously 47 .

276
Finally, we observed nuclei, mitochondria, and microtubules in a three-dimensional volume spanning ~6 277 m in the axial axis of a live COS-7 cell (Fig. 6h, Supplementary Video 14). Again, the axial FWHM of a 278 13 microtubule filament decreased from ~465 nm in the raw dataset to ~228 nm after Sparsity ×2 processing ( Fig.   279 6i). In contrast, TV deconvolution 48 (Supplementary Fig. 28) failed to improve the xy-z axes contrast. From 280 the volumetric reconstruction, it was apparent that the continuous, convex nuclear structure bent inward and 281 became concave at regions in some axial planes that were intruded by extensive microtubule filaments and 282 mitochondria (Fig. 6j). Such reciprocal changes suggest that the tubulin network may affect nucleus assembly 283 and morphology 49 .

285
It is long believed that microscopic optics determines its bandwidth limit. Therefore, it is difficult to imagine 286 how sparse deconvolution extracts the high-frequency information beyond the microscope OTF. By  Fig. 2, 7). 294 As we have elaborated in Supplementary Note 3, the sparsity and continuity priors are general features 295 of high-resolution fluorescence microscopes. Correspondingly, using the sparse deconvolution on images 296 obtained with the point-scanning confocal microscope, we observed features of nuclear pores and 297 microtubules comparable to those obtained with STED 5 (Extended Data Fig. 18). Moreover, as compared to 298 the normal STED, Sparse-STED also provided increased resolution and showed images of actin, ER, and 299 microtubules in live cells at ~40 nm FRC resolution (Extended Data Fig. 19). Finally, sparse deconvolution 300 also extended the observable spatial frequency spectrum of a miniaturized two-photon microscope 25 , nearly 301 14 doubled its resolution quantified by the decorrelation method at different axial positions 51 , and enabled 302 numerous dendritic Thy1-GFP-labeled spines to be visualized in the live mouse brain (Extended Data Fig.   303

304
Unlike the content-dependent SR imaging achieved by the deep-learning algorithms 30, 31 , our sparse 305 deconvolution is content-agnostic, such as revealing both rings and punctuated beads in the same FOV 306 (Extended Data Fig. 10), distinguishing both the bisected ring mixed with irregular lines (Supplementary 307 Fig. 6), and appreciating tubulin filaments and ER tubules after expansion microscopy (Fig. 3). Even for cells  Fig. 11). It 319 is also worth noticing that high SNR images afford large fidelity values, while low SNR ones require small 320 fidelity numbers (Supplementary Note 7). Inappropriate choice of the fidelity and sparsity values may lead 321 to either no increase in resolution, the emergence of artifacts (Supplementary Fig. 12), or the removal of 322 weak signals (Supplementary Fig. 16, Supplementary Fig. 21d)   After three times washing with PBS, the sample was ready for imaging.    image is divided into independent blocks, and the local FRC value is calculated individually using the method 497 reported in 37 . If the FRC estimations in a block are sufficiently correlated, this region will be color-coded in 498 the FRC resolution map. Otherwise, the region will be color-coded according to its neighbor interpolation.

499
Note that before calculating the FRC resolution map of SD-SIM raw images in Fig. 5a, 5h, and 6a, the PURE 500 denoise method 46 was applied in advance to images to avoid the ultralow SNR of SD-SIM raw images 501 perturbing the FRC calculation. We also used the structural similarity (SSIM) values 60 and peak signal-to-502 noise ratio (PSNR) to evaluate the quality of reconstructions in Extended Data Fig. 3. (simulated by the convolution of synthetic filaments with out-of-focus PSF 1 μm away from the focal plane).

512
Moreover, we incorporated Gaussian noise with different variance extents (0%, 11%, 25%, 50%, 80%) to the 513 peak fluorescence intensity of the filaments. The raw images were acquired by a camera with a pixel size of 514 65 nm and pixel amplitudes of 16 bits (Extended Data Fig. 3b).

515
The corrections of bead FWHMs and pore diameters. To extract FWHMs of fluorescent beads and linear 516 structures and the double-Gaussian-peak in ring structures, we used the multiple-peak fit of the Gaussian 517 function in OriginPro. When the sizes of system PSF and the size of camera pixel were comparable to the size 518 of the structure been imaged, fitted diameters of punctated and ring-shaped structures differently deviated 519 from their real values (Fig. 2e, Extended Data Fig. 14c, Supplementary Fig. 23, 24, also detailed in 520 24 Supplementary Note 9). For narrowed fitted diameters of nuclear pores and fusion pores under the Sparse-521 SIM (Fig. 2e, 4q), we corrected these values followed the protocol in Supplementary Note 9.1. For apparent 522 enlarged sizes of fluorescent beads under the microscope (Supplementary Fig. 24 and other details in 523 Supplementary Note 9.2), we included a bead-size correction factor for the beads with the diameter of 100-524 nm to estimate the real resolution of Sparse SD-SIM (Extended Data Fig. 14c).

525
Mesh pore diameters of actin networks. We analyzed the mesh pore diameters (Fig. 4h)  image (known as flat-field), and the additive term D (x) (known as the dark-field) is dominated by the camera 545 offset and its thermal noise that are present even in the absence of incident light. BaSiC estimates the S (x) and 546 D (x) by low rank and sparse decomposition to correct the shading in space and background variations in time. 547 We employed BaSiC to correct the uneven illumination to avoid removing weak signals at the edge of the 548 FOV in the following deconvolution process (Supplementary Fig. 21b-21e). However, we shall be cautious 549 in using such a correction step, avoiding overcorrected and degraded images.

550
Image rendering and processing. The color map SQUIRREL-FRC 38 associated with ImageJ was used to 551 present the FRC map in Fig. 4e. The color map Morgenstemning 67 was applied to show lysosomes in Fig. 5h, 552 and the Fourier transform results in Fig. 6d, Extended Data Fig. 5b, 5d, 7a, 18c, and 20b-20d. The 16-color 553 projection was used to show the depth in Fig. 5k. The jet projection was used to show the depth in Fig. 6k   554 and Extended Data Fig. 20a-20d, time series in Fig. 5g, Extended Data Fig. 15a, and 15b, and RSE error 555 map 38 in Extended Data Fig. 12e, and 12f. The volumes in Fig. 6h, Supplementary Video 11, and 13 were 556 rendered by ClearVolume 68 . The volume in Supplementary Video 18 was rendered by 3Dscript 69 . All data 557 processing was achieved using MATLAB and ImageJ. All the figures were prepared with MATLAB, ImageJ,

558
Microsoft Visio, and OriginPro, and videos were all produced with our light-weight MATLAB framework, 559 which is available at https://github.com/WeisongZhao/img2vid.

560
The adaptive filter for SD-SIM. Confocal-type images often exhibit isolated pixels (1×1 ~ 5×5) with 561 extremely bright values caused by voltage instability or dead/hot camera pixels. The magnitudes of these 562 pixels are approximately 5 to 100 times higher than the normal intensity amplitudes of the biostructure. These 563 isolated pixels are ill-suited for the sparse reconstruction. Unlike the method in 61 , which used percentage 564 image normalization to overcome this problem, we created an adaptive median filter to remove these improper 565 pixels. More specifically, instead of the normal median filter, which replaces each pixel with the median of 566 26 the neighboring pixels in the window, we set a threshold for our developed adaptive median filter. If the pixel 567 intensity is larger than threshold × median in the window, the pixel is replaced by the median; otherwise, the 568 window moves to the next pixel. By using this method, we can filter the isolated pixels without blurring the 569 images. The related method has been written as an ImageJ plug-in and can be found at