Multichannel meta-imager for parallel front-end optical computations

doi:10.21203/rs.3.rs-4275158/v1

Download PDF

Article

Multichannel meta-imager for parallel front-end optical computations

https://doi.org/10.21203/rs.3.rs-4275158/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Optical computing has demonstrated significant advantages over electronic computing, including parallelism, high-speed processing, extensive capacity, and low energy consumption. Optical computing front ends leveraging metasurfaces provide advantages such as miniaturization and seamless integration, but have a serious constraint of single computing functionality. Here, we propose a meta-imager, optical computing front end that integrates two coherent transfer functions corresponding to differential and integral convolution kernels into a built-in metasurface. In this architecture, the meta-imager enables parallel processing of multiple all-optical operations for signal computing tasks such as edge enhancement and denoising. We demonstrate the robust integral and differential operations on image signals of noisy patterns and onion cells at multiple visible wavelengths. This optical computing meta-imager paves a promising pathway towards multifunctional image processing for artificial intelligence and biological observation, and shows the potential to expedite and potentially supplant certain digital neural network algorithms.

Physical sciences/Optics and photonics/Optical materials and structures/Metamaterials

Physical sciences/Optics and photonics/Optical techniques/Imaging and sensing

The swift progress of artificial intelligence ^{[1, 2]} heavily depends on the efficacy of electronic computing. However, the computing power of electronic computers is inherently impeded by electronic migration, leading to bottlenecks in both speed and energy consumption. Typically, this process is implemented through circuits, entailing drawbacks such as elevated energy consumption, sluggish operational speed, and restricted data capacity. All-optical computing refers to the use of optical front ends and architectures to simulate basic mathematical operations such as differentiation, integration, and convolution, which has the advantages of low energy consumption, rapid processing, and 2D parallel operation, and has garnered increasing interest in recent years. Pioneer optical computations typically involve incorporating a filtering device at the spatial spectrum plane of the 4f system, a method based on the diffraction imaging theory introduced by Abbe in 1873 ^[3]. Subsequently, the continuous practice of many researchers laid the foundation of optical computations based on Fourier transform spatial filtering ^[4–7]. Nevertheless, traditional optical operation systems are hampered by drawbacks such as large size, integration challenges, and limited functionality, which impede the further progression of optical computing towards modern applications.

Metasurfaces are artificial sub-wavelength materials with the advantages of small size, easy integration, and versatile functionalities ^[8–18], providing a new platform for designing optical computation front ends ^[9–21]. One typical option is to design flat differentiator based on spatiotemporal modulation metasurfaces according to the Green’s function method ^[22–36], which directly realize desired differentiations in real space. For example, Jin ^[28] and Liu ^[29] et al. utilized a multi-layer dielectric metasurface incorporating two types of dielectric materials to achieve first- and second-order spatial differentiations. However, in this method, the response of metasurface must adhere to resonance conditions, requiring specific incident angle and working wavelength. Another strategy refers to using metasurfaces with elaborate complex amplitude transmittance corresponding to optical transfer functions to replace the filtering devices in 4f systems ^[37–49]. This architecture makes it easy to integrate optical computation front end with imaging systems. For instance, Fu et al. ^[43] devised several metasurface-based image processors from the inverse design of convolution kernels of 4f systems. Intarvanne et al. ^[44] utilized a multifunctional metasurfaces to achieved simultaneous microscopic imaging, edge detection, and polarization imaging at the output plane. This method offers a much greater degree of flexibility for metasurface design, resulting in features such as wideband response ^[13]. Despite the tremendous developments in this emerging platform in past years, the multichannel parallel computing issue has not been well addressed, that is, metasurface-based optical processors only implement a single computing functionality, resulting in a significant functional gap compared with electronic computing.

In this work, we propose a multichannel meta-imager that can realize parallel optical computations. The meta-imager is composed of a 4f imaging system and a single-layer metasurface designed in reverse according to two convolution kernels corresponding to differential and integral operations, in which the parallel computing architecture is implemented by the polarization encoding metasurface. This front-end optical processor can perform differential and integral operations on the input signal, and output images with edge enhancement and denoising functionalities. Furthermore, we validate this robust integral and differential operations on image signals of noisy patterns and onion cells at multiple visible wavelengths.

Design principles

Optical imaging systems generally satisfy linear and spatial invariance, wherein the relationship between the input and output images of the system can be described through convolution operation. Specifically, for coherent light imaging systems, the input light field complex amplitude g_in(x₀,y₀), output light field complex amplitude g_out(x,y), and the system’s spatial impulse response function h(x,y) satisfy the equation g_out(x,y) = g_in(x₀,y₀)∗h(x,y), here we refer to the spatial impulse response function h(x,y) as a convolution kernel. Since the metasurfaces are typically 10²∼10³ µm in size, so we integrate it into a 4f system to perform the spatial spectrum modulation, of which the complex amplitude transmittance is namely the coherent transfer function (CTF), denoted by H(f_x,f_y), and its relationship with the convolution kernel is H(f_x,f_y)=Φ{h(x,y)}. As per the Fourier transform convolution theorem, this system can directly achieve the convolution of the input image with h(x,y). Consequently, diverse convolution operations can be realized by using specially designed CTF within this imaging system.

Integration and differentiation are two fundamental mathematical operations that have practical value in image processing applications for denoising and edge detection. We formulate a 3×3 binary convolution kernel depicted in Fig. 2a to perform the integral operation, whose central part, marked with a value of 0, constrains the maximum extent of the intensity smooth range. Similarly, the convolution kernel corresponding to differential operation can be expressed in another 3×3 matrix shown in Fig. 2c. Upon Fourier transformation, the normalized intensity distributions of these CTFs corresponding to two operations emerges, as illustrated in Figs. 2b and 2d. Obviously, the normalized intensity distribution of the CTF for differential operation exhibits approximate complementarity to the integral operation. To align with the demands of multichannel parallel computation, the intensity distribution of the differential operation CTF is approximated as 1−|H(f_x,f_y)|² with H(f_x,f_y) depicting the CTF of integral operation.

In order to achieve parallel convolution operations mentioned above, we adopt the polarization encoding scheme to design a vectorial metasurface that enables wideband operation capability (12, 14). In the context of geometric phase metasurfaces, when the rotation angle of the polycrystalline silicon nanopillar is denoted as θ, for the incidence of linearly polarized light, the transmitted light induces a polarization rotation depending on the geometric phase, i.e., ϕ = 2θ. Assuming that the intensity of incident light with polarization in a certain direction is depicted as I₀, according to the Malus' law, the intensity of the transmitted component, which coincides with the direction of incident polarization, is I = I₀cos²(ϕ). The arrangement angle of each nanopillar unit on the metasurface can be designed according to the convolution kernel of integral operation, ensuring that the intensity transmittance of the metasurface aligns with the intensity distribution function of the integral operation CTF, i.e., I₀cos²(ϕ)=|H(f_x,f_y)|². For the orthogonal polarization component transmitted from the metasurface, opposite intensity distribution 1−|H(f_x,f_y)|² is obtained, which approximates the CTF of differential operation. The specific design schematic of the metasurface is illustrated in Fig. 2e. From the center to the edge, the rotation angle of the nanopillar gradually varies from 0° to 45°, and the local polarization rotation of incident light field with horizontal polarization progressively changes from 0 to π/2. As depicted in Fig. 2f, this metasurface is positioned at the spatial spectrum plane of the 4f system. When post-selection polarization direction is consistent with the incident one, this imaging front end performs the integral operation, while implements the differential operation in the orthogonal polarization channel.

We fabricated the vectorial metasurface with transmission-type configuration by using standard electron-beam lithography and inductively coupled plasma etching. Figure 3 shows the microscope images of the metasurface, which consists of polycrystalline silicon (Poly-Si) rectangle nanopillars deposited on a glass substrate. The height and period of the nanopillar are H = 600 nm and P = 400 nm, respectively, and the refractive index is n = 4.018 + i0.057. The geometric size of the nanopillar is W = 100 nm and L = 188 nm, which was selected by employing the finite difference time domain (FDTD) method at 633 nm wavelength. The optical microscope image of the metasurface is shown in Figs. 3a, whose transmittance is uniform, therefore it will not affect the imaging functionality of this optical computing front end. We further characterized the transmittances of two polarization channels, whose transmissions are shown in Figs. 3c and 3d.

Integral and differential operations

We used the experimental setup depicted in Fig. 3d to demonstrate the computational performance of the meta-device. The output light from a supercontinuum laser (SC-Pro, YSL photonics) was firstly selected by an acousto-optic tunable filter (AOTF) system, which generated monochromatic light within the range of 400 ~ 2400 nm, then passed through a polarizer (P₁). The horizontally transmitted light illuminated an object (O) positioned at the front focal plane of the 4f system, which consists of two lenses (L₁ and L₂) with a focal length of 7.5 cm. The output image that contains integral and differential operation results were obtained by the CCD camera placed on the imaging plane of 4f system. We used the combination of a polarizer (P₂) and a common COMS camera (acA2040-90 µm, Basler) to individually observe two operations. Importantly, to perform parallel computation, the results of two computing operations can be obtained simultaneously with a single exposure of a polarization camera.

As a proof of concept, the integration and differentiation were performed on the UASF1951 resolution test chart shown in Fig. 4a under 633 nm wavelength. Figures 4b and 4c show the numerically simulated results, the corresponding experimental results are shown in Figs. 4d and 4f. To facilitate a comprehensive comparison between experimental and simulated outcomes, we draw the intensity distributions along the white dashed lines of the experimental results shown in Figs. 4d and 4f, and compared it with the input and simulated results, as shown in Figs. 4e and 4g. Notably, the experimental results are consistent with the numerically simulated results, for image signal, the integral operation achieves the smooth denoising effect, while the differential operation results in edge enhancement.

From an intuitive perspective, the integral operation on an image can produce image blur and eliminate noise, and the smoothness increasing as the image size decreases. This is attributable to our designed integral operation, which assumes a value of 0 in the central region and 1 in the surrounding positions of the convolution kernel. Upon convolution operation with the incident image, the average intensity of the surrounding eight pixeled regions replaces the one of internal 0-value region, resulting in robust smoothing and noise removal effects. When the feature image size is greater than the 0-value region of the convolution kernel h(x,y), it does not completely remove the image blurring. Conversely, when the feature size is significantly smaller than the 0-value region, image denoising occurs. Therefore, the size of the 0-value region is critical for convolution operations on objects with a feature size. Considering the circular symmetry of the meta-imager, we take the 0-value region as a circle and define its diameter as the denoising diameter. In experiment, we set it to 88.39 µm, which is equivalent to the width of the 4th unit stripes in the second group of USAF1951 resolution test chart. Consequently, the central part of stripes with a line width > 88.39 µm retains the intensity of the original image, while stripes with a line width < 88.39 µm experience reduced or nearly complete elimination of intensity.

Performing differential operation on images will result in edge enhancement. Notably, the internal intensity of stripes in the fourth unit of the second group significantly decreases, falling below the edge intensity. For stripes with a line width > 88.39 µm, the interior is nearly completely eliminated, preserving only the edge strength. Stripes smaller than 88.39 µm experience varying degrees of intensity reduction without an evident edge detection effect. Consequently, the minimum resolution of edge detection corresponds to the denoising diameter of the integral convolution kernel mentioned earlier.

We further validated the application of these computing operations of the meta-imager in denoising and edge enhanced imaging, which were performed on specifically devised samples containing noise and onion cells at 633 nm wavelength. The denoising imaging corresponding to the integral operation was firstly tested. Figure 5a displays a flower pattern with a diameter of 1000 µm, featuring randomly distributed noise points with a diameter of 20 µm. The integral operation result is shown in Fig. 5b. Due to the denoising diameter of the meta-imager being 88.39 µm, which is much larger than 20 µm, although the noise can be completely eliminated, the edges of the flower pattern are severely blurred, making it difficult to recognize the image that should be retained. To solve this problem, a micro magnification system was incorporated into the experimental setup, enlarging the sample by 9.4 times, the denoising result is depicted in Fig. 5c. For this case, the convolution kernel diameter is smaller than the noise, the denoising effect is not significant. Further, a convolution kernel with diameter of 250 µm was used, of which the structure and corresponding CTF are shown in Figs. 5d and 5e, respectively. The integral computation result is shown in Fig. 5f, clearly, the matching convolution kernel can effectively eliminate the noise while preserving the edge contours. In practical applications, designing convolution kernel necessitates consideration not only of noise size but also other image parts and resolution of the image acquisition device. Theoretically, the greater the difference between noise and image size, the more effective the denoising achieved by the integral operation under this principle.

Finally, differentiation and integration were performed on onion cells. Figure 5g displays onion cells without computing operations. To facilitate result observable, these cells were enlarged by 9.4 times, and then differential and integral operations were carried out on these cells using a convolution kernel with diameter of 88.39 µm. Figure 5h illustrates the result of the differential operation, revealing significant elimination of the cells' interiors while preserving relatively complete edges. Figure 5i displays the outcome of the integral operation, wherein wrinkles and pits inside the cells in the original image are eliminated, and the onion cell edges are distinctly visible. The experimental results affirm that our designed meta-imager serves two operation functions: image denoising and edge detection, showing potential applications in image processing and biological sample observation.

In this work, the one of the key factors in achieving parallel computations is the polarization-multiplexing optical operation. Although in the experiments, we independently demonstrated differential and integral operations in the two linear polarization channels, these polarization-dependent outputs can be recorded simultaneously a single exposure by using a polarization camera. Compared with previous optical computations, it highlights our advantages of more functionality and faster working speed. In addition, the use of the geometric phase metasurface provides the advantage of a wideband response, which are shown in the supplementary materials.

This work proposes a multichannel meta-imager based on geometric phase metasurfaces, to perform parallel optical front-end computing. This parallel computation architecture combines two convolution kernels associated with integration and differentiation by inversely designing polarization-encoding metasurface. The metasurface has polarization-multiplexing complex amplitude modulations, which enables the meta-imager to perform the integral and differential operations simultaneously. As proofs of concept, computing operation on image signals of standard resolution test chart, designed samples, and onion cell were performed. Importantly, this reverse design of the meta-imager allows customization of noise removal and edge detection resolution, optimizing calculation results to suit a broader range of application scenarios.

Fabrication

The metasurfaces were fabricated based on the processes of deposition, patterning, and etching. Firstly, a 600 nm thick Poly-Si film was deposited on a 500 µm thick fused silica substrate by inductively coupled plasma enhanced chemical vapor deposition (ICPECVD), and then a 100 nm thick Hydrogen silsesquioxane electron beam spin-on resist (HSQ, XR-1541) was spin-coated onto the Poly-Si film. And secondly, the desired structures were imprinted by using standard electron beam lithography (EBL, Nanobeam Limited, NB5) and subsequently developed in NMD-3 solution (concentration 2.38%) for 3 minutes. Finally, by using inductively coupled plasma etching (ICP, Oxford Instruments, Oxford Plasma Pro 100 Cobra300), the desired structures were transferred from resist to the Poly-Si film.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Competing interests

The authors declare no conflicts of interest.

Author contributions

X.B. and P.L. conceived the idea, designed the experiments and wrote the manuscript. X.B. and X.F. conducted the numerical simulations. X.W. and C.Z. fabricated the samples. X.B. conducted the experiments and date analysis. D.W. and S.L. contributed to data analysis and manuscript revision. P.L., X.G. and J.Z. Supervised the work. All the authors engaged in result discussions. The authors read and approved the final manuscript.

Acknowledgments

This work was supported by the Key Research and Development Program (2022YFA1404800); National Science Foundation of China (NSFC) (12174309, 12074312), Shaanxi Fundamental Science Research Project for Mathematics and Physics (Grant No. 22JSY013), the Fundamental Research Funds for the Central Universities (Grant No. 5000230111), the Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University (Grant No. CX2022076).

Yin, C. C., Zhang, Q. J. & Ren, W. Q. A new semantic edge aware network for object affordance detection. J. Intell. Robot Syst. 104, 2 (2022).
Lee, S. Jeong, B. Park, K. Song, M. & Kim, S. Y. On-CMOS image sensor processing for lane detection. Sensors 21, 3713 (2021).
Abbe, E. Beiträge zur Theorie des Mikroskops und der mikroskopischen Wahrnehmung. Archiv für mikroskopische Anatomie. 9, 413–468 (1873).
Zernike, F. Phase contrast. Z Tech Physik. 16, 454 (1935).
Elias, P. Grey, D. S. & Robinson, D. Z. Fourier treatment of optical processes. J. Opt. Soc. Am. 42, 127–134 (1952).
Maréchal, A. & Croce, P. Un filtre de fréquences spatiales pour lamélioration du contraste des images optiques. C R Acad Sci Paris. 237, 607–609 (1953).
Davis, J. A. McNamara, D. E. Cottrell, D. M. & Campos, J. Image processing with the radial Hilbert transform: theory and experiments. Opt. Lett. 25, 99–101 (2000).
Yu, N. F. et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science 334, 333–337 (2011).
Sun, S. L. et al. High-efficiency broadband anomalous reflection by mradient meta-surfaces. Nano Lett. 12, 6223-6229 (2012).
Lin, D. M. Fan, P. Y. Hasman, E. & Brongersma, M. L. Dielectric gradient metasurface optical elements. Science 345, 298–302 (2014).
Arbabi, A. Horie, Y. Bagheri, M. & Faraon, A. Dielectric metasurfaces for complete control of phase and polarization with subwavelength spatial resolution and high transmission. Nat. Nanotechnol. 10, 937–943 (2015).
Guo, X. Y. et al. Full-color holographic display and encryption with full-polarization degree of freedom. Adv. Mater. 34, 2103192 (2022).
Guo, X. Y. et al. On-demand light wave manipulation enabled by single-layer dielectric metasurfaces. APL Photonics 6, 086106 (2021).
Guo, X. Y. et al. Stokes meta-hologram toward optical cryptography. Nat. Commun. 13, 6687 (2022).
Huang, Z. R. et al. High-resolution metalens imaging polarimetry. Nano Lett. 23, 10991–10997 (2023).
Dai, A. L. et al. Multifunctional metasurfaces enabled by multifold geometric phase interference. Nano Lett. 23, 5019–5026 (2023).
Chen, C. et al. Neural network assisted high-spatial-resolution polarimetry with non-interleaved chiral metasurfaces. Light-Sci. Appl. 12, 288 (2023).
Wang, X. W. et al. Single-shot isotropic differential interference contrast microscopy. Nat. Commun. 14, 2063 (2023).
Zangeneh-Nejad, F. Sounas, D. L. Alù, A. & Fleury, R. Analogue computing with metamaterials. Nat. Rev. Mater. 6, 207–225 (2021).
Badloe, T. Lee, S. & Rho, J. Computation at the speed of light: metamaterials for all-optical calculations and neural networks. Adv. Photonics 4, 064002 (2022).
Zhou, H. Q. et al. Optical computing metasurfaces: applications and advances. Nanophotonics 13, 419–441 (2023).
Silva, A. et al. Performing mathematical operations with metamaterials. Science 343, 160–163 (2014).
Kwon, H. Sounas, D. Cordaro, A. Polman, A. & Alu, A. Nonlocal metasurfaces for optical signal processing. Phys. Rev. Lett. 121, 173004 (2018).
Dong, Z. W. Si, J. N. Yu, X. Y. & Deng, X. X. Optical spatial differentiator based on subwavelength highcontrast gratings. Appl. Phys. Lett. 112, 181102 (2018).
Cordaro, A. et al. High-index dielectric metasurfaces performing mathematical operations. Nano Lett. 19, 8148–8423 (2019).
Zhou, Y. Zheng, H. Kravchenko, I. I. & Valentine, J. Flat optics for image differentiation. Nat. Photonics 14, 316–323 (2020).
Komar, A. et al. Edge detection with Mie-resonant dielectric metasurfaces. ACS Photonics 8, 864–871 (2021).
Zhu, T. F. et al. Topological optical differentiator. Nat. Commun. 12, 680 (2021).
Jin, C. & Yang, Y. Transmissive nonlocal multilayer thin film optical filter for image differentiation. Nanophotonics 10, 3519–3525 (2021).
Liu, Y. Huang, M. Chen, Q. & Zhang, D. Single planar photonic chip with tailored angular transmission for multiple-order analog spatial differentiator. Nat. Commun. 13, 7944 (2022).
Huang, J. Y. Zhang, J. H. Zhu, T. F. & Ruan, Z. C. Spatiotemporal differentiators generating optical vortices with transverse orbital angular momentum and detecting sharp change of pulse envelope. Laser Photonics Rev. 16, 2100357 (2022).
Doskolovich, L. L. Kashapov, A. I. Bezus, E. A. & Bykov, D. A. Spatiotemporal optical differentiation and vortex generation with metal-dielectric-metal multilayers. Phys. Rev. A 106, 033523 (2022).
Tanriover, I. Dereshgi, S. A. & Aydin, K. Metasurface enabled broadband all optical edge detection in visible frequencies. Nat. Commun. 14, 6484 (2023).
Cotrufo, M. Singh, S. Arora, A. Majewski, A. & Alu, A. Polarization imaging and edge detection with image-processing metasurfaces. Optica 10, 1331–1338 (2023).
Chen, H. Gui, L. L. Guo, J. Y. & Xu, K. Efficient dual-wavelength metasurface for second-order differential edge detection in the ultraviolet. Appl. Opt. 62, 8552–8557 (2023).
Sol, J. Smith, D. R. & Hougne, P. D. Meta-programmable analog differentiator. Nat. Commun. 13, 1713 (2022).
Zhou, J. X. et al. Optical edge detection based on high-efficiency dielectric metasurface. PNAS 116, 11137–11140 (2019).
Huo, P. et al. Photonic spin-multiplexing metasurface for switchable spiral phase contrast imaging. Nano Lett. 20, 2791–2798 (2020).
Yang, H. et al. Switchable imaging between edge-enhanced and bright-field based on a phase-change metasurface. Opt. Lett. 46, 3741–3744 (2021).
Chen, G. Zhou, J. Bopp, S. E. Zhao, J. & Liu, Z. Visible and near-infrared dual band switchable metasurface edge imaging. Opt. Lett. 47, 4040–4043 (2022).
Xu, D. Y. et al. Inverse design of Pancharatnam–Berry phase metasurfaces for all-optical image edge detection. Appl. Phys. Lett. 120, 241101 (2022).
Zhou, J. X. et al. Fourier optical spin splitting microscopy. Phys. Rev. Lett. 129, 020801 (2022).
Fu, W. W. et al. Ultracompact meta-imagers for arbitrary all-optical convolution. Light-Sci. Appl. 11, 62 (2022).
Intaravanne, Y. et al. Metasurface-enabled 3-in-1 microscopy. ACS Photonics 10, 544–551 (2023).
Bi, X. Y. et al. Wideband optical edge detection based on dielectric metasurface. Appl. Phys. Lett. 123, 061702 (2023).
Wang, S. et al. Metalens for accelerated optoelectronic edge detection under ambient illumination. Nano Lett. 24, 256–361 (2023).
Liang, X. et al. All-optical multiplexed meta-differentiator for tri-mode surface morphology observation. Adv. Mater. 35, 2301505 (2023).
Wang, Y. Yang, Q. He, S. S. Wang, R. S. & Luo, H. L. Computing metasurfaces enabled broad-band vectorial differential interference contrast microscopy. ACS Photonics 10, 2201–2207 (2023).
Zheng, H. Y. et al. Multichannel meta-imagers for accelerating machine vision. Nat. Nanotechnol. https://doi.org/10.1038/s41565-023-01557-2 (2024).

There is NO Competing Interest.

NCSM.docx
Supporting Information for “Multichannel meta-imager for parallel front-end optical computations”

Download PDF

Version 1

posted

You are reading this latest preprint version

Multichannel meta-imager for parallel front-end optical computations

Status:

Version 1

Abstract

Figures

Introduction

Results

Design principles

Integral and differential operations

Discussion

Materials and Methods

Fabrication

Declarations

Data availability

Competing interests

Author contributions

Acknowledgments

References

Additional Declarations

Supplementary Files

Status:

Version 1