Optical mode division multiplexing inspired photonic neural network accelerator

doi:10.21203/rs.3.rs-3091427/v1

On-chip photonic neural network (PNN) is emerging as an attractive solution for artificial neural networks due to its high computing density, low energy consumption, and compact size. Matrix-vector multiplication (MVM) plays a key role in on-chip PNN, which can achieve high-speed multiply-accumulate operation. Most of the current schemes implement MVM by adopting wavelength division multiplexing technology to accumulate the power of different wavelengths together, resulting in using a mass of laser sources. Additionally, real-number-field MVM is inevitable for realizing precise PNNs, while limited by the nature of light, effective solutions to perform negative value computing are still inadequate. Here, we propose and demonstrate a PNN accelerator based on wavelength and mode hybrid multiplexing technology to reduce the use of multi-wavelength lasers, which can satisfactorily play real-number-field computing (including positive and negative domain) based on a newly presented transformation mapping approach, avoiding the demanding experimental setup and the sacrificing weight modulation depth. As a proof-of-concept, a fabricated accelerator for image convolution and letter pattern detection has been demonstrated successfully, achieving a computing density of 1.37 TOPS/mm² under the 22.38 Gbaud modulation rate.

Photonics/optics

Mode division multiplexing (MDM) technology, using various optical modes orthogonally transfer in a bus waveguide, can significantly extend channel capacity. At the present stage, a variety of devices and architectures based on MDM such as mode (de)multiplexers^[38], mode exchangers^[39], multimode crossings^[41], and optical routers^[39,40] are demonstrated. Benefiting from its low insertion loss, low crosstalk, and compact size, MDM becomes an attractive solution for complex and large-scale optical information transmission systems.

The artificial neural network (ANN) is critical in diverse applications including computer vision^[1], decision-making^[2], medical analysis^[3], virtual reality^[4], autonomous vehicle navigation^[5], etc. However, given the acknowledged fact that the electronic circuits conducted by Moore’s law will soon hit a physical bottleneck^[6], the development of electronic ANN is suffered from a mixed blessing between the limited supply and urgent demand. Fortunately, photonic neural network (PNN), benefiting from its ultra-high modulation rate^[7,8], super-large parallel channel capability^[9-14], and ultra-low power consumption^[15,16], is emerging as a vigorous and promising successor to electronic ANN. Among all the components in PNN, the matrix-vector multiplication (MVM) device which multiply and accumulates vectors and matrices together (the data changes in both the positive and the negative domain), is the most significant one because it accounts for the majority of computing time, computing power, and energy consumption^[15,16]. A high-performance MVM device can greatly improve the working speed of neural networks while decreasing their energy consumption. The photonic MVM is first realized based on space light modulation and on-chip planar interference devices^[17-20]. However, these devices are rigidly and bulkily constructed and this problem is compounded since the transform weight matrixes are hard to be adjusted once the devices are fabricated, not to mention the difficulty to achieve both positive and negative value computing simultaneously for practical applications^[21-23]. Consequently, two important challenges emerged for PNN, one is to realize compact-size and low-power MVM with reconfigurable weight matrixes adjustment, and the other is to achieve high accuracy computing in both positive and negative value domains.

On-chip Mach-Zehnder interferometers^[24-26], phase-change materials^[27-29], and micro-ring resonators (MRRs) array^[30-33] offer the possibilities to realize high-order and weight-adjustable MVMs adopting wavelength division multiplexing (WDM) technology. In those schemes, multiple-wavelength lasers are inevitable to implement large-scale computing. However, the current hybrid or heterogeneous integrated laser sources are suffered from high power consumption and complex fabrication process, which is limited the realization of high computing density and low energy consumption optical computing^[34]. On the other hand, although the state-of-art optical frequency comb can provide hundreds of incoherent wavelength resources without the drawbacks of on-chip lasers^[35], overcoming the large chip area and the heterogeneity of the comb teeth are big challenges^[29,32,36]. At this point, MDM technology can be hybrid with WDM to reduce the multi-wavelength lasers without downsizing the calculation scale or multiplying the original calculation matrix. Meanwhile, there is no need for complicated fabrication processes and additional power consumption^[28,42]. Moreover, due to the limitation that light cannot represent negative values like voltage well, negative domain computation is a crucial challenge in PNN^[16]. At the present stage, a few software and hardware methods have been developed, for example, the universal normalized weight^[27,29], the positive & negative field separate computation^[28,31], the employment of balanced photodetectors^[36,43], etc. Nevertheless, the software solution of universal normalized weight sacrifices the modulation depth and significantly decreases the calculation accuracy, while the positive & negative field separate computation and the employment of BPDs as hardware solutions are not suitable for future large-scale optical calculation due to the demanding experiment setup and the complexity of integration. Hence, it is essential to develop a method to calculate the photonic MVM in the negative domain without deteriorating performance and being amenable to an easy fabrication process at the same time.

In this contribution, we present an optical mode division multiplexing technology inspired solution for small size, low power consumption, and easy integration on-chip PNN accelerator for the first time, which avoids the large chip area, the high-power consumption, and the complex fabrication processes needed for multi-channel integrated light sources by utilizing the dimensional hybrid multiplexing technique. What’s more, we propose a novel transformation mapping approach for optical real-number-field computing to maintain the modulation depth and single-channel detection. With a computing density of 1.37 TOPS/mm², we experimentally demonstrate the optical convolution on a specified image and recognize all desired letter pattern and their corresponding positions in randomly generated grayscale images. In the end, we propose methods for potential future high-accuracy and large-scale extensions.

MAC is the fundamental part of the MVM^[29], the proposed scheme to implement optical MAC-based MVM is shown in Figure 1, which is formed by a three-mode demultiplexer, a 3 × 4 MRR array, and a three-mode multiplexer. Figure 1(b) shows the micrograph of the fabricated device, and the details of the device fabrication can be found in Supplementary Notes 1. The weight kernel is loaded on the MRR array and each element can be adjusted by thermal tuning. Benefiting from WDM and MDM technologies, the input signals which include three modes TE₀ ~ TE₂ and four wavelengths λ₁ ~ λ₄ can orthogonality be transferred in the bus waveguide, then be demultiplexed to the MRR array and be modulated with different weights one by one before they are accumulated together. If the input wavelength of light fits with the resonant wavelength of the specific MRR, the MRR will be in the on-resonance state and almost the whole energy of light would be lost in the MRR, the through port of the MRR will obtain a weight adjustment value of ‘0’, Identically, if the input wavelength is far away from the resonant wavelength, there is no energy loss since the MRR is in off-resonance state, and the corresponding weight adjustment value is ‘1’. The transmission schematic diagram is shown in the lower right corner of Figure 1(a), the red, green, blue, and dark blue arrows corresponding ‘1/8’, ‘1’, ‘1/2’, and ‘0’ weight positions. In other cases, weight values between ‘0’ and ‘1’ can be easily achieved by thermal tuning the resonant wavelength of MRR.

However, due to the inherent limitations of light, it is unable to perform negative values perfectly with intensity. Therefore, a crucial endeavor is to find ways to deal with optical negative computing before we start to perform the photonic accelerator^[16]. Here, we propose a transformation mapping method for optical real-number-field computing in Figure 2. Figure 2(a) shows the basic MAC process, note that the original weights are w_1~9Î [-1,+1], a hypothetical condition that the w₂, w₄, w₆, w₈ < 0 is set, and each bit of the encoded Data 1 ~ 9 is initially set Î [0,1]. Because the negative weights are unable to perform photonic computing directly, the method for mapping the negative weights to the positive domain while maintaining the computing result waveform shape is introduced in Figure 2(b). If the computing weight w < 0, the transformed weight |w| is equal to -w. At the same time, the corresponding data serials for the negative weights (Data 2, Data 4, Data 6, Data 8) are subtracted from the value ‘1’ to be mapped to (1 - Data i). Because all of the original data Î[0,1], (1 – Data i) also Î [0,1]. At this point, the shapes of the original and subtracted data are opposite. Consequently, the multiplied result for [|w|·(1-Data i)] has the same shape as the original computing result (w·Data i) except for an additional bias value (-w). The MAC operation process with transformed weights and mapped data is shown in Figure 2(c). Accordingly, the transformed result’s shape is as same as the original output, while the whole waveform has a bias that can be eliminated by the normalizing process. With this method, the optical MAC operations of real-number in both positive and negative fields can be implemented.

The convolution operation is widely used in image processing, such as image edge detection, image denoising, fast Fourier transform, discrete Fourier transform, and the convolution layer in image recognition^[44,45].

The working principle of the proposed device to realize optical convolution operation for image edge detection is schematically illustrated in Figure 3. Typically, 3 × 3 kernels are used in image convolution operations to achieve relatively accurate calculation results, different kernels represent different convolution functions. In Figure 3(a), the convolution kernels stride on the input image matrix from left to right side and then up to down side, The 3 × 3 data box below the kernels represents the i-th elements of the input Data 1 ~ Data 9. To implement a complete convolution operation with a 3 × 3 kernel, 9 input data serials should be generated, and each data serial will be multiplied with one weight value in the kernel. Each data serial will include part of the data elements in the input image matrix as shown in Figure 3(b). The MAC operation between the kernel which flattened to a 1 × 9 vector and the 9 elements for data serials which formed a 9 × 1 data vector at i-th order can obtain the i-th value (Y_i) of the entire convolution result. The optical convolution logic diagram is shown in Figure 3(c). Encoded Data 1 ~ 9 is converted to vector tracks with the same length so that data at the exact location will process simultaneously. Each track multiplies the corresponding kernel weight and then adds up to get the output vector Y. The output vector finally transforms into the output image matrix and is shown as an output image in Figure 3(d). The detailed 2D convolution method for image processing is introduced in Supplementary Notes 3. The experiment part is shown in Figure 3(e), the encoded Data 1 ~ 9 are modulated on the spitted three wavelength sources in specific modes (Data 1 ~ 3 are modulated in TE₀ but different wavelengths, the same as Data 4 ~ 6 and Data 7 ~ 9) by EOMs. Then, the data are transferred into the accelerator for MAC operation, and record the waveform after the amplification.

To demonstrate the convolution function of the MVM accelerator, the edge detection of an image named ‘Lanzhou University’ is carried out. The original image is firstly converted into a 154 × 215 gray image with 33110 pixels in Figure 4(a). Through the ‘valid’ convolution, where a 3 × 3 convolution kernel is used, the image is convoluted into 152 × 213 = 32376 pixels. Note that the pixels are encoded by the method shown in Figure 3. The time interval of the adjacent pixel is , which means the modulation rate is . Though 3 × 3 convolution kernels are adopted, the max 3 × 4 kernel has 12 weight elements, which leads to computing size . While the footprint (S) of the optical computing core is about , hence the compute density of our proposed optical MVM accelerator is . The on-chip energy consumption achieves 4.48 fJ per operation, the performance details of the proposed accelerator can be found in Supplementary Notes 2.

Four image edge detection convolution kernels are chosen to test the performance of the fabricated device. As one can see in Figure 4, the upper, bottom, left, and right edges of the original image ‘Lanzhou University’ are highlighted in the dark background. The specific convolution operation process and calculated waveforms are shown in Supplementary Notes 3.

Object detection, which aims to find specific objects in the image and provide the location information of each object, has important applications in video analysis and image understanding^[46,47]. In this section, the function of letter pattern detection will be demonstrated.

Figure 5 illustrates the schematic process of how the proposed MVM accelerator can achieve pattern recognition. In that figure, the white pixel represents data ‘0’, and the blue pixel represents data ‘1’. For a given image composed of random binary pixels, the patterns to be detected in the image are ‘A’, ‘b’, and ‘C’. The goal is to recognize these letters and provide their location information. The input image is pre-encoded with 3 × 4 convolution operation and modulated by electro-optical modulators to generate optical Data 1 ~ 12, similar to the processing method adopted in Section 2. The weight matrixes on the right corner of Figure 5(a) are used as the fully-connected layers in the pattern recognition. The encoded i-th element of Data 1 ~ 12 is first flattened into a 1 × 12 vector X_i, then the vector is transmitted to the fully-connected layer and multiplied with the matrix weight. Finally, the output waveform of the fully-connected layer passes a nonlinear function filter to extract the positions of patterns. According to the data encoding of convolution in Figure 3(a), every encoding data has pixels, where m and n are rows and columns of the image and the detail of 2D convolution can be found in Supplementary Notes 3. The position information of one specific pattern is related to the order number of the pixels. In the experimental concept, three WDM and MDM based 3 × 4 MRR arrays are adopted to adjust the fully-connected layer weights of ‘A’, ‘b’, and ‘C’. Each array is worked with three modes (TE₀, TE₁, and TE₂) and four wavelengths of lasers. The real-number field MVM conversion discussed in Section 2 also processes the pattern detection weights on MRR arrays. After detecting the optical power on PD, the output waveform passes the nonlinear function to get the position of each pattern.

In the experiment, four patterns (‘A’, ‘b’, ‘C’, and ‘D’) are chosen to be detected. The experiment result is shown in Figure 6. Figure 6(a) shows the absolute vision of four patterns to be detected, where the blank pixel represents ‘1’ (black pixel) while the blue pixel represents ‘0’ (white pixel) as we have defined before.

To make the research further in-depth and close to practical application, the gray level of image pixels is randomly set in [0,1]. Two 12 × 13 images are stochastically generated before the test. It is easy to find that there are possible positions after the 3 × 4 convolutional encoding. The fully-connected layer has 12 × 4 neurons, and the fully-connected neurons of different patterns are independent, so four MRR arrays with different weights are utilized to represent the 12 × 4 weight matrix W. The input of the data vector X_i of every array is the same to fit the operation of the fully-connected layer’s MVM. The middle of Figure 6(b) shows waveforms before the threshold nonlinear function. The peaks of different colors represent the positions of the corresponding patterns. Then the detected result and their location information are summarized in the table in Figure 6(b). By comparing the detected results with the original two images, one can find that the patterns and their location information are well recognized as expected.

From the results in Figure 6(b), one can see that the proposed system can accurately identify the positions of the corresponding patterns. In some particular positions, if the pattern pixels have a small difference between the target pattern, this position might have a secondary high peak, but after the threshold function, the secondary peaks will turn into ‘0’ in the binary system, which ‘1’ means ‘Yes, one pattern is here’ and ‘0’ means ‘No, the pattern is not here’. The obvious pattern positions and more details can be found in Supplementary Notes 5. The calculation error between the optical results and the target electrical floating point results for four patterns on two test images is shown in Figure 5(c), note the average error , equals to 5-bit computing precision. So to that extent, all of the locations of patterns are effectively detected.

In the previous sections, the mission-critical applications of image processing and CNN on integrated MVM accelerators, including image convolution operation, pattern detection, and multi-layer optical CNN operations are demonstrated successfully. Also, the optical computing is extended to the real-number field with an efficient method. Therefore, the various applications show the potential for complex tasks in various situations based on the proposed and fabricated fundamental MVM accelerator.

For large-scale optical computing extension, the processor can be further improved in kinds of ways. Increasing the number of wavelength sources and optical modes appropriately are effective ways for expanding the computing size as needed ^[29,36]. Besides, because the MVM is based on the fundamental MAC operation, the matrix partition method is also feasible to increase the computing scale, which separates the MVM into several MAC units while the signal sources and device structures are the same^[25,31].

The conceptional structures to implement large-scale computing based on the proposed MVM accelerator are shown in Figure 7. The number of optical wavelengths, modes, and computing units can all be increased to scale up the computing size as shown in Figure 7(a). The input light consists of (j + 1) × k signals, j represents the highest optical mode order and k represents the number of wavelengths. After passing the signal spitter, the input signal is divided into several parts with equal power, each part is transferred into a MAC unit to achieve parallel optical computing. Other multiplexing techniques such as polarization^[13] can also be utilized to increase the computing scale. In these cases, the depth of weights will be related to the thermal modulation accuracy of MRR^[30]. To increase the weight-adjusting accuracy, other weight controllers such as metasurface^[28,29] and interference devices^[24] can also be used.

Benefiting from the orthogonal transmission of optical modes and incoherent wavelengths, the signals in the bus waveguide are mutually independent. For the complicated MVM weight matrix, the MAC units can be cascaded to adjust weights step by step to implement complicated optical computing as shown in Figure 9(b). From the PD at each unit, one can monitor the output power of any MVM step to guide the weight adjustment.

Table 1 lists the comparison of mainstream optical computing solutions. It shows our proposed PNN accelerator reaches a great performance, and can significantly cut down the use of multi-wavelength sources, while not increasing the complexity of weight adjustment.

Table 1 Comparison of our proposed PNN accelerator

Type	Calculation scale	Computing density (OPS/mm²)	Energy /OP	Precision (bits)	Number of wavelengths	Adjust units /weight
MZI-MVM^[24]	4´4	0.56 T^a	30 fJ^a	5-bit	Coherent light	4
MZI-MVM^[48]	2´4	/	3.8 pJ	/	Coherent light	2
OCDC^[25]	1´9	/	/	6-bit	Coherent light	1
PVCA^[36]	3´3	System	1.58 pJ	7-bit	N´N	/
PCM-MVM^[29]	8´8	0.60 T	0.5 pJ	7-bit^c	N´N	1
PMMC-MVM^[28]	2´2	82 T^b	/	7-bit^d	N´N	1
PPU^[32]	2´2	1.04 T	2.38 pJ	6-bit	N´N	1
PTFP^[31]	3´3	0.59 T	33 fJ	3-bit	N´N	1
OCPU^[43]	2´2	12.74 T	4.84 pJ	5-bit	N	N
This work	3´4	1.37 T	4.48 fJ^e	5-bit	N	1

^aThese data can be finished by the experiment setup with existing technologies.

^bData from the outlook of future extension.

^cThe weight adjustment precision.

^dThe theoretical analysis precision from the experiment setup.

^eThe on-chip energy cost.

Fabrication of photonic chip

The proposed optical matrix-vector accelerator is fabricated by electron beam lithography (EBL) and inductively coupled plasma (ICP) etching processes. A silicon-on-insulator wafer with 220 nm silicon in thickness was used. The EBL process was adopted to define the pattern of the waveguides and then the ICP process was used to fully etch the structure of the device. After that, a silicon dioxide layer with 2 μm in thickness was deposited. To tune the optical weights, Ti micro-heaters with 100 nm thick and 5 μm wide were fabricated. To contact the micro-heaters, Au nanowires, and contact pads are deposited. Finally, another 500 nm thick silicon dioxide layer was deposited to protect the device, and contact windows were etched above the Au pads. The detail of the fabrication and design of the device can be found in Supplementary Notes 1.

Experiment setup

Light from a commercial tunable laser (Santec 570A) is injected into the on-chip optical MVM accelerator. An arbitrary waveform generator (Keysight M8196A) is adopted for generating programmed electric signals before they are applied to the high-speed lithium niobate modulator (IXblue MX-LN-40-PD-P-P-FA-FA). The weights of the chip are controlled by a mass-tunable voltage source (Time-Transfer T-MS32T-10CVF). The computed light from the fabricated chip is then amplified and filtered by an EDFA (Keopsys CEFA-C-PB-LP-SM-23-MSA1-B201-FA-FA) and a tunable optical filter (Santec OTF350) respectively. At last, an oscilloscope (Keysight DCA-X 86100D) with PD embedded is used to measure the final computing results. More details about static and dynamic measurements can be found in Supplementary Notes 2.

Data availability

All the data supporting the findings in this study are available in the paper and Supplementary Information. Additional data related to this paper are available from the corresponding authors upon request.

Acknowledgments

This work is partially supported by the National Key R&D Program of China (2022YFB2804202), National Natural Science Foundation of China (NSFC) (62075091, 62205135), China Postdoctoral Science Foundation (2022M711443), Natural Science Foundation of Gansu Province (22JR5RA493), and Australian Research Council (ARC) grants (DP190102773). The authors acknowledge the facilities, and the scientific and technical assistance, of the Micro Nano Research Facility (MNRF) and the Australian Microscopy & Microanalysis Research Facility at RMIT University. This work was performed in part at the Melbourne Centre for Nanofabrication (MCN) in the Victorian Node of the Australian National Fabrication Facility (ANFF).

Author contributions

R.Y., H.X., and Y.T. conceptualized this research. R.Y., Y.J., and X.H. performed the simulation and chip layout of the device, and G.R. contributed to the fabrication. P.Z., L.C, X.Z., and M.Y. assisted in static and high-speed measurement. R.Y. and H.X. conceived the architecture and wrote the manuscript with contributions from Y.T., G.R., and A.M. Y.T. supervised the full project.

Conflict of Interest

The authors declare no conﬂict of interest.

Xu, S. et al. Computer Vision Techniques in Construction: A Critical Review. Arch. Comput. Method E 28, 3383–3397 (2020).
Porciello, J., Ivanina, M., Islam, M., Einarson, S. & Hirsh, H. Accelerating evidence-informed decision-making for the Sustainable Development Goals using machine learning. Nat. Mach. Intell. 2, 559–565 (2020).
Saldanha, O. L. et al. Self-supervised attention-based deep learning for pan-cancer mutation prediction from histopathology. NPJ Precis. Oncol. 7, 35 (2023).
Siriwardhana, Y., Porambage, P., Liyanage, M. & Ylianttila, M. A Survey on Mobile Augmented Reality With 5G Mobile Edge Computing: Architectures, Applications, and Technical Aspects. IEEE Commun. Surv. Tutor. 23, 1160–1192 (2021).
Almalioglu, Y., Turan, M., Trigoni, N. & Markham, A. Deep learning-based robust positioning for all-weather autonomous driving. Nat. Mach. Intell. 4, 749–760 (2022).
Khan, H. N., Hounshell, D. A. & Fuchs, E. R. H. Science and research policy at the end of Moore’s law. Nat. Electron. 1, 14–21 (2018).
Xu, M. et al. Dual-polarization thin-film lithium niobate in-phase quadrature modulators for terabit-per-second transmission. Optica 9, 61 (2022).
Li, M. et al. Lithium niobate photonic-crystal electro-optic modulator. Nat. Commun. 11, 4123 (2020).
Sivalingam, K. M. A Comparison of Bit-Parallel and Bit-Serial Architectures for WDM Networks. Photonic Netw. Commun. 1, 89–103 (1999).
ichi Kitayama, K. et al. Novel frontier of photonics for data processing—Photonic accelerator. APL Photonics 4, 090901 (2019).
Li, C., Liu, D. & Dai, D. Multimode silicon photonics. P. Soc. Photo-opt. Ins. 8, 227–247 (2018).
Xu, H., Dai, D. & Shi, Y. Silicon Integrated Nanophotonic Devices for On-Chip Multi-Mode Interconnects. Applied Sciences 10, 6365 (2020).
Goossens, J.-W., Yousefi, M. I., Jaou¨en, Y. & Hafermann, H. Polarization-division multiplexing based on the nonlinear Fourier transform. Opt. Express 25, 26437 (2017).
Yan, Y. et al. High-capacity millimetre-wave communications with orbital angular momentum multiplexing. Nat. Commun. 5, 4876 (2014).
Zhou, H. et al. Photonic matrix multiplication lights up photonic accelerator and beyond. Light Sci. Appl. 11, 30 (2022).
Shastri, B. J. et al. Photonics for artificial intelligence and neuromorphic computing. Nat. Photonics 15, 102–114 (2021).
Goodman, J. W., Dias, A. R. & Woody, L. M. Fully parallel, high-speed incoherent optical method for performing discrete Fourier transforms. Opt. Lett 2, 1 (1978).
Wright, L. G. et al. Deep physical neural networks trained with backpropagation. Nature 601, 549–555 (2022).
Zhu, H. H. et al. Space-efficient optical computing with an integrated chip diffractive neural network. Nat. Commun. 13, 1044 (2022).
Qu, Y. et al. Inverse design of an integrated-nanophotonics optical neural network. Sci. Bull. 65, 1177–1183 (2020).
Wang, T. et al. An optical neural network using less than 1 photon per multiplication. Nat. Commun. 13, 123 (2022).
Zhou, T. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photonics 15, 367–373 (2021).
Fu, T. et al. Photonic machine learning with on-chip diffractive optics. Nat. Commun. 14, 70 (2023).
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photonics 11, 441–446 (2017).
Xu, S. et al. Optical coherent dot-product chip for sophisticated deep learning regression. Light Sci. Appl. 10, 221 (2021).
Sludds, A. et al. Delocalized photonic deep learning on the internet’s edge. Science 378, 270–276 (2022).
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
Wu, C. et al. Programmable phase-change metasurfaces on waveguides for multimode photonic convolutional neural network. Nat. Commun. 12, 96 (2021).
Feldmann, J. et al. Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52–58 (2021).
Zhang, W. et al. Silicon microring synapses enable photonic deep learning beyond 9-bit precision. Optica 9, 579 (2022).
Xu, S., Wang, J., Yi, S. & Zou, W. High-order tensor flow processing using integrated photonic circuits. Nat. Commun. 13, 7970 (2022).
Bai, B. et al. Microcomb-based integrated photonic processing unit. Nat. Commun. 14 (2023).
Cheng, J. et al. A small microring array that performs large complex-valued matrix-vector multiplication. Front. Optoelectron. 15, 66 (2022).
Zhou, Z. et al. Prospects and applications of on-chip lasers. eLight 3, 1 (2023).
Zhang, M. et al. Broadband electro-optic frequency comb generation in a lithium niobate microring resonator. Nature 568, 373–377 (2019).
Xu, X. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021).
Deng, L., Xu, Y., Jin, R., Cai, Z. & Liu, Y. On-Demand Mode Conversion and Wavefront Shaping via On-Chip Metasurfaces. Adv. Opt. Mater. 10, 2200910 (2022).
Su, Y., He, Y., Chen, H., Li, X. & Li, G. Perspective on mode-division multiplexing. Appl. Phys. Lett. 118, 200502 (2021).
Han, X. et al. On-Chip Non-Blocking Optical Mode Exchanger for ModeDivision Multiplexing Interconnection Networks. J. Lightwave Technol. 39, 6563–6571 (2021).
Han, X. et al. Integrated non-blocking optical router harnessing wavelength- and mode-selective property for photonic networks-on-chip. Opt. Express 29, 1251 (2021).
Zhao, W. et al. Silicon Multimode Waveguide Crossing Based on Anisotropic Subwavelength Gratings. Laser Photonics Rev. 16, 2100623 (2022).
Luo, L.-W. et al. WDM-compatible mode-division multiplexing on a silicon chip. Nat. Commun. 5, 3069 (2014).
Meng, X. et al. Compact optical convolution processing unit based on multimode interference. Nat. Commun. 14, 3000 (2023).
Hirschman, I. & Widder, D. The Convolution Transform. Dover Books on Mathematics (Dover Publications, 2012). URL https://books.google.com/books?id=TAqbPjWNQEQC.
Pang, Y., Sun, M., Jiang, X. & Li, X. Convolution in convolution for network in network. IEEE Trans. Neural Networks Learn. Syst. 29, 1587–1597 (2018).
Zhao, Z.-Q., Zheng, P., Xu, S.-T. & Wu, X. Object Detection With Deep Learning: A Review. IEEE Trans. Neural Networks Learn. Syst. 30, 3212– 3232 (2019).
Zaidi, S. S. A. et al. A survey of modern deep learning based object detection models. Digit. Signal Process. 126, 103514 (2022).
Huang, Y. et al. Easily scalable photonic tensor core based on tunable units with single internal phase shifters. Laser Photonics Rev. 2300001 (2023).

Supplementary Notes 1-5 are not available with this version

Optical mode division multiplexing inspired photonic neural network accelerator

Status:

Version 1

Abstract

Figures

1. Introduction

2. Optical real number field matrix-vector multiplication

3. Optical convolution

4. Optical pattern detection

5. Discussion

6. Materials and methods

Declarations

References

Supplementary Information

Status:

Version 1