The advancement in the field of 3D integration circuit technology leads to new challenges for quality assessment of interconnects such as through silicon vias (TSVs) in terms of automated and time-efficient analysis. In this paper, we develop a fully automated high-efficient End-to-End Convolutional Neural Network (CNN) model, utilizing two sequentially linked CNN architectures, suitable to classify and locate thousands of TSVs as well as provide statistical information. In particular, we generate interference patterns of the TSVs conducting a unique concept of Scanning Acoustic Microscopy (SAM) imaging. Scanning Electron Microscopy (SEM) is used to validate and also disclose the characteristic pattern in the SAM C-scan images. By comparing the model with semi-automated machine learning approaches its outstanding performance is illustrated, indicating a localisation and classification accuracy of 100% and over 98%, respectively. The approach is not limited to SAM-image data and presents an important step towards zero failure management.
Article
An End-to-End Convolutional Neural Network for Automated Failure Localisation and Characterisation of 3D Interconnects
https://doi.org/10.21203/rs.3.rs-2495953/v1
This work is licensed under a CC BY 4.0 License
Journal Publication
published 08 Jun, 2023
You are reading this latest preprint version
Imaging-based techniques are highly important for modern non-destructive failure analytics1 in various fields ranging from aerospace, rail-track inspection, civil engineering, automotive industry, power generation to microelectronics2. Machine Learning (ML) algorithms provide novel opportunities for an efficient failure analysis of the generated complex data sets that previously relied mainly on human expertise3. Recently, research has been conducted4–7 by applying various ML models in 3D integration components which enjoy high interest in microelectronics industry. Essential for the application within an industrial environment are fully automated models, which do not necessarily rely on specific training features. Recently, first effort has been made for the application of ML-based testing as shown in 4,6,7. However, mainly semi-automated approaches, which lack in their application for a generalized analysis due to the necessary specific feature definition for the training have been demonstrated so far. Supervised semi-automated ML models like K - Nearest Neighbours (KNN) and a Random Forest classifier are used for example to detect voids in through silicon vias (TSVs) processed in 3D integrated circuit components4. As shown, for instance in4, such models use specific feature extraction for training, like the High-Frequency Structural Simulator (HFSS) data for the “TSV with” and “TSV without” void.. In6 a similar semi-automated approach is used, to identify functional failures including open and short circuits for TSVs. Further in this context general regression neural networks are discussed in7 to detect defects in solders using SAM.
The Convolutional Neural Network (CNN) is a well-known deep learning ML architecture capable to extract multi-level features from an image8. The main advantage of CNN lies in its ability to recognize patterns or relevant features directly from the raw pixels by exploring temporal and spatial correlation in data without any complex pre-processing9,10. Recently in5, a CNN-based model has been utilized to predict the condition of a single micro-bump after the reflow process based on an image data taken before the reflow process by 3D X-ray tomography.
Modern failure inspection of TSVs demands cost- and time-efficient characterization of hundreds or even up to thousands of TSVs11,12 including the concomitant statistical information, the localization and status of the individual TSV covering the entire geometry with its bottom and sidewall as well as the classification of the TSV failure. There are various defect types related to TSVs including voids resulting from electroplating13, delaminations arising due to thermal expansion mismatch14, cracks resulting from global stress in the die warpage15 and so on16,17. To detect such defects, non-automated laboratories techniques like Scanning Electron Microscopy (SEM), X-ray Computer Tomography (XCT), Emission Microscopy (EMMI) or automated techniques18,19 such as Electrical Measurements (EM), Automatic Optical Microscopy (AOM), and Scanning Acoustic Microscopy (SAM) are mainly used20. All these techniques have respective advantages as well as disadvantages that restricts their applications. For instance, EM displays a fast and common method, however fails to localize the failure within the TSVs20,21. AOM is only suitable to detect bottom defects20,22 but fails for defects in the sidewall. SEM provides high-resolution image data for the sidewall and the bottom of the TSVs. Nevertheless, the latter is inadequate for high-throughput inspection and ineligible to provide statistical information, due to the very time-consuming data aquisition23,24. µ-XCT or X-ray microscopy (XRM) show limitations with respect to the necessary long scanning times to gain sufficient resolution and statistical output5,25,26,15. EMMI can only detect defects with an electrical signature and fails in detecting defects without any electrical signature27. Scanning acoustic microscopy (SAM) displays a non-destructive technique28 capable to characterize time- and cost-efficiently large areas in the field of microelectronics29. Nevertheless, the main challenge of this method lies in the limited resolution and contrast as well as in the post-processing of the generated image data set, namely to extract efficiently knowledge about the location of the individual failure but also about the statistical distribution of the failures within the array including the type of the defect class. This requires in general a careful manual inspection of the collected image data. Such a manual inspection highly depends on the experience of the human user, is therefore subjective and in addition prone to error.
In this paper, we conduct a unique scanning acoustic microscopy (SAM) approach and develop an End-to-End Convolutional Neural Network (E2E-CNN) workflow to (1) efficiently characterize up to thousands of TSVs on wafer level including concomitant statistical information, (2) localize faulted and non-faulted TSVs, and (3) classify the individual TSVs according to their degree of fault level. We reveal by comparing the gained SAM-data with correlated scanning electron microscopy (SEM) that the SAM technique is suitable to provide information from the bottom as well as from the sidewall of the TSV. The fully automated E2E-CNN network, using two sequentially linked CNN architectures, assures an accuracy for the detection and classification of the TSVs of up to 100% and 98%, respectively. Furthermore, we discuss the developed E2E-CNN algorithm with semi-automated binary TSV classification using Multi-Layer Perceptron (MLP), Decision Tree, and Random Forest and show its superiority. Notably, the presented novel approach is not limited to SAM-based image data, but rather displays a general approach applicable to other imaging methods e.g. µ-XCT, SEM, optical microscopy etc.
Scanning acoustic microscopy - experiment and statistical-relevant image data. We apply scanning acoustic microscopy to scan the TSV arrays. In particular, we conduct a unique technique utilizing special acoustic lenses with a nominal frequency of 100 MHz and an opening angle of 60°, suitable to fulfill the demands for modern TSV inspection and analysis, see Method section for further details. The methodology represents an extremely time and cost-efficient approach with the ability to collect a statistically relevant amount of TSVs with sufficient resolution for further image post-processing and image quantification. Figure 1A shows exemplarily the projection of the ultrasound data onto the x-y plane to create the so-called C-scan image of a quarter-wafer piece. The region of interest (ROI) of the C-scan contains approximately 10,000 TSVs. Each TSV can be associated with a characteristic pattern generated by defocusing the lens, see Method section. For the better visualization of the characteristic patterns, the region is divided into a C-scan image patch illustrating about 800 TSVs (ROI-1), as shown in Fig. 1B. It can be further subdivided into an ROI (ROI-2) with six TSVs (Fig. 1C). In Fig.1D we present exemplarily two patterns generated by defocusing the acoustic lens and exciting Rayleigh waves30. The two patterns indicate a TSV with and without an inhomogeneity, marked with a ‘chartreuse’ green and a red rectangle, respectively.
Workflow of the End-to-End CNN model approach. Figure 2 illustrates the automated TSV failure analysis workflow with respect to training and testing, based on the extracted C-Scan SAM data. The workflow consists of two sequentially linked CNN architectures, which we appoint as End-to-End Convolutional Neural Network (E2E-CNN). The first CNN (CNN1) is dedicated to localize the TSVs, whereas the second CNN (CNN2) is capable to classify thousands of TSVs, see further details in the method section.
As an input for CNN1, SAM C-scan image data that is not limited by the image size is used. The output provides image patches with characteristic TSV patterns. CNN2 classifies the TSVs according to their quality and utilizes the output of CNN1 as an input. In order to train CNN1, we use two sets of labelled data incorporating C-scan image patches with and without TSVs, respectively. The output of CNN1 detects all the characteristic patterns which are marked in ‘chartreuse’ color for the exemplary ROI with 36 TSV. Whereas the output of CNN2 is color-coded, based on the quality of the TSVs. We train CNN2 with five different classes indicated by class 1 to 5. The different classes are assigned according to patterns found in the C-scan data. Within the exemplary output of CNN2, shown here, 33 TSVs are assigned to class 1, two to class 2 and one TSV to class 3. Class 4 and 5 is not found in the exemplary ROI.
Efficient TSV localization based on a non-sequential sliding window detection for CNN1. There have been several advancements in the field of computer vision for object detections31. Many authors proposed object localization techniques like CNN-based segmentation, sliding window approach and so on32–34. Figure 3A shows an illustration for the sliding window detector processing utilized for the TSV localization. A window with a size of 100 × 100 pixels is chosen to slide over the C-scan image with strides Sx and Sy in x and y directions. This specific window size fits well to cover the characteristic patterns of every TSV. For the training of CNN1, each of these windows is individually fed to locate the TSVs in the SAM C-scan images. For the test images two sets are generated. The first set contains the C-scan images of TSVs in the center of the bounding box. For the second set the image with background and/or any image with TSVs not centred (see supplementary figure S3) are used. Since there are only two categorical features in the dataset for CNN1, by using hot encoding35,36, we assign a binary code ‘1’ to the first set and a binary code ‘0’ to the second set while training.
We conduct a non-sequential sliding window detector approach as illustrated in Fig. 3B, see Method section for further details. A major disadvantage of a sequential approach34, see also Supplementary materials, Fig. S6, is the computational cost as well as the time consumption for the training and testing result. We show that by using a convolutional layer at the end node34 the training time can be reduced from hours to minutes. While the testing procedure takes place, the model predicts multiple bounding boxes37 based on whether the extracted features from the window belong to the first set or the second set, i.e. 1 or 0, respectively. Non-Maximal Suppression (NMS) is applied (see supplementary Fig. S4) to find the predictions with the highest confidence score and gives the best bounding box with a size of 100 × 100 pixels, defining the TSV as an object. The prediction of CNN1 in a C-scan image exemplary with three TSVs is shown in Fig. 3B. Detected TSVs are represented in ‘chartreuse’ colored bounding boxes. The predictions of CNN1 in a larger C-scan image with 864 TSVs is shown in Fig. S5.
Training of the CNN2 and Input for TSV Classification. The second CNN (CNN2) classifies the located TSVs in the SAM C-scan image. Therefore, the inputs of CNN2 are the predictions from CNN1. According to the C-scan data obtained from the SAM characterization, we define five classes for the training datasets, see Fig. 4. The first TSV-class exhibits concentric circular or non-disturbed fringes at the TSV location within the C-SAM image. The second TSV-class, indicate a single inhomogeneity within the circular fringes at various positions along the circumference of the fringes. The third TSV-class represents patterns with multiple inhomogeneities along the fringe circumference. Class 4 and class 5 represent patterns originating from water bubbles and further scanning artefacts e.g. originating from gating error or high scanning speed. Further details with respect to the architecture of the CNN2 and corresponding layer parameters are detailed in supplementary Fig. S7 and Method section.
Validation of the E2E-CNN. Figure 5A and B illustrate the training and testing accuracy for the two CNN models. For the TSV-localization (CNN1) we achieve an accuracy of 100% for the testing and training, that is we are able to detect every TSV from the SAM C-scan image, see Fig.5A. Figure 5B provides an accuracy for CNN2 of over 98% dedicated for the TSV-classification alone with respect to training and testing. Further, to show the performance of the E2E-CNN model, we plot the training and validation loss as a function of epochs for CNN1 and CNN2 in Supplementary Fig. S8.
In Fig. 5C a representative SAM C-scan image displays the fully automated localization and classification of TSVs, exemplarily for class 1, 2 and 3. Images for class 1 to 3 with higher magnification are shown in Fig.5D, indicating the different patterns also shown in Fig.4. For further validation, we compare the SAM C-scan images for class 1, 2 and 3 with correlated SEM characterization results. As indicated by the SEM data in Fig. 5E, no inhomogeneity on the sidewall of the TSV nor at the bottom is detected. This matches with the observation made for the C-scan SAM image where no inhomogeneity in the fringes is exhibited. The SEM data for class 2 shows a large accretion on the bottom of the TSV as well as on the sidewall. Here, a characteristic pattern in the SAM-image indicating a single inhomogeneity within the fringes is shown, see Fig. 5C. For class 3, the SEM data shows a delamination within the sidewall, see Fig. 5E. Here, the C-scan SAM image shows for class 3 a pattern with multiple inhomogeneities in the fringes, see Fig. 5C. According to the correlated SEM data, a clear assignment of the different C-scan patterns can be made.
The detection and quality prediction of 864 TSVs from SAM C-scan images from the wafer is shown in supplementary (Fig. S9).
Comparison of model performance between automatic E2E-CNN and semi-automated ML models. In the following, we compare the developed E2E-CNN model with the semi-automated ML-models. For the semi-automated models, we utilize MLP, Decision Tree and Random Forest, as shown in Table 1. For the semi-automated analysis, to detect the TSVs, it is necessary to apply a geometry based pattern recognition algorithm like the circular Hough transform38,39. The data labelling applied for the training and feature extraction steps are the same for MLP, Decision Tree and Random Forest. For the training of the semi-automated ML model, we define two TSV configurations. The first configuration shows TSVs with non-disturbed fringes and the second one TSVs with disturbed fringes in the SAM C-scan images, see S10.
Figure 6A illustrates the TSV localization for the semi-automated ML analysis. Patches with a size of 100 × 100 pixels showing the characteristic patterns are used, followed by the detection of TSVs using circle Hough transform, see Fig.6A. For the extraction of relevant feature we compare two procedures, namely the Canny Edge Detection (CED)40 and further developed a unique way of segmentation using the Fringe Segmentation Technique (FriST), see further details in Supplementary figures S11, S12 and Method section.
For the binary classification of TSVs, the feature extraction using CED or FriST is followed by the dimensionality reduction using Principal Component Analysis (PCA), see Fig. 6B. By applying PCA, we select the most important features from the output of the CED or FriST as an input to train the model, see Methods for further details. The model performance from all investigated models is summarized in Table 1.
The use of the FriST technique shows for all the three semi-automatic models an improvement in accuracy over the CED technique (see supplementary figure S13) due to the specific extraction of desired features for training.
However, a general disadvantage for the semi-automatic model concerns the requirement for a specific feature extraction to train and test the model. Here, the quality and resolution of the SAM C-scan images is crucial for the subsequent labeling of the pattern associated with the TSV. Therefore, the semi-automated models do not provide an optimal solution when it comes to detecting and classifying large statistics of TSVs, since by increasing the ROI, resolution and contrast will decrease.
Clearly the E2E-CNN workflow, which does not rely on any manual feature extraction technique, outperforms the semi-automated ML-based prediction model performance with respect to testing time and accuracy, as shown in table 1. Indeed, none of the semi-automated models reaches an accuracy of over 90 % and testing times below 10 min.
Table 1. Comparison of TSV classification performance using E2E-CNN and semi-automated TSV classification models
Comparison between the semi-automated ML models (MLP, DT & RF) and the developed fully automated E2E-CNN model for TSV classification from a SAM C-scan image data with 96 TSVs for validation. For the necessary feature extraction of the semi-automated model, we use the CED and the FriST techniques.
Statistical analysis of SAM C-scan images obtained from the E2E-CNN model. In the following, we utilize the developed E2E-CNN model to highlight the statistical possibilities for the failure analysis. Figure 7a illustrates the defect map from the state-of-the-art automatic optical microscopy (AOM) inspection. The black dots mark in Fig. 7A are the position of probable defect locations. For comparison with the SAM measurements and incorporated E2E-CNN model we select four ROIs, labeled with A, B, C and D at distinct wafer locations. We evaluate the statistics with respect to the occurrence of TSVs with and without inhomogeneities. Each individual ROI selected for the underlying analysis consist of 576 TSVs. Figures 7B and C illustrate exemplarily the defect map obtained from the SAM image data and subsequent E2E-CNN analysis for ROI D and a further magnification for ROI D-1. The latter indicates the TSV location as well as TSV classification according to class 1 - 5. Further C-scan with prediction results based on the E2E-CNN model for ROI A, ROI B and ROI C are shown in Supplementary Material S14.
Notably, according to Fig.7D the extracted statistics illustrates a similar trend for the AOM- and the SAM-based method. ROI C shows for both approaches the highest count with inhomogeneous TSVs and ROI D indicates the lowest one. The depicted results are summarized in Table 2. However, the SAM-based inspection utilizing the E2E-CNN model depicts a higher number of TSVs with inhomogeneities than the optical inspection.
Further, we provide in Fig. 7E statistical information with respect to different classes predicted by the E2E-CNN model for ROI A-D. Indeed, the model shows a high count of detected inhomogeneities with class 2. Class 2 as indicated by Fig. 5 shows a mixture of sidewall and bottom defects. Further, it displays the class with the highest counts. We argue that the observed difference between AOM and SAM is explainable due to the restricted ability of the optical inspection to detect inhomogeneities at bottom and not in the sidewall. The results indicate that there is a higher amount of sidewall than bottom inhomogeneities present in the TSV array.
Table 2. Statistics showing the different classes 1 to 5 extracted from the E2E-CNN model for ROI A, ROI B, ROI C and ROI D in comparison with the AOM inspection result.
In conclusion, we have developed a convolutional neural network-based workflow enabling the (1) characterization of thousands of TSVs on wafer level, (2) localization of defective TSVs, (3) classification of the individual TSVs according to their degree of defect level and generation of statistical information about the classified TSVs. We use a unique SAM approach to generate cost and time efficiently the image data of the industrial relevant TSV arrays consisting of up to thousands of TSVs. Compared to state-of-the-art automated optical microscopy inspection, the SAM approach is also capable not only to retrieve information from the bottom but also from the sidewall of the TSV. The fully automated E2E-CNN workflow, provides an accuracy for the localization and classification of about 100% and 98%, respectively. Moreover, we explore the possibilities of TSV detection using further machine learning approaches like MLP, Decision Tree and Random Forest models in comparison with the E2E-CNN. Indeed, the analysis shows that those ML approaches cannot compete with the developed E2E-CNN model in terms of time, cost and accuracy. Training as well as testing time for the semi-automated models are time consuming, because specific feature extraction procedures for both training and test images are essential. A major disadvantage is that necessary pattern recognition algorithms strongly depend in general on the quality of the image data. Therefore, sufficient image resolution and contrast to detect the TSV is mandatory. Pre-processing of the gained SAM C-scan image using pixel brightness transformations like histogram equalisation41or by applying different thresholding techniques such as adaptive thresholding42, binary thresholding43 may help to improve the image quality for further processing. However, our high accuracy model eliminates any time-consuming requirement for manual inspection or such pre-processing of the C-scan images. This makes the E2E-CNN model highly applicable in analyzing larger C-scan images with characteristic patterns of thousands of TSVs. Further, the presented E2E-CNN workflow for automated failure inspection is not limited to the analysis of SAM image data but rather can be applied also for other imaging methods.
Sample. Through Silicon Vias (TSV) are key components for 3D integration technology that plays an important role in miniaturization and improving the functionality of microelectronic devices12,44. TSVs enable an electrical connection through different layers of the 3D stacks. These metallized vias are kept by etching holes into the silicon, and subsequently filling or coating with a conductive material – closed and open TSVs, respectively. Even though closed TSV design has very low contact resistance, they suffer from a high degree of mechanical stress due to the mismatch of coefficient of thermal expansion between silicon and the filling material45. Therefore, tungsten-lined open TSV replaces the closed TSV technology when thermal expansion is a concern46. In this work, we use an open TSV technology with a 100 µm diameter and 250 µm depth as shown in Fig. S1. One potential issue associated with an open design is the high residual stress of tungsten after the deposition process, which can lead to cracks, delamination, or accretions either on the sidewall or on the TSV bottom45 depending on the quality of underlying layers. For this research, ams-OSRAM AG, Premstaetten, Austria, provided wafers with artificially induced defects in TSVs by alterations in the normal fabrication process. These TSVs with unknown defects at unknown locations are detected non-destructively using SAM.
Experimental Setup. The SAM used for this study is a modified setup from PVA TePla Analytical Systems GmbH, Germany, with an ADC card of 8-bit resolution, sampling rate of 5 Giga samples/S, and a tone burst configuration with 1 ns burst-length. Figure S2A and B show the SAM operating in conventional mode (focused lens) and our approach where the transducer utilizes a defocused acoustic lens, respectively. In both setups, the piezoelectric transducer is operating in the pulse-echo mode capable of generating and receiving ultrasound signals. The major difficulty for defect detection in TSVs lies in the necessary large penetration depth and resolution in the low mm-regime. The penetration depth and the area of the sample that can be examined decreases with the square of frequency30. Therefore, we conduct a unique method to gain information beyond the lateral resolution limit of the transducer. For this purpose, an acoustic lens with 100 MHz with a large opening angle (60°) to excite so-called Rayleigh waves is used30. The opening angle of the lens in our experiments is chosen in such a way to be larger than the critical Rayleigh angle of Si (17°). The reflections from the sample are analysed using the software called ‘WinSAM’ provided by PVA TePla. Figures S2. C and S2. D show the C-scan images of the TSV with and without an inhomogeneity obtained by the conventional way (Z0) and defocused with Z < Z0, respectively. The lens at a position greater than Z0 (defocused) gives indirect information about the TSVs within the C-scan image in the form of characteristic patterns or fringes sensitive to the TSV quality, see also the red arrows in Fig. S2 D. The scan rate is about 1000 TSVs per 45 minutes with a resolution of 2 µm/pixel.
Machine Learning (ML): We classify the characteristic C-scan SAM patterns using semi and fully automated ML models. For the fully automated model an End-to-End Convolutional Neural Network (E2E-CNN) with two CNN architectures, indicated by CNN1 and CNN2 is dedicated to localize and classify the TSVs, respectively.
(1) Fully automated ML model:
CNN1: The architecture of the CNN1 comprises three convolutional layers with 32, 32, 64 filters and two max pooling layers arranged in a stacked sequence. Each of these convolutional layers have a kernel size of 3×3. The output layer of CNN1 represents a convolutional layer with sigmoid activation function47 that predicts in the range of zero to one, depending on whether the TSV is detected or not. The output convolutional layer consists of one filter with a kernel size of 21×21. To train CNN1, we use a dataset with overall 20.000 images. We split the entire image data of CNN1 with 70 % for the training and 30 % for the testing to evaluate the model performance. That is, the training batch and testing batch consists of 14,000 and 6,000 images, respectively.
CNN2: It consists of six convolutional layers with 32, 32, 64, 64, 64, 128 filters and three max pooling layers. The output layer of CNN2 is a fully connected layer with a softmax activation function47, which assigns decimal probabilities to each class. The whole dataset of CNN2 consists of 7,000 images with 100×100 pixel. It shows five output class labels representing the different characteristic patterns illustrated in the C-scan image, Fig. 4. From a dataset of 6865 images, 5,151 images are used for training the model and 1,714 images for testing the model. As described above, the input to CNN2 is a C-scan image patch with 10,000 pixel-values, i.e. image patches with the characteristic pattern of each individual TSVs. The input is convolved with two 32 3×3 filters. This reduces the dimension of input to 96×96×32. The third layer is a max-pooling layer with stride 2. The following three layers are convolutional layers that are stacked up and learn 64 features through a 3×3 filter. The output of these layers is of size 42×42×64 and this is forwarded to next max pooling layer. The eighth layer of CNN2 is another convolutional layer that learns 128 features through a 3×3 filter. The output of following max pooling layer is flattened and provide as input to the dense layer with five output nodes. All the convolutional layers of CNN2 is with a stride=1 and Relu activation.
(2) Semi automated ML model:
The input for the feature extractors are 3044 SAM C-scan images of TSVs with 10,000 pixels, i.e., a bounding box with characteristic pattern of TSVs. This bounding box is obtained from the center of TSVs detected using Hough circles. Two sets of features are extracted from this image patches using approach 1 (CED) and approach 2 (FriST). For the training of the MLP, DT and RF models as shown in Table 1, we use 70 % of total dataset and 30% of total dataset for testing the model. For the training and testing of the Decision Tree and Random Forest, we extracted the same features as for the MLP using CED and FriST techniques. The modelled MLP consists of an input layer with 1000 nodes, hidden layers with 250, 125, 50 nodes, and one output node. The solver used for weight optimization is Limited Memory Broyden Fletcher Goldfarb Shanno (LBFGS) and cross-entropy for measuring the penalty associated with the predictions (loss function).
Canny Edge Detection (CED): Before extracting any edge information from TSV fringes, we applied the Gaussian filter inorder to remove any scanning artefacts or noises present on the image. Then we detected fringe edges from this by applying a CED algorithm. The edge features thus obtained are further narrowed by applying PCA to this data.
Fringe Segmentation Technique (FriST): Here, the bounding box contains the pre-processed characteristic patterns using adaptive thresholding. Then a circle with unit radius is considered at the center of the pre-processed image. The radius of this circle is increased iteratively until it reaches the end of the bounding box (see supplementary figure S11). From the C-scan images, the information concerning the TSVs mainly lies in the inner two fringes. At each step, the total number of black pixels that lies in the circumference of this circle is plotted. The first and the second peaks in the plot correspond to the fringes of interest and therefore, retain those areas of image segments by setting all the pixels lying beyond these two peaks to 255 (see supplementary figure S11 and S12). These steps are iterated sequentially for each TSV to extract the relevant features from the fringes. The relevant ROIs segmented using FriST technique is also fed to PCA.
Image pre and post-processing: For the E2E-CNN model no pre or post processing of SAM C-scan images is required whereas in semi-automated TSV localisation and classification, depending the quality of SAM C-scan image, we need to prepare the image using various image processing techniques like contrast enhancement, filtering, thresholding and so.
Data availability
All data that support the findings of this study are available from the corresponding author upon reasonable request.
Code availability
All code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
We acknowledge the financial support by Die Österreichische Forschungsgesellschaft (FFG) under Bridge Young Scientist, Proj. No. 872629, “REFORM”. We acknowledge support from M. Gritzner, ams-OSRAM AG for SEM measurements.
Contributions
R.B., P.P. planned and performed the ML workflow. P.P. developed the CNN architecture under supervision of R.B.. J.S. provided the samples. R.B. planned the SAM measurements. P.P. conducted the SAM measurements. I.W. supported the modification of the SAM. P.P. performed the analysis under supervision of R.B.. R.B. and P.P wrote the paper. All authors discussed the results and commented on the paper.
- Raj, B., Jayakumar, T. & Thavasimuthu, M. Practical non-destructive testing. (Woodhead Publishing, 2002).
- Kumar, S. & Mahto, D. G. Recent trends in industrial and other engineering applications of non destructive testing: a review. Int. J. Sci. \& Eng. Res. 4, (2013).
- Virkkunen, I., Koskinen, T., Jessen-Juhler, O. & Rinta-Aho, J. Augmented ultrasonic data for machine learning. J. Nondestruct. Eval. 40, 1–11 (2021).
- Huang, Y.-J., Pan, C.-L., Lin, S.-C. & Guo, M.-H. Machine-learning approach in detection and classification for defects in TSV-based 3-D IC. IEEE Trans. Components, Packag. Manuf. Technol. 8, 699–706 (2018).
- Hsu, P.-N. et al. Artificial intelligence deep learning for 3D IC reliability prediction. Sci. Rep. 12, 1–7 (2022).
- Liu, H., Fang, R., Miao, M., Yang, Y. & Jin, Y. Defect detection for the TSV transmission channel using machine learning approach. in 2019 IEEE 69th Electronic Components and Technology Conference (ECTC) 2168–2172 (2019).
- Wang, Z., Liu, X., He, Z., Su, L. & Lu, X. Intelligent detection of flip chip with the scanning acoustic microscopy and the general regression neural network. Microelectron. Eng. 217, 111127 (2019).
- Albawi, S., Mohammed, T. A. & Al-Zawi, S. Understanding of a convolutional neural network. in 2017 international conference on engineering and technology (ICET) 1–6 (2017).
- Aloysius, N. & Geetha, M. A review on deep convolutional neural networks. in 2017 international conference on communication and signal processing (ICCSP) 588–592 (2017).
- Gu, J. et al. Recent advances in convolutional neural networks. Pattern Recognit. 77, 354–377 (2018).
- Koester, S. J. et al. Wafer-level 3D integration technology. IBM J. Res. Dev. 52, 583–597 (2008).
- Motoyoshi, M. Through-silicon via (TSV). Proc. IEEE 97, 43–48 (2009).
- Shen, W.-W. & Chen, K.-N. Three-dimensional integrated circuit (3D IC) key technology: through-silicon via (TSV). Nanoscale Res. Lett. 12, 1–9 (2017).
- Lu, K. H. et al. Thermal stress induced delamination of through silicon vias in 3-D interconnects. in 2010 Proceedings 60th Electronic Components and Technology Conference (ECTC) 40–45 (2010).
- Ranganathan, N., Prasad, K., Balasubramanian, N. & Pey, K. L. A study of thermo-mechanical stress and its impact on through-silicon vias. J. micromechanics microengineering 18, 75018 (2008).
- Tsai, M., Klooz, A., Leonard, A., Appel, J. & Franzon, P. Through silicon via (TSV) defect/pinhole self test circuit for 3D-IC. in 2009 IEEE International Conference on 3D System Integration 1–8 (2009).
- Chakrabarty, K., Deutsch, S., Thapliyal, H. & Ye, F. TSV defects and TSV-induced circuit failures: The third dimension in test and design-for-test. in 2012 IEEE International Reliability Physics Symposium (IRPS) 5F--1 (2012).
- Dom, B. E. & Brecher, V. Recent advances in the automatic inspection of integrated circuits for pattern defects. Mach. Vis. Appl. 8, 5–19 (1995).
- Bazu, M. & Bajenescu, T. Failure analysis: A practical guide for manufacturers of electronic components and systems. vol. 4 (John Wiley \& Sons, 2011).
- Kozic, E. et al. Metallization defect detection in 3D integrated components using scanning acoustic microscopy and acoustic simulations. Microelectron. Reliab. 88, 262–266 (2018).
- Chen, Q., Huang, C., Wu, D., Tan, Z. & Wang, Z. Ultralow-capacitance through-silicon vias with annular air-gap insulation layers. IEEE Trans. Electron Devices 60, 1421–1426 (2013).
- Abd Al Rahman, M. & Mousavi, A. A review and analysis of automatic optical inspection and quality monitoring methods in electronics industry. IEEE Access 8, 183192–183271 (2020).
- Reimer, L. Scanning electron microscopy: physics of image formation and microanalysis. at (2000).
- Zhou, W., Apkarian, R., Wang, Z. L. & Joy, D. Fundamentals of scanning electron microscopy (SEM). in Scanning microscopy for nanotechnology 1–40 (Springer, 2006).
- Pacheco, M. & Goyal, D. X-ray computed tomography for non-destructive failure analysis in microelectronics. in 2010 IEEE International Reliability Physics Symposium 252–258 (2010).
- Zschech, E. et al. Laboratory computed X-ray tomography--A nondestructive technique for 3D microstructure analyis of materials. Pract. Metallogr. 55, 539–555 (2018).
- Cassidy, C., Teva, J., Kraft, J. & Schrank, F. Through Silicon Via (TSV) defect investigations using lateral emission microscopy. Microelectron. Reliab. 50, 1413–1416 (2010).
- da Fonseca, R. J. M. et al. Scanning Acoustic Microscopy—recent applications in materials science. Adv. Mater. 5, 508–519 (1993).
- Zhang, G.-M., Harvey, D. M. & Braden, D. R. Microelectronic package characterisation using scanning acoustic microscopy. NDT \& E Int. 40, 609–617 (2007).
- Briggs, A., Briggs, G. A. D. & Kolosov, O. Acoustic microscopy. vol. 67 (Oxford University Press, 2010).
- Zou, Z., Shi, Z., Guo, Y. & Ye, J. Object detection in 20 years: A survey. arXiv Prepr. arXiv1905.05055 (2019).
- LeCun, Y., Huang, F. J. & Bottou, L. Learning methods for generic object recognition with invariance to pose and lighting. in Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. vol. 2 II--104 (2004).
- Delakis, M. & Garcia, C. text Detection with Convolutional Neural Networks. in VISAPP (2) 290–294 (2008).
- Sermanet, P. et al. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv Prepr. arXiv1312.6229 (2013).
- Lu, H. Quasi-orthonormal encoding for machine learning applications. arXiv Prepr. arXiv2006.00038 (2020).
- Chen, T., Min, M. R. & Sun, Y. Learning k-way d-dimensional discrete codes for compact embedding representations. in International Conference on Machine Learning 854–863 (2018).
- Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You only look once: Unified, real-time object detection. in Proceedings of the IEEE conference on computer vision and pattern recognition 779–788 (2016).
- Shih, F. Y. Image processing and pattern recognition: fundamentals and techniques. (John Wiley \& Sons, 2010).
- Grünwald, E. et al. Automatized failure analysis of tungsten coated TSVs via scanning acoustic microscopy. Microelectron. Reliab. 64, 370–374 (2016).
- Rong, W., Li, Z., Zhang, W. & Sun, L. An improved CANNY edge detection algorithm. in 2014 IEEE international conference on mechatronics and automation 577–582 (2014).
- Petrou, M. M. P. & Petrou, C. Image processing: the fundamentals. (John Wiley \& Sons, 2010).
- Bradley, D. & Roth, G. Adaptive thresholding using the integral image. J. Graph. tools 12, 13–21 (2007).
- Kohler, R. A segmentation system based on thresholding. Comput. Graph. Image Process. 15, 319–338 (1981).
- Puech, M. et al. Fabrication of 3D packaging TSV using DRIE. in 2008 Symposium on Design, Test, Integration and Packaging of MEMS/MOEMS 109–114 (2008).
- Singulani, A. P., Ceric, H. & Langer, E. Stress evolution on tungsten thin-film of an open through silicon via technology. in Proceedings of the 20th IEEE International Symposium on the Physical and Failure Analysis of Integrated Circuits (IPFA) 212–216 (2013).
- Filipovic, L., Singulani, A. P., Roger, F., Carniello, S. & Selberherr, S. Intrinsic stress analysis of tungsten-lined open TSVs. Microelectron. Reliab. 55, 1843–1848 (2015).
- Sharma, S., Sharma, S. & Athaiya, A. Activation functions in neural networks. Towar. data Sci. 6, 310–316 (2017).
No competing interests reported.