MaGNIFIES: Manageable GAN Image Augmentation Framework for Inspection of Electronic Systems

Electronic counterfeiting is a long-lasting problem that continues to cost original manufacturers billions, fund organized crime, and jeopardize national security and mission-critical infrastructures. Manual inspection is a popular and standardized way to detect counterfeit electronic components, but, it is time-consuming and requires subject matter experts for classification. State-of-the-art machine learning, deep learning, and computer vision-based physical inspection methods are promising to alleviate these issues. However, the main bottleneck for doing so is a lack of high-quality, publicly available counterfeit image data for training. Producing such datasets is also time-consuming and often requires expensive equipment. In addition, most test labs are not allowed to freely publish images taken from their customer’s chips. One solution to this data shortage bottleneck can be addressed by augmenting synthetic data. In this paper, (i) data multiplication using Progressive GAN, Style-GAN, and classical methods in counterfeit data domain are explored; (ii) a novel framework, named MaGNIFIES, is proposed; and (iii) an efficient Convolutional Neural Network architecture is proposed, that can detect defective parts by training only on the synthetic dataset generated using (i) or (ii). For proof of concept, we have used low-quality images of resistors and capacitors with and without scratch defects as counterfeit and golden components respectively. We have also illustrated how our approach using MaGNIFIES addresses the shortcomings of the existing augmentation meth-ods. Separate data augmentation detection models are trained with each type of augmented data generated using MaGNIFIES, as well as existing techniques, and tested on a test set of real data.


Introduction
Counterfeit and pirated products create an enormous drain on the global economy.It costs billions of revenues in legitimate economic activity and facilitates a black market that deprives governments of revenues for vital public services, forces higher burdens on taxpayers, dislocates hundreds of thousands of legitimate jobs, and exposes consumers to dangerous and ineffective products.Moreover, counterfeit electronics can be hazardous, especially when incorporated into safety critical systems such as aircraft, trains, and submarines [1].The counterfeit chip market has an estimated worldwide value of $75 Billion, and such chips are integrated into electronic devices reportedly worth more than $169 Billion [2].It is projected that net job losses due to counterfeiting in 2022 will be between 4.2 and 5.4 million [3].In addition, counterfeits have huge reliability and security concerns.It is estimated that 15% of all spare and replacement electronic parts purchased by The Pentagon are counterfeit [4].In 2017, a Presidential Executive order was passed to assess and strengthen the manufacturing and defense industrial base and supply chain resiliency of the United States.Moreover, the ongoing chip shortage due to the COVID-19 pandemic only further aggravates the situation by creating huge gaps in the supply chain.Counterfeiters are exploiting such gaps to fill supply chain needs with more and more counterfeit components [5].Fortunately, counterfeits can often be identified by surface defects on component packages [6,7] such as scratches, differences in indent size and position, sanding/grinding marks, ghost markings, burn markings, mold variation, package damage, corrosion/contamination, and extraneous markings.However, manual physical inspection is costly, time-consuming, and subjective.To pinpoint counterfeits, subject matter experts (SMEs) need to inspect each component for defects and require regular training to keep up with counterfeit detect trends.To address this, a few works from academia introduce automation, but validation is limited to small datasets.Notable works in this domain include [8][9][10][11][12][13][14][15][16], and almost all highlight that the lack of data is a major barrier.Moreover, computer vision and machine learning (especially deep learning) are becoming more popular tools to boost automation.This further increases the need for large datasets, as deep learning requires very large amounts of data on the scale of at least hundreds of thousands of samples for both golden and counterfeit classes.This paper explores different GAN-based and classical data augmentation methods and also proposes a novel data augmentation framework, named MaGNI-FIES, to bridge the gap in automated counterfeit detection and machine learning.An overview of the framework is illustrated in Figure 1.Toward this end, we investigate Generative Adversarial Models (GANs) to generate arbitrary amounts of synthetic "golden" (known authentic) component images as well as defective counterfeit components.After training, such models can rapidly generate any number of synthetic component images without the need for expensive imaging equipment or the immense time to manually take images.Our framework is even capable of generating a variety of defects on counterfeit components, even when some type of original defective data is in scarcity.For example, it can generate components with defects such as scratches, marking imperfections, texture variation, etc.Additionally, such an approach can alleviate privacy concerns associated with test labs sharing counterfeit image datasets.Rather than directly sharing data, the GAN model trained by the data can be shared instead.In other words, derivatives of the original dataset can be generated by researchers or practitioners in the community while the original training dataset can be kept private.Afterward, these synthetic genuine models can be combined with other generative methods to create sets of component data superimposed with expected surface defects to create robust and effective counterfeit detection models in the face of data volume bottlenecks one would traditionally face in this space.This paper focuses mainly on scratch defects commonly found on counterfeit chips, but it can be extended to any other type of physical defect, such as corrosion, ghost markings, and others [6].This paper's contributions can be summarized as follows: To the best of our knowledge in the hardware security and assurance domain, this is the first attempt to generate models that can generate unlimited realistic synthetic data to meet the large dataset requirement for deep learning models.Previously some work used data augmentation to address the bottleneck in Machine learning, but they were mostly classical data augmentation techniques capable of generating only a limited amount of data.In addition, the contributions here are not necessarily confined to the counterfeit detection domain.Research in other domains such as automated physical inspection for hardware Trojan detection, printed circuit board (PCB) bill of materials extraction, etc. also run into the same bottleneck.Thus, the promising results, tools, and datasets from this paper shall enable novel research in a variety of applications.
The remainder of this paper is organized as follows.A brief discussion on the available methods and datasets is provided in Section 2. The background on different tools and techniques used in this paper is described in Section 3. The data augmentation techniques and our proposed framework is explained in detail in Section 4 followed by the defect detection techniques in Section 5. Experimental setup and results are given in Section 6. Brief discussions on the pros and cons of the framework and future work are given in Section 7. Finally, the paper is concluded in Section 8.

Related Works
Some efforts have been made to manually create image datasets for integrated circuits (ICs) and PCBs.One of them is WACV [17], which has more than 8,000 annotated components from 47 images of 32 PCBs.FICS-PCB [18] has over 29,000 annotated components (about 6,000 of which are unique annotations) from 418 images of 31 PCBs and FPIC [19,20] has over 40,000 semantically annotated components from 230 images of 73 PCBs.Although these datasets are relatively large, there are very few defects present in them.Also, even if there is one, the defects are not annotated.The available defect datasets are too small.Some of them include D-PCB [21], Amazon Lookout [22], and HRIPCB [23].D-PCB consists of pairs of PCB images (one with synthetic defects and one without), but only offers 20 pairs.Amazon Lookout [22] consists of only 40 images of PCBs with defects and 40 images of PCBs without defects.HRIPCB [23] consists of about 3,000 annotated components; however, they are all from only 10 PCB images (each PCB has multiple defects).Another dataset that has a relatively decent amount of data is PCBexperiment [24] but it too lacks proper annotation of the defects and is classified based on manual quality assurance checks.

Counterfeit IC Defects
Counterfeit ICs are often recycled, remarked, or cloned versions of authentic ICs.There are a list of defects that are commonly observed on the package indicating their fake nature.For example, the cloned version of an authentic IC may have the same lot number but a different country of origin, or a recycled IC may have some corrosion and/or scratches present on its surface due to wear and tear.A list of commonly observed package defects is tabulated in Table 1 [8].Identification and localization of any of these defects will help in the identification of probable counterfeit chips.

Local Binary Pattern
Texture is a very useful feature that is widely used in solving many computer vision problems.An exact objective definition of "texture" is difficult, but subjectively texture is a property that gives information about the spatial arrangement of the color or intensities in an image.A popular technique used to define texture is Local Binary Pattern (LBP).For each pixel position, it compares the neighboring pixel values with its own value and generates a binary pattern.In this pattern, a value of '1' signifies that the neighboring pixel value is more than the pixel under observation, and '0' means it is less.Then the texel value, representing the pixel value of the texture map, is computed by converting the binary pattern to decimal.The entire texel value computation is illustrated using a 3X3 image in Figure 2.This technique is performed on all the pixels, across the image.Thus, using this map, the texture feature map is obtained.A sample texture feature map obtained from a capacitor is shown in Figure 2.Many state-of-the-art automated counterfeit detection approaches used LBP for classification of defects in smaller dataset and observed high accuracy [8][9][10][11]13].

Data Augmentation
Data in the domain of security is not always readily available.Counterfeit components are one such security domain that suffers from data scarcity.Although the global supply chain is flooded with counterfeit components, incomplete documentation, due to the lack of a centralized forum to report the counterfeit components discovered, is a major reason for such scarcity.Another factor that has induced incomplete documentation is that companies tend to hide counterfeiting cases found in their name to protect their reputation from unfortunate publicity.Automated counterfeit detection via machine learning is efficient but data-hungry.Insufficient data may lead to over-fitting of model, which is one of the major problems with using machine learning.In other words, the training will not be complete without a considerable amount of data leading to inaccurate detection.Data augmentation is one useful approach to overcome over-fitting from data scarcity.It approaches overfitting from the root cause; that is, training data.It considers that raw data contains a load of information and can be expanded.Using data augmentation, new synthetic dataset can be created that follows the same distribution as the input data.It mainly preserves the features of the raw data, and only expands by varying the noise.In this paper, we have explored some of the existing classical data augmentation techniques as well as generative modeling-based data augmentation techniques.We have also proposed a new framework capable of over-coming the shortcomings of the existing techniques.

Classical Data Augmentation
A set of classical augmentation techniques, including blurring and noise, is used to generate new sets of augmented data.For blurring, two sets of data are generated with different sigma (s) values for Gaussian blur.Similarly for noise, two sets of augmented datasets are generated with different variance of random distribution for the Gaussian noise.The main constraint with classical augmentation is that image quality degrades with an increased number of output images.With GAN, any number of similar images can be generated.

Generative Models
Generative modeling is a special branch of unsupervised learning where the model learns the regularities and patterns in the input data so that, once trained, it can generate new data points which plausibly could have been drawn from the original dataset [25].It learns the true data distribution of the training set so that it can generate new data points with some variations.One such technique of generative modeling is the Generative Adversarial Network or GAN [26].This type of training involves two networks -generator and discriminator.Generator models are unsupervised models that summarize the distribution of input variables used to create or generate new examples that plausibly belong to the input distribution.It takes a fixed-length random vector as input which is drawn randomly from a Gaussian distribution.
The discriminator is the predictive model which discriminates or classifies the output of the generator as fake or real.The two models are trained together with the aim of achieving equilibrium.
As the discriminator is updated to improve its ability to discriminate real and fake samples for the next round, the generator is likewise updated based on how well its generated samples fooled the discriminator.This technique uses a game theory approach to achieve Nash equilibrium between the two networks.After the training process, the discriminator model is discarded, and the generator model can be used to generate new, realistic data samples.There are a variety of GAN models available [27].Out of all these models, two categories of the most advanced state-of-the-art models are described here: controlled GANs and uncontrolled GANs.

Uncontrolled GAN
The main purpose of using the uncontrolled GAN model is to multiply the input dataset.It generates synthetic data points following the same distribution as the training data.Two of the most advanced state-of-the-art uncontrolled GAN models are -Progressive GAN (ProGAN) [28] and StyleGAN [29].ProGAN and StyleGAN produce high-quality, realistic images and offer superior control and understanding of the generated images.These qualities make it easier to generate believable fake images.For most GANs, generating realistic, high-resolution images is difficult because higher resolutions drastically amplify the gradient problem where the distributions do not have substantial overlap and the gradients point in random directions.Large resolutions also necessitate the use of smaller minibatches due to memory constraints, which further compromises training stability.However, ProGAN and Style-GAN address this gradient problem by training in an innovative way.Hence, they are among the first GANs that can generate high-resolution images: up to 1024 × 1024 resolution.Moreover, the models can adapt to any resolution and produce high-quality images.

Controlled GAN
Controlled GAN is a special type of generative model which learns pixel-level information like texture, color, etc. for each of the labels provided in semantic maps corresponding to the input images.Semantic Maps are image maps corresponding to a particular input image where each pixel of the input image is color coded in the semantic map and each color belongs to a particular label.It can then generate a realistic image using a user-created semantic map containing similar labels.The user has complete control in creating the semantic map of the output synthetic image.The only disadvantage is that to get an output image, the user needs to provide a semantic map as input.So, to create innumerable output images the user would need the same number of semantic maps.The controlled GAN model that we have used in our work is Pix2pix [30].It learns the mapping between the semantic maps and the real images, provided as training input.
The semantic maps contain separate colors for each label.Once trained, the model can generate realistic synthetic images by using any provided semantic map with similar labels.The pros and cons of the controlled and uncontrolled GAN is illustrated in Table 2.

MaGNIFIES Data Augmentation Framework
The MaNGIFIES framework consists of an uncontrolled GAN in series with a controlled GAN.The reasoning behind this choice and the overall framework are discussed in this section.

Purpose of Controlled GAN
As discussed in Section 4, a controlled GAN mainly helps in creating a synthetic image based on a user-provided semantic map.The main purpose of using the controlled GAN model in the MaGNIFIES framework is to exploit this capability to boost the variety of synthetic golden (i.e., known authentic) and counterfeit images.Specifically, it can be used to create an assortment of counterfeit ICs based on domain knowledge of different types of package defects, commonly found in counterfeit ICs to that of the resistors in the training set, the pix2pix model was able to create reasonably good synthetic images from the semantic map provided by the user.Thus, without even using the original IC, MaGNIFIES was able to generate synthetic reference ICs.However, the main bottleneck of using the pix2pix model alone is that for each synthetic IC it needs one semantic map which the user needs to provide.In order to generate unlimited counterfeit IC images, one needs an unlimited source of semantic maps.Uncontrolled GANs can generate unlimited data following the same distribution as the input data.Hence, the generation of such masks is automated to some extent using an uncontrolled GAN.

Purpose of Uncontrolled GAN
The primary advantage of uncontrolled GAN is that it can multiply the input dataset.The only problem with uncontrolled GAN is that it cannot create an assortment of counterfeit components,

Overall Framework
MaGNIFIES utilizes both controlled and uncontrolled GANs.The main goal of this framework is to generate unlimited counterfeit resistors and capacitors with one or more defects, given a set of golden ones and domain knowledge about the defect that needs to be inserted to generate the counterfeit component.The framework is a twostep method, involving pre-processing and two trainings asshown in Fig. 5, 6 and 7.The overall framework as shown in Figure 8.Each step is discussed in detail in the following sub-sections.
Step 1: This step involves pre-processing of the existing dataset and training the GAN models.In the pre-processing step, synthetic defects are added using two methods-image processing and/or controlled GAN.The pre-processing has two sub methods, semantic map creation and defect insertion based on domain knowledge.
• Semantic Map Creation: Semantic maps are generated for both the real and defective dataset, if available, using image processing.Since, this is the first work, no existing semantically labeled dataset was found.Each of the real data points are labeled manually.From these labeled datasets, the defective counterparts are generated using image processing.In future, this smaller labeled data can be used to automate there is a list of package defects that are commonly observed in counterfeit components.These defects can be either inserted in golden IC images using an image processing algorithm or using a trained controlled GAN.Image processing is mainly needed to add defects with new labels, i.e., the texture, color and/or pattern are not present in the input data and need some new label assignment.Controlled GAN is needed when existing labels are modified to create defects.For marking defects like letter mismatch, we can use the same semantic map label of the existing data and no new label is needed.However, if we want the counterfeit component to have markings with different color or scratch defect, which is not present in the real input dataset, we will need a new label for that, and the defect needs to be added externally on the component using image processing.
The flow diagram along with some counterfeit components with scratches is shown in Figure 5.
For the purpose of proof-of-concept and lack of original counterfeits, we have used only scratch defects, where random scratches are added to the data and a new defective dataset is generated.At the end of this step there is real golden dataset with two classes: defective and golden.Each of the images in this newly created dataset also has a semantic map.uncontrolled GAN models are trained with only the semantic maps of each of the defect and golden IC category.This uncontrolled GAN is used later as a semantic map multiplier as it is capable of generating any number of semantic maps similar to input semantic map distribution.The workflow of the uncontrolled GAN is shown in Figure 7.
Thus, we see defect insertion is taking place at two places in step 1.One at defect addition, where new types of defects are introduced and second by manipulating the input semantic to a controlled GAN.Uncontrolled GAN just multiplies the images by inserting randomnessin the semantic, for example creating randomness in the defect location, shape and size in the semantic.This semantic is then given asinput to the controlled GAN to generate a new data point.
Step 2: This step is essentially the multiplication part of the framework, combining the different modules obtained in step 1.From step 1, the trained controlled and uncontrolled GANs along with a classes w/ and w/o defect real IC dataset are obtained.This dataset is small, but each data point has a corresponding semantic map.This small dataset may have more categories of data.For N number of defect categories there can be N number of categories and one real data category.In this step, each of these categories is multiplied using an uncontrolled GAN trained in Step 1.Each of the trained models are used to generate unlimited semantic maps visually similar to the trained category.Since these semantics are segmentation maps, a little poor quality of the generated image would not create any problem, as long as all pixels belonging to the same category have the color as the training data.Once these synthetic semantic maps are generated using the uncontrolled GAN, they can be used in the trained controlled GAN model, also obtained from step 1, to generate more real and counterfeit component images.The entire MaGNIFIES workflow is given in Figure 8.An interesting fact about MaGNIFIES is that the generated data has both the image as well as the semantic maps from which the data was generated.Hence, we plan to use this framework in the future, to generate a huge dataset which can be used to perform many segmentation tasks using deep network architectures and to detect defects.The pros and cons of each of the discussed data augmentation techniques is given in Table 2.

Defect Detection
To illustrate the power of data augmentation in defect detection and to find the most efficient data augmentation technique, different machine learning algorithms are trained with each of the synthetic datasets generated using different augmentation techniques, i.e., classical data augmentation, controlled, uncontrolled and MaGNIFIES, as discussed in Section 4, and tested on a set of real data.The different detection algorithms and their respective setup and parameters used are discussed in this section.These algorithms can be broadly classified under two categories.They are classical machine learning techniques and deep learning.The number of synthetic data points used for training is also varied to illustrate the power of data augmentation.A Convolutional Neural Network architecture is also proposed for defect detection in counterfeit components using deep learning.

Classical Machine Learning Techniques
The different classical machine learning algorithms used for defect detection and quality analysis can be grouped under three categories: linear, non-linear, and a basic neural network.The linear models used are Support Vector Machine (SVM) with linear kernel [31] and Logistic Regression [32].The basic neural network approach used Fig. 9: Quantity Analysis: CNN architecture used for CNN is Multi-layer Perceptron (MLP) [33] with 300 iterations.Local Binary Pattern (LBP), as discussed in Section 3, extracted from each of the images are used as input features representing the images, for each of these models.

Deep Learning
The main bottleneck of deep learning algorithms in detecting defects is necessity of large datasets.Even if the data is of very good quality, if the number of data points is not enough the model fails to learn all the features present in the data and leads to over-fitting.In such cases data augmentation plays an important role by spreading the data and increasing the amount of training data.
The major utilization of the data augmentation techniques used in this paper are in the domain of deep learning.In this paper a simple and novel Convolutional Neural Network architecture is used for defect detection.The detail architecture of the model is given in the Figure 9.Each of the augmented sets of data, explained in Table 3, is used to train a CNN model, with batch size of 32 and 20 epochs.Each of the training set data is divided into 80% training data and 20% validation data.The training and validation curves for resistors and capacitors converges to an accuracy of 99.9% for both after 100 iterations.Once trained the models are tested using the same real test dataset which is used by the other machine learning methods and given in Table 4.
6 Experimental Setup and Results

Generative Adversarial Network Setup
In the following subsection, a brief introduction to the GAN architectures used is provided.The trade off in accuracy between using only unconditional GAN, to generate only one type of defective class, and our framework, capable of generating multiple class, is shown.

ProGAN and StyleGAN
In this paper, ProGAN and StyleGAN performances are first compared.ProGAN is an innovative way to train a GAN which involves training with low-resolution images, and then progressively increasing the resolution by adding layers to the networks.More details about this GAN model and the network parameters can be found in [28].
StyleGAN is a more advanced version of Pro-GAN, as its baseline configuration setup is the same, but its generator network has some modifications [29,34].More details of the architecture can be found in [29,34].As discussed in 3, Style-GAN and ProGAN are not capable of generating a variety of defects.Hence, we have combined Style-GAN with our framework and compared the trade off in accuracy.
Since the input capacitor and resistor images of the FICS-PCB Image Collection (FPIC) dataset [35] are low-resolution and vary in shape and size.Before training GANs, the images are resized to 256 × 256 for training.After training, both GANs generate synthetic capacitor and resistor images, when given a 512 dimensional vectors drawn randomly from a Gaussian distribution.The synthetic images have the same size as the training images: 256 × 256.

Pix2pix
The Pix2Pix GAN architecture involves the careful specification of a generator model, discriminator model, and model optimization procedure.Both the generator and discriminator models use standard Convolution-Batch Normalization-ReLU blocks of layers as is common for deep convolutional neural networks.The generator model is a modified U-Net model.Unlike the traditional GAN model that uses a deep convolutional neural network to classify images, the Pix2Pix model uses a PatchGAN as the discriminator.Details about the model and the architecture can be found in [30].Unlike ProGAN and StyleGAN, a Pix2pix GAN is not capable of generating synthetic data without semantic.Hence it is used only in our framework.

Dataset
For training these models, we have used the resistor and capacitor images from FICS-PCB Image Collection (FPIC) dataset [18] with 424 capacitor images and 11,645 resistor images.For each set of augmented datasets, two classes of images are generated.They are 'with scratch' and 'without scratch'.Out of these images, 200 images are randomly selected from each of resistor and capacitor.The defective class of images with scratch is created from these images by superimposing scratches.Semantic maps are generated for each of these images, capacitor and resistors both, collectively referred as Real Train set or (A) in this paper.Another set of 200 data points are randomly sampled from the remaining dataset.Scratches are again superimposed on them to create the 'scratch' class of images.This set of total 200 images for each class of with and without scratch, mutually exclusive of the Real Train set, for each of capacitor and resistor is collected and named as Real Test or (H) on which each of the models are tested.Parts of Real Train set is used to train the uncontrolled GAN, as well as, to generate augmented images by adding blur and noise.The semantic labels for Real Train are used to train the uncontrolled GAN of the MaGNI-FIES framework and images of Real Train along with the semantic maps are used to train the controlled GAN of the framework.After training the uncontrolled GANs (ProGAN and StyleGAN) with the Real Train dataset, 10000 synthetic datapoints are generated using each of the models for each class.These new datasets are named as ProGAN10000 or (B) and StyleGAN10000 or (C).The labeled Real Train dataset is also used to train different parts of the Magnifies Framework.Once trained, a set of 10000 datapoints are generated using the framework for each class and is referred as MaGNIFIES10000 or (D).Another set of 200 augmented data points are generated by adding blur to each class of 200 Real Train dataset for each of Sigma values 1 and 2. They are referred as classical blur or (E) and (F).In detection methods, for increasing datapoints, (E) and (F) are combined and have commonly referred as Classical Blur.Similarly for noise two datasets, each of 200 datapoints for each class, are generated by adding random noise in the images using floating point values of 0.1 and 0.2, where the range of noise that can be applied is from zero to 1, 1 being the highest.These datasets are referred as (G) and (H) and in Detection methods they are mixed and referred as Classical Noise.All these datasets are also shown in Table 3.The final dataset for each type contains two subclasses -with and without scratch.

Data Augmentation and Quality Analysis of Generated Data
In this section, different no-reference quality metrics of the images generated using MaGNIFIES are compared with the quality of the real images as well as the synthetic image datasets given in Table 3.Each of the experiments is performed on capacitor and resistor datasets separately.No-reference image quality assessment (NR-IQA) is a class of IQA that is used to predict the quality of an image as perceived by human observers without using any pristine, reference images.GANs generate images from input data distributions and, hence, there are no particular data points to compare the images with.In such a situation, NR-IQA helps evaluate the quality of the output images without requiring any reference images.In this paper, the NR-IQA distributions of 100 randomly chosen images of each of the generated datasets are evaluated and compared.Among these no-reference image quality measures, there are two types of techniques.One of the techniques uses trained models which are trained on good and bad quality natural images and the score gives an idea about the image quality.10.As NIQE mainly identifies the statistical regularities (NSS) and performs less successfully than BRISQUE in identifying distortions, the range of NIQE values of all the datasets is almost similar.But when the mean value is observed, it is visible that the quality of the GAN-generated images is better.

Sharpness
Image sharpness measures distortions by detecting blurriness, rather than training [39].Although there are numerous ways to quantify sharpness, this paper uses a popular gradient-based approach [40].In general, higher values indicate higher clarity.Very high sharpness values can indicate graininess or noise, which are other types of distortion.Ideally, the sharpness distribution of the augmented data is the same as that of the real data.The result obtained after quantifying the sharpness on 100 randomly chosen data points from each of the datasets created in this paper is shown in the box plots in Figure 10.It clearly illustrates how the addition of blur and noise degrades the quality of the images, whereas the quality of the images generated by the generative models is similar and sometimes better than the real images.Similar sharpness values would indicate that the generated synthetic images are similar to the real images, and the GAN is not causing unnatural distortions.In addition, the better values indicate that the GAN models are able to remove noise and improve the quality of the generated images.

Observations
There is a clear trend of an increase in accuracy with the increase in data for both classical machine learning techniques and deep learning techniques.Overall, the accuracy of the deep learning methods is much higher than the classical methods.Hence, it is evident if the data bottleneck for deep learning methods is solved, then much better accuracy can be obtained.Another interesting fact observed in the result is that, in some cases, the model trained on synthetic data outperforms the model trained on real data.One reason for this is the real data is of very low resolution and extremely noisy.Noise affects the quality of the images to a great extent as we have seen in quality analysis.The GAN-generated images are relatively less noisy.The quality of the images generated by all three generative models is similar to or better than the real images in terms of noise.This may have resulted in increasing the accuracy to some extent.Also, in most cases, the number of data points used to train the models is more than the model trained with only real data, which has also resulted in better accuracy.

Discussion
This paper illustrates a preliminary synthetic counterfeit component data generation method using different generative models as well as the proposed MaGNIFIES framework.The paper also proves how data generated by each of the proposed techniques, i.e., StyleGAN, ProGAN, and MaG-NIFIES generally improves the accuracy of scratch detection in components.Not only in terms of volume, but our proposed framework MaGNIFIES can also generate different categories of counterfeit data by incorporating domain knowledge of counterfeit components.Quantity and quality analyses of the synthetic data also show that it is as good as the real data, sometimes performing better because of less noise and/or more quantity and helps in increasing the accuracy of counterfeit detection.There is room for improvement and expansion in future work.In this paper, because of the lack of high-resolution real data, the synthetic images generated by both our framework as well as the StyleGAN and ProGAN are of equally low resolution.To generate high-quality images, the framework would need some number of highresolution images.The dataset in this paper lacks pin and lead data to properly capture ICs which can be added in the future.Many other counterfeit data types and defects will also be created.
In this work we have used data which has very less variation, i.e, addressed only one category of capacitor or resistor, and addressed only one category of defect.To address all categories of data with all the defects, we would require a larger dataset with sample subset of each category and defect.Also, since each of the output images has a semantic map associated with it, the paired huge dataset can be used in advanced computer vision and image analysis algorithms, such as automated segmentation, that require such semantic maps for training.Finally, our promising results coupled with the sheer quantity of synthetic images that the generative models can produce shall enable deep-learning based classification in future work, which cannot currently be employed due to the scarcity of real data.

Conclusion
Lack of data is a long-standing bottleneck in the automated detection of counterfeit IC using machine learning and deep learning techniques.Manually creating datasets by taking and labeling images is time-consuming, requires expensive image collection setups, and requires one to buy vast quantities of electronic chip/component samples.Also, such counterfeit electronics are not always readily available.In this paper, to alleviate such problems of data acquisition, different data augmentation techniques are explored.The cons of each of the methods are then addressed using our proposed framework, that uses generative models along with domain knowledge of the counterfeits to generate synthetic chip/component images having properties similar to the actual authentic and counterfeit chips/components.As a proofof-concept, a dataset of 10,000 images each of capacitors and resistors with and without scratch defects was generated using our framework.The quality analysis of the generated synthetic images and the real images showed that the generated images are as good as the real ones.To prove the utility of uncontrolled GANs and the MaG-NIFIES framework, quantitative analysis is also performed using different machine learning algorithms.The results clearly show how an increase in data using GANs increases detection accuracy.
In the future, we plan to use higher-resolution real images to illustrate the generation of other types of counterfeit defects using this framework.

Funding
The work was supported by National Science Foundation (NSF) Awards -1821780 and 2131480.

Availability of data and materials
Dataset and Code will be made available in Github after acceptance of the paper.

Fig. 3 :
Fig.3: An assortment of counterfeit data generated from semantic masks using Pix2pix.dissimilarsynthetic components and ICs with different marking and pin structures compared to the training set is shown in Figure4.Here the model is trained on resistors and their semantic maps are similar to the images shown in Figures4(a) and 4(b).But, when the trained model is provided with semantic map of a reference IC, as shown in Figure4(d) and 4(c) respectively, it successfully generates the synthetic IC that looks similar to the original reference IC image, from which the semantic map is inspired, as shown in Figure4(e).Note that these original reference ICs are not present in the training set.Nevertheless, since the textures of each of component in semantic map locations of the reference ICs are similar to that of the resistors in the training set, the pix2pix model was able to create reasonably good synthetic images from the semantic map provided by the user.Thus, without even using the original IC, MaGNIFIES was able to generate synthetic reference ICs.However, the main bottleneck of using the pix2pix model alone is that for each synthetic IC it needs one semantic map which the user needs to provide.In order to generate unlimited counterfeit IC images, one needs an unlimited source of semantic maps.Uncontrolled GANs can generate unlimited data following the same distribution as the input data.Hence, the generation of such masks is automated to some extent using an uncontrolled GAN.

Fig. 4 :
Fig. 4: Structurally different IC generated by training pix2pix with only resistors and capacitors.Reference IC not present in training set.unlike the controlled GAN.But, if we have a small amount of assortment of counterfeit IC images, then uncontrolled GAN can be used to multiply the number of images.Hence, we have used an uncontrolled GAN in the MaGNIFIES framework as a multiplier of semantic maps.The uncontrolled GAN which we have integrated in our framework is StyleGAN.

Fig. 5 :
Fig.5: Pre-processing of data: Defect insertion using image processing and/or controlled GAN the process of generating larger dataset with many defects and defect detection using uncontrolled GAN.This new semantic map dataset is one of the contributions of this work as well and plays a vital role in the automation of counterfeit resistors and capacitors.Although small in size, this can be considered as the first semantically labeled resistor and capacitor dataset.•Defect Addition: As discussed in Section 3, there is a list of package defects that are commonly observed in counterfeit components.These defects can be either inserted in golden IC images using an image processing algorithm or using a trained controlled GAN.Image processing is mainly needed to add defects with new labels, i.e., the texture, color and/or pattern are not present in the input data and need some new label assignment.Controlled GAN is needed when existing labels are modified to create defects.For marking defects like letter mismatch, we can use the same semantic map label of the existing data and no new label is needed.However, if we want the counterfeit component to have markings with different color or scratch defect, which is not present in the real input dataset, we will need a new label for that, and the defect needs to be added externally on the component using image processing.The flow diagram along with some counterfeit components with scratches is shown in Figure5.For the purpose of proof-of-concept and lack of original counterfeits, we have used only scratch defects, where random scratches are added to the data and a new defective dataset is generated.At the end of this step there is real golden dataset with two classes: defective and golden.Each of the images in this newly created dataset also has a semantic map.

Input: 1 )YỲFig. 6 :
Fig. 6: Pix2pix Training (T1) (Controlled): Creation Using Controlled GAN • Controlled GAN training: Each class of the data with (w/) and without (w/o) defects along with their semantic maps are then used to train a separate controlled GAN architecture.This controlled GAN is then capable of generating more defective ICs as discussed in Sections 3 and 4.3.1 by providing the random semantic maps created by the user.In this paper, although we have used only the real, and scratch defect category for the next steps, it can be extended to any number of defect categories using trained controlled GANs in the future.• Uncontrolled GAN training: Separate uncontrolled GAN models are trained with only the semantic maps of each of the defect and golden IC category.This uncontrolled GAN is used later as a semantic map multiplier as it is capable of generating any number of semantic maps similar to input semantic map distribution.The workflow of the uncontrolled GAN is shown in Figure 7.

Fig. 10 :
Fig. 10: BRISQUE and Sharpness image quality comparison.The x-axis represents the datasets given in Table3Table4: Quantitative Analysis Results for Capacitors(C) and Resistors(R): Each cell represents a model trained on the dataset mentioned in column-head and algorithm mentioned in row-head.Each of these models are tested using the Real Test Capacitor data.

Table 1 :
Commonly observed package defects in counterfeit components and chips.

Table 2 :
Pros and Cons of the different data augmentation techniques in defective components generation.

Table 3 :
Datasets used in Quality and Quantitative Analysis.
The other technique is just to compare the different properties of the image like sharpness, brightness, blur, etc.For the technique with the trained model, we have used two methods Blind/Reference-less Image Spatial Quality Evaluator (BRISQUE) and Naturalness Image Quality Evaluator (NIQE), and for the technique which evaluates the property of an image, measurement of sharpness is used.Each of these measures has benefits and limitations, which are discussed in the following subsections.

Table 4 :
Quantitative Analysis Results for Capacitors(C) and Resistors(R): Each cell represents a model trained on the dataset mentioned in column-head and algorithm mentioned in row-head.Each of these models are tested using the Real Test Capacitor data.
Dr. Domenic Forte and Dr. Damon L. Woodard provided supervision throughout the work.Data labeling done by Pallabi Ghosh and Gijung Lee.Coding and data generation done by Pallabi Ghosh, Gijung Lee and Mengdi Zhu.Pallabi Ghosh wrote the manuscript.Olivia P. Dizon-Paradis provided supervision throughout the writing process and contributed text in the literature reviews to the paper.All authors reviewed the paper.