A Deep Learning Solution for Real-time Quality Assessment and Control in Additive Manufacturing Using Point Cloud Data

This work presents an in-situ quality assessment and improvement technique using point cloud and AI for data processing and smart decision making in Additive Manufacturing (AM) fabrication to improve the quality and accuracy of fabricated artifacts. The top surface point cloud containing top surface geometry and quality information is pre-processed and passed to an improved deep Hybrid Convolutional Auto-Encoder decoder (HCAE) model used to statistically describe the artifact's quality. The HCAE’s output is comprised of 9*9 segments, each including four channels with the segment's probability to contain one of four labels, Under-printed, Normally-printed, Over-printed, or Empty region. This data structure plays a significant role in command generation for fabrication process optimization. The HCAE’s accuracy and repeatability were measured by a multi-label multi-output metric developed in this study. The HCAE’s results are used to perform a real-time process adjustment by manipulating the future layer's fabrication through the G-code modification. By adjusting the machine's print speed and feed-rate, the controller exploits the subsequent layer’s deposition, grid-by-grid. The algorithm is then tested with two defective process plans: severe under-extrusion and over-extrusion conditions. Both test artifacts' quality advanced significantly and converged to an acceptable state by four iterations.


Introduction
Additive Manufacturing (AM) systems and techniques have been evolving rapidly in recent decades.But still, despite their tremendous capabilities, they suffer from insufficient repeatability and a lack of advanced online and adaptive controllers to prevent loss of time and material.In the era of Industry 4.0, as more sensors and technologies emerge into conventional industrial machinery, many opportunities for growth and advancement are possible.Engagement of various sensors to collect data, advanced computing infrastructures to process the data, and novel Machine Learning and Artificial Intelligence (AI) based algorithms to develop awareness from the processed data are among the few great tools available nowadays which can significantly help researchers to develop an understanding of the machine processes and create smart manufacturing units.
Due to fabricating complex geometries in a layer-by-layer approach, AM technologies have been emerging in many industrial and consumer-level productions.However, due to their layer-by-layer fabrication nature, an insignificant deviation from the desired quality in the earlier layers could lead to an enormous error buildup and deviation from the desired objective after the layers stack up.Recent studies (Alammar et al., 2022) show that the geometry and the mechanical properties of the AM structures deviate from the expected ideal properties as the error builds up.Therefore, in-situ error detection and mitigation strategies are necessary to ensure the fabrication process's repeatability and part quality, even with random faults, inappropriate process parameters, or faulty machinery components.

Background
As proposed by Mostafaei et al. (2022) anomalies and defects in AM can be categorized into three types based on their accordance timeline, as in material-related defects, which happen before the fabrication, process-related defects during part fabrication, and post processes defects occurring after artifact creation.Among those, defects accruing before or after the printing due to their offline nature can be addressed offline; however, detecting process-related defects during the printing process is crucial for the part's integrity and quality assurance.
Characteristics of the layer-by-layer material depositions significantly affect the overall part's quality and properties.To name a few, the temperature of the deposited material (Mahmoud & Manoochehri, 2021;Mondal et al., 2020), the deposited material's shape (Papon et al., 2021), material's flow and deposited amount (Bakhshi et al., 2021;Luo et al., 2020), and the material's trajectory (Suzuki et al., 2020) are some of the important aspects that impact the quality of the final fabricated artifact.To illustrate better, the material temperature plays a significant role in bonding the old and the new material.Too-low temperatures wouldn't allow bonding, while a too-high temperature causes material degradation and flow of the material to unwanted sections (Papon et al., 2021).On the other hand, the shape of the previous depositions, as well as the amount of the material and the trajectory used for deposition, can also impact the part's integrity (Luo et al., 2020;Mondal et al., 2020;Papon et al., 2021;Suzuki et al., 2020).If the material is deposited over a rough surface, gaps and voids might occur, significantly reducing the inner layer bonding and the part's strength.Many studies aimed to optimize the processes to minimize the gaps and increase the bonding in the AM processes (Ravi et al., 2016).To ensure the part's fabrication is undergoing a predictable and repeatable process, achieving an even layer fabrication would be necessary to reduce the effect of the previous depositions on the future layers.
Anomalies detection in the process or machinery relies on integrating at least one sensor with the machinery or performing offline tests to determine the fabrication's deviation from the goal.The sensor-based monitoring technologies have shown immense potential for part anomaly detection in real-time during AM manufacturing processes.
Different sensors and methods have been used for online monitoring purposes such as various vision-based approaches with single and multiple cameras, thermal cameras, laser surface profilometer, acoustic sensors, and et. with various data processing methods from manual to AI automated approaches.(Delli & Chang, 2018;Moretti et al., 2021;Petsiuk & Pearce, 2020;Preissler et al., 2018;Xu et al., 2022).Considering AM fabrication's real-time monitoring requirements and complexity, the non-contact and non-destructive sensors among the available methods are highly recommended.In recent studies, vision-based sensing techniques such as cameras, thermal profilers, and surface profilometers have attracted significant attention due to their non-contact and non-destructive monitoring.Vision-based sensing can be categorized into 2D image-based and 3D profile-based monitoring systems.
Various works have shown vision-based monitoring techniques' promising detection and process analysis capabilities using 2D image-based sensors.Among the recent literature, Moretti et al. (2021) implemented a camera to identify the layer contour fabricated by an FFF machine.Computer vision algorithms extracted the layer contours from the high-resolution images.The contour quality could be monitored precisely by comparing the measurement and a reference contour obtained from a simulation using digital twins.The monitoring system demonstrated the capability to identify the defected test geometry instances.Also, Jin et al. (2020) utilized a single camera for process observation.The author installed a single camera next to the extruder head of an FFF machine to monitor the delamination of fabricated parts.The delamination in the FFF process is mainly caused by an improper gap between the current nozzle height and the existing print.Based on the severity of delamination of the final print, the offset nozzle height in the real-time images was classified into four categories, 'High+', 'High', 'Good', and 'Low'.With the help of the DL algorithm, 10800 images obtained by the single camera are classified by the CNN model with 91.0% accuracy.
The previous studies (Delli & Chang, 2018;Jin et al., 2020;Moretti et al., 2021) showed that the singlecamera monitoring systems could identify defects during the AM fabrication process.However, the robustness and reliability of the systems highly rely on the environmental light intensity.Different light environments can significantly change the thresholding parameters in computer vision algorithms (Fastowicz et al., 2019;Moretti et al., 2021;Petsiuk & Pearce, 2020).Meanwhile, the single-camera system could only provide the 2D information of fabricated parts, while the depth information is missing during this process.Therefore, 3D profile-based monitoring systems are widely studied for process monitoring.Preissler et al. (2018) utilized a structure light system with dual cameras for FFF process monitoring.The structure light system was installed on the top of the print platform to monitor the dimensional information.By comparing the point cloud captured by the monitoring system and the STL model, the fabrication deviation could be evaluated if it is within the tolerance.With the help of dual cameras and a projector, real-time 3D information could be monitored during the FFF process.This system suffers deeply from low resolution and reliability of the environmental lighting conditions.Therefore, laser-based 3D scanner-based systems are suggested to address the shortcomings mentioned earlier.The laser-based scanner packages a laser line emitter and a camera with a fixed relative position.The camera captures the reflected light of the laser emitter from the target part.Based on the optical triangulation principle, the changes in single line height are calculated to obtain the 3D profile of the target part.Because the camera only receives the known light source of the laser emitter, the 3D laser scanners have better repeatability and performance on finedetailed and free-form shapes.The laser scanner's output forms a point cloud at which the geometry of the artifact can be described.Although the point cloud requires massive memory and processing capabilities, the information generated has excellent potential for many research applications and requires efficient processing algorithms.
Point cloud processing and awareness building can occur efficiently with the proper selection of algorithms and processing units.Processing the raw data acquired from sensory modules is crucial in generating awareness and extracting knowledge from the undergoing processes.Researchers proposed various methods, from conventional thresholding and image processing techniques (Baumann & Roller, 2016;Borish et al., 2020;Harbig et al., 2022;Zhao et al., 2021) to the latest DL-based algorithms (Lyu et al., 2021;Pandiyan et al., 2022).Traditional methods require manual or semi-manual filters and feature generation and selection, which in small-scale problems would be feasible.However, the manual methods would get complicated regarding bigger-scale projects involving various types of data, big scale data inputs, or numerous sensory modules.Therefore, the latest advancement in ML-based data processing played a noticeable role in filter and threshold selections due to their auto feature creation and feature selection capabilities.Despite the traditional methods in which the data and features were known to the researcher for an unknown result, the data and results in ML-based methods are known.Therefore, the goal is to find the appropriate features and formula correlating the input data to the output.Harbig et al. (2022) proposed a manual filter selection and calibration approach for melt pool anomaly detection.The author utilized two photodiodes to gather 13800 frames per second with 160*140 pixels per frame.Then by manually selecting indicators of the raw data, the author reduced the dimensionality of the input and created a feature map.Then, by applying manually selected thresholds, the abnormal conditions were detected.Although this approach showed outstanding achievements in abnormality detection, the manual selection of features and thresholds needs to show generalizability to more complex and non-linear problems.
On the other hand, Tian et al. (2021) suggested DL-based feature extraction for a metal AM machine.Their setup consisted of a pyrometer working alongside a thermal camera and a 3D CT scan machine.The data gathered from the CT machine was used to label the Pyrometer and Thermal camera automatically.Once labeled, they were used to train a deep network that created feature maps and selected the most important ones through the learning process.Their proposed method successfully categorized the input data into the desired classes.Larsen and Hooper (2022) proposed a deep semi-supervised learning model which considers a dynamic physical model to identify the anomalies during the laser powder bed fusion system.The method explored the correlations between images required from a high-speed camera and the laser input and the time-series correlations between images to reduce the rate of false detections.The proposed model could distinguish between process signatures at porosity levels of 0.1% with an AUC of 0.944.Similarly, Kwon et al. (Kwon et al., 2020) also implemented deep neural networks for melt pool imagebased classification.The deep neural network model include several hidden layers and fully connected neurons.The model could determine whether powder bed AM process state is normal or abnormal and to separate the defective products.Therefore, the DL algorithms(J.Li et al., 2021;X. Li et al., 2020) have shown the great capability to extract useful information for AM part quality analysis.
Identifying the anomalies during the AM fabrication process is not enough to improve the part quality (Akhavan & Manoochehri, 2022).The implementation of quality control techniques is indispensable.The control strategies in AM could be classified into two categories, offline and online strategies (Devesse et al., 2016;Durão et al., 2019;).As for offline strategies, most studies focused on process parameter optimization.The relationship between process parameters and part quality is analyzed with several designed experiments.The optimal process parameter combination can be identified and applied to real fabrication to realize quality improvement based on the results.Durão et al. (2019) conducted experiments based on the Design of Experiments (DOE) to optimize material consumption, manufacturing time, and dimensional accuracy (including linear error and surface flatness) for FFF technology.They identified that the print speed and the number of contours are the most critical factors in the quality of the final part.However, due to the complexity of AM process, the optimal process parameters could not be constant during the entire fabrication process.Therefore, it is necessary to implement real-time control strategies to adjust the parameters according to the fabrication environment dynamically.Development and implementation of autonomous controllers based on artificial intelligence techniques have been rapidly studied in the recent literature (Askari et al., 2022;Azimirad et al., 2022;Farbiz et al., 2022;Kumar et al., 2022;Tercan & Meisen, 2022).Advanced control strategies, like adaptive control, intelligent control algorithm, or predictive control, were utilized to achieve closed-loop control in AM system.Devesse et al. (2016) designed a controller including a linear state feedback controller and PI controller for a laser-based AM system.A hyperspectral camera is used to measure the temperature of the melt pool.The state-space equation of the controller was deduced from the physical heat conductivity model of the melt pool.This hardware-in-the-loop (HIL) system could control the melt pool temperature during the laser-based AM fabrication process to guarantee the part quality.
Diminishing anomalies rooted during the part's fabrication is critical for quality improvement and assurance, as cited in the reviewed literature.Among the possible irregularities, surface evenness has shown excellent controllability in previous studies (Gockel et al., 2019;Z. Li et al., 2019;Obilanade et al., 2021) Therefore, in this paper, the focus will be on optimizing the top surface roughness.Although many parameters affect the surface roughness, such as printer's parameters like print speed, and G-code generation algorithm parameters like path planning approach, and many more, in this study, all the variables related to the preprocessing step like G-code generation strategies are fixed, and only printer's parameters are variable.
To measure the evenness and quality of a surface, a surface roughness criterion such as the mean of the absolute heightmap (Ra) or root mean square of the height map (Rms) are suggested repeatedly in the literature.In this study, the Ra value of the processed point cloud data measured in-situ is selected as the fabrication quality criteria.The use of ML algorithms, especially DL methods, paired with laser surface profilometers provides a deeper assessment of the fabrication quality.In this study, DL algorithms were found to be an excellent investigation method due to their capabilities for detection and classification as well as their ability to create and select the necessary features.Ultimately, by taking advantage of the DL methods' output, a closed-loop controller is then utilized to adjust process parameters in real-time to mitigate the detected abnormalities.

Methodology
The process diagram implemented in this study is illustrated in Fig. 1.The process starts with a preprocessing module to process the CAD input and generate the printer's machine codes.Then, the machine codes are executed to fabricate the part layer-by-layer.The laser scanner will start recording the artifact's top surface of any deposited layer given that it's not the part's final layer to be deposited.The data gathered would then be used to create a point cloud and turned into a depth map.The depth map generated would be transferred to the cloud for further analysis.Once the data becomes available on the cloud, the multi-region HCAE algorithm will extract important information and statistical representation of the built surface.Ultimately, the statistical representation would generate control signals to adjust the quality by manipulating the machine codes of succeeding layers.This section will first introduce the implemented machine setup, including the laser scanner, printer, and software needed for operating them.Then, the data acquisition and pre-processing of the point cloud will be illustrated.Thereafter, the multi-region HCAE V.2 deep model with two main upgrades compared to the previous work (Lyu et al., 2021(Lyu et al., , 2022;;Lyu & Manoochehri, 2021) will be demonstrated.In the end, a controller algorithm for the G-code manipulation and the system's plan adjustment would be introduced.

Fig. 1: Process diagram of multi-region HCAE classifier working in series with the laser scanner and G-code
manipulator controller for in-situ quality advancement.

Experimental Setup and Point Cloud Processing
The experiment in this study is conducted on a desktop FFF machine, Creality Ender 5, as shown in Fig. 2. The machine is equipped with a house-made frame used to carry the laser scanner to satisfy the experimental requirements.In the meantime, as an open-source FFF machine, it can communicate and be controlled by a workstation in real-time.The laser scanner used in this study is KEYENCE high-speed Laser Profiler LJ-V7200.As shown in Fig. 2(b), the laser scanner is mounted on the metal frame and runs in the plane above and parallel to the X-Y plane of the FFF machine.The laser scanner could provide 1 µm resolution in the Z direction, which channel to record the surface detail.The communication protocol between the workstation and laser scanner uses TCP/IP.
The CAD file is first sliced by the Simplify3D software and exported as a G-code file.The workstation sends the G-code commands line by line to the FFF machine during fabrication.As soon as the deposition of any new layer is completed, the workstation commands the printer head to move to a corner.It then triggers the house-made frame motion mechanism to drive the laser scanner to go over the fabricated surface.An angle encoder continuously triggers the laser scanner until it scans the surface.A batch of 2D laser line profilers is published as a raw point cloud dataset and saved in the workstation.Before extracting features from the point cloud dataset, the dataset should be processed and transferred into a 2D depth image.To process the point cloud, it is necessary to denoise the captured raw point cloud first.The laser scanner obtains the depth information of the target based on the 3D triangulation method.The shadow effect can mislead the laser scanner by making wrong measurements to where the height changes abruptly, as shown in Fig. 3.When the built-in camera of the laser scanner cannot receive the laser projection line, some outliers can be generated erroneously by the algorithm, as shown in Fig. 3(c).The point cloud library (PCL) (Rusu & Cousins, 2011) is modified and utilized to remove the noise.The statistical outlier removal algorithm can effectively eliminate the noise caused by shallow effects.The denoised raw point cloud always remains in reference to the laser scanner coordinates.But, during the fabrication process, the platform keeps moving down for new layers, which keeps updating the referenced FFF printer coordinates.So for consistency, it was deemed necessary to transform the denoised point cloud data into the FFF printer coordinates, to have them share the same coordinates.This will simplify the following calculations.Since the 2D images have better computational efficiency than the 3D point cloud in the feature extrusion process.The 3D point cloud, after processing, needs to be transferred into a 2D depth image using the proposed rasterization method (Lyu & Manoochehri, 2021), as shown in Fig. 4. The upper surface in the 3D point cloud is segmented using a random sample consensus (RANSAC) algorithm.The inliers of the upper surface can be identified as blue points in Fig. 4, and a virtual plane of this surface can also be obtained.First, the depth value   of each scan point is calculated as the distance from the scan point to the virtual plane.Then all the scan points are projected on the virtual plane.The upper surface area is projected on a region with 300 × 300 grid nodes.Every projected scan point finds the nearest grid node and calculates the distance , as shown in Fig. 4. The accumulated depth   of the ℎ grid node is calculated by, (1) Subsequently, the 300 × 300 grid nodes are pixels of the depth image.The value of each pixel is the ratio of depth   and layer thickness.The pixel value is rendered into eight colors to visualize the depth image, as shown in Fig. 5.

Statistical representation of regions and multi-region structure
In the previous work by the author (Lyu et al., 2021) a rigid labeling method was implemented to describe the data by assigning only one of the possible labels to the top surface scan segment of each part.With this approach, the dominant label present in the segmented grid was successfully predicted with high accuracy.However, rigid labeling of the top surface raises some concerns due to the inability to generalize and elaborate the information within each grid.In some cases, the grid might also simultaneously include a portion of various printing conditions.This example can be seen in Fig. 6.As shown in Fig. 6, some segments might contain more than one condition by discretizing the image into smaller segments.To better illustrate this, three regions within Fig. 6 are highlighted.Region A is mostly shown to be normally printed with some small cracks and under printing signs.By relying on the rigid labeling method, this grid would have been labeled as "Normal", when obviously, it contains cracks.Region B, on the other hand, contains some over extrusion marks.By ignoring this fact, after a few layers are deposited, the overall error would stack up into a severe problem.This can be avoided by using the relaxed labeling method, in which the controller can be informed of the presence of such overprinting signs, as it happens.In grid C, we see strong overprinting and under printing situations in which assigning only one label to describe it is insufficient.So, a relaxed labeling method based on the statistical possibility of containing each label is suggested to address these issues and enable the network better to describe the printing situation for further controller interaction.
Features selected for this study can be categorized into four dominant situations 1) Under-printed regions, 2) Over-printed, 3) Normally printed, and 4) Empty regions.To implement a relaxed labeling method, simplify the labeling process, and describe the grid's information as accurately as possible, the OUVE method is suggested.In the OUVE method, we created four channels for Okay/Under/Over/Empty labels, respectively, with values between 0-1 assigned manually by an expert.
The first three channels suggest the presence probability of each printing mode observed in the grid understudy, and the last channel indicates the emptiness of that grid.Therefore, all possible combinations of labels, such as complex situations with multiple situations in each grid, can be described and used to train a machine-learning algorithm.To better illustrate the labeling method, a region is considered consisting of overprinting and under-printing anomalies at the same time, as illustrated in section C, in Fig. 6.To label this region in the previous rigid labeling method, the user tasked with labeling had to decide between "Overprinting" or "Under-printing", whichever has the strongest presence.While in the suggested method, the user can suggest different weights for each label and describe how strong each label is present in the specific grid.It is suggested using [0.5, 0.5, 0, 0] to label the example mentioned above, which means a probability of 50% Under and 50% overprinting situation has been observed.
A summary of pre-set labels and their according categories used to label the dataset and train the network is presented in Fig. 7.For labels with a dominant presence of a label and no significant sign of other labels, it is suggested using a full channel for the dominant label and low on other channels as in: fully okay, fully under, or fully over the situation.For cases with two labels present in the grid understudy, a half channel for the top two dominant channels has been utilized as okay_under, under_okay, or okay_over.For cases in which the grid contains less than 5%, pixel-wise, of the part understudy, a full empty label has been assigned automatically using the pixel counting approach.On top of rigid labeling, which limits the insight gathered from the data, another common problem observed in the previous studies is that researchers are either focused on analyzing the whole surface at once to assign a label to the surface.Or they discretized the surface into smaller grids and performed the analysis to gather a matrix of labels.Although method one would suggest the dominant printing situation present in print, it wouldn't be able to describe all the sub-dominant possible situations present or the features which require different treatments than the others, which for a thorough study of the print's dynamic would be necessary.
The top surfaces might consist of different features and geometries, leading to different printing scenarios, just as the different strategies used for printing walls of the part and the infill regions.The later studies that used grid surface representation in some levels addressed this shortcoming.By assigning labels to the smaller grids, more information for studying the process would be possible; however, by studying each grid individually, we would lose the neighboring and correlations present between the grids.For example, if a severe bulgy region is present, the regions around that bulgy section must also get impacted in the next depositions.At the same time, by looking at the neighboring segments individually, the severity of the problem might not be feasible to detect.Correlating each grid to the neighboring grids allows the study to consider more in-depth information regarding the print dynamics and minimizes the effect of noise in the labeling process.In our implemented method, a multi-region analysis would be performed to study the laser-scanned top surface of the artifact.The implemented method would cast a 9 × 9 grid onto the scan and simultaneously predict all labels in a 9 × 9 map.This way, the machine-learning algorithms used after each layer of neurons would consider the correlation of the grids and their neighbors for a spatial study.

Multi-region Hybrid Convolutional Auto Encoder
With the appearance of growing graphical processing units (GPU), Deep Convolutional Neural Networks (DCNN) have become popular for processing image-based data and features and information extraction that are not feasible to do manually in many cases.CNN layers apply a filter over the input data and output the result.After exposing the network to the data, the filter's weights will be learned and applied to the backpropagation of network output and the expected return.Although CNN-based networks can extract useful information and features that ease the pass for researchers, they need a significant amount of labeled training data to learn the filters' weights and generalize the problem to future unseen data points.A combination of Auto Encoder networks with CNNs suggested by (Dosovitskiy et al., 2014) will address this shortcoming.The implemented architecture for this study is illustrated in The network implemented for this study mainly consists of 3 branches, 1) Encoder branch, 2) Decoder branch, and 3) Classification branch tasked with dimensionality reduction, data reconstruction, and labeling, respectively.
Encoders are suitable for unsupervised data dimensionality reduction and feature extraction due to their symmetric structure when coupled with decoders.In this design, the encoder branch gets a batch of input images with size 288*288*1 and outputs a 9*9*4 tensor 25.6 times smaller than the input size.This branch is structured of 5 Halving Blocks followed by an inception block.Halving blocks are designed to reduce the dimensionality of the image by half.They are constructed of two consecutive CNN layers on top of an Inception layer followed by a max-pooling layer.A detailed representation of the halving block is depicted in Fig. 9.The filter size in the CNNs utilized is decided to be 15 filters of size 5*5 after hyperparameter optimization.The activation function for each inner CNN layer in this branch is rectified linear activation function (ReLu) to bond the values to a non-negative number.The ReLu activation function outputs a linear value for positive inputs and zeros for negative inputs.The mathematical representation of the ReLu activation function is as follows: The batch normalization method and drop-out layers have been utilized to ensure this branch's generalization capabilities and prevent overfitting.The batch normalization method (Ioffe & Szegedy, 2015) is an approach to increase the network's generalization by centering and normalizing the layers' weights, in this case, the CNN layers.This approach guarantees that the layers' weights wouldn't diverge to really large numbers, which would cause the network to fail if it happens.On the other hand, the drop-out method focuses on the fact that the network should understand and process the info even if some data is missing.This idea encourages the network to generalize its feature detection and prediction by working on a portion of the input.The drop-out layer randomly removes some inputs to the next layer in every training cycle, depending on what percentage it is set to.For this study, a drop-out rate of 20% has been picked after hyper-parameter study and optimization, which indicates that only 80% of the data points to the next layer would be passed through the drop-out layer, and 20% of the data would be valued at 0.
The inception module, which is the base of the Halving module, has been used various times in many other sections of the HCAE network to analyze the input and apply multiple filters with various sizes over the input data to consider all possible aspects and extract the most meaningful features by widening the model.This module was first introduced by (Szegedy et al., 2015) and showed its feature creation and data processing capabilities.Inception was utilized because this module keeps the input shape while adding additional processed layers to the input, which helps with a multi-aspect study of the data.The inspection module is depicted in Fig. 11.This structure performs multi-aspect analysis by creating four data processing channels, each with a different number of filters and filter sizes.The first branch utilizes a 1*1 filter over the input to get the smallest features.Then, the second branch applies a 1*1 followed by 3*3 kernels to get the neighboring correlations.The third branch expands the analysis using a 5*5 kernel to consider further data correlations.The last branch utilizes a max-pooling layer before a 1*1 kernel to add more layers of information.After that, a big tensor of processed information the same size as the input data would be generated by aggregating and concatenating all the resulting data.
The details of the channels in the Inception module are summarized in Table 1.To reduce the dimensionality of the data stream, a Max-Pool layer is attached right after the CNNs.Depending on the kernel's size, the Max pooling layer reduces the data's dimension.Here, the kernel size of 2*2 is used to cut the data size by 4-times, half in width and half in length.
In summary, this branch not only lowers the data dimensionality but also creates a data bottleneck that will be used for both Decoder and Classification branches which will be discussed next.The lowered dimension data will then be fed into the other branches to reconstruct the input data or perform categorization analysis.The decoder branch is designed to reconstruct the input data both in shape and value from the lowerdimensional representation of the input generated by the encoder branch.The input to this branch is of size 9*9*40 and outputs a 288*288*1 height map matrix that, throughout the learning process, is encouraged to get as close as possible to the input.The Decoder branch consists of 6 stacked up-sampling modules designed to reverse the dimensionality reduction process in the Encoder branch.The architecture of the upsampling modules is represented in Fig. 10.This layer utilizes 2 CNN layers followed by an up-sampling layer, increasing the input dimension.The CNNs used in this module have 12 filters of size 2*2 and a sigmoid activation function.Since the height map must be a number between [0-1], the Sigmoid activation function will ensure that the data would be limited to this bound.With a kernel size of 2*2 and a bilinear interpolation method, the up-sampling layer is the key for the decoder branch.After dimensionality increases, an Inception block is applied to process the interpolated data better.
Since this reconstruction only relies on the raw data, the Decoder and Encoder branches can be trained automatically without supervision, which would help us train the network with all the available non-labeled data.Using AE networks to generate a meaningful lower-dimensional representation of the input data that doesn't rely on any labeling has shown its capabilities and efficiency in the author's previous work (Lyu et al., 2021).In every training attempt, the Decoder branch generates a batch of outputs.The branch would be trained by comparing them to the initial inputs using Mean Square Error (MSE) loss function and applying a backpropagation technique in both Decoder and Encoder.After sufficient auto-training of the AE branches, the bottleneck of the network, which is the layer connecting the Encoder to the Decoder, will be used to introduce the Classification branch to the network.
The classification branch consists of an Inception module followed by a simple CNN layer with a customized SoftMax activation layer.The input of this branch is the same as the Decoder branch, 9*9*40, and the ideal output for this branch is set to be 9*9*4.This means we have a 9*9 grid and four channels of probability data for each grid.The SoftMax activation must apply on each grid individually to convert the numerical data to a statistical probability representation.The mathematical representation of SoftMax is: Model training for HCAE model consists of 3 main steps, first by turning off the classification branch and using all the data, basic feature extraction is created.Then by switching to the classification branch and turning decoder branch off, 80% of the labeled data as training data are introduced to the network to create a basic label estimation.And finally, by including both decoder and classification on top of the encoder branch, and using the training data, the model is fine-tuned.Multi-step, dynamically varying learning rates (0.01-0.00001) and batch sizes (124 -8) method is used to train and fine-tune the model to the highest accuracy while preventing over-fitting.After sufficient training over the training dataset, the encoder branch would be discarded.An example of the network's input and the corresponding output is illustrated in Fig.
12.This Figure shows seven height maps in grayscale and their corresponding prediction of labels by transferring the statistical channels to an RGB format.The blue represents the region with the Underprinting situation, Red represents overprinting, Green represents normal printing, and Black represents empty regions.As shown in Fig. 12, the network successfully represented the regions containing more than one dominant label.In mixed-condition regions, a mixed-color representation was utilized.For a region with over and under-printing, purple (Blue+Red) color was used to suggest the presence of multiple situations in the segment.The stronger the statistical result of a specific label, its color channel is stronger.

Feedback Control for Process optimization
The additive manufacturing processes heavily rely on the pre-determined parameters set by the slicing algorithms.Finding the best set of parameters in many cases requires many trials and the Design of Experiments (DoE).On top of that, due to the lack of machinery's health monitoring systems, many systematic and random errors might appear within the process, which makes the AM techniques less reliable.To address this problem, a direct and dynamic adjustment of process parameters is needed to consider any anomalies that might happen during the print, either from the machine degradation or an inefficient selection of process parameters in the pre-processing and G-code generation.In this article, we have selected two main attributes of the machine printing speed and extrusion rate to control and optimize the quality of the part.These two parameters are chosen since they have a significant and fast effect on the part's printing quality.Their impact on the quality has been seen to be the reverse of another.The higher printing speed, the less material is deposited along the printing path, decreasing the print density.In contrast, a higher extrusion rate causes more material deposition and higher density.
The control algorithm starts with pre-processing the future layers' G-code.The main part's coordinates are extracted by analyzing the G-code, and the control frame would be limited to the main part's coordinates and excludes the axillary prints and supports.After extracting the main part's coordinates, the G-code of the first three layers would be sent to the printer's computer to actuate the machine and initiate the print.The three layers deposition builds up a base for future depositions and allows the point cloud processing to localize the main part's data and distinguish the part from the printing bed.After the successful deposition of the initial layers, the laser scanner system would be engaged to capture the first part's top surface point cloud.Then, the point cloud would be processed, aligned, filtered, as mentioned before, and transferred to the main processing machine over the cloud.Once the height map information has arrived at the main processing unit, the multi-region HCAE network will be engaged to process the height map and study the surface quality to generate the grid-by-grid information.The output of the multi-region HCAE network would then be transferred to the control algorithm to generate the control signals.First, the Control algorithm processes the next layer's G-code in line and manipulates the speed and extrusion rates within the code based on the earlier quality assessment.In the end, a new manipulated and extended G-code would be sent to the printer's computer to perform the adjusted print.

G-code manipulation algorithm
G-code manipulation starts with loading the future layer's G-code and the statistical study of the surface from multi-region HCAE.In this algorithm, from processing the previous layers, we have gathered two important information 1) boundaries of the main part and its coordinates to register in space and cast a 9by9 grid over it, also 2) the previously used parameters and control signals used to perform the print of each grid to be used for closed-loop control.After loading the prior info, the algorithm loads the G-code line by line and determines whether the command belongs to the main body.The algorithm skips the command without manipulation if it doesn't belong to the main body's print.If the G-code falls within the part's boundary coordinates, the movement would be discretized into smaller motions bounded by the grids' borders.The control algorithm would adjust the properties of each segment of the movement.Print speed and extrusion rate have been selected for this research to optimize the process.In each line of G-code, a few parameters are passed to the machine to determine the movement coordinates and their attributes.The code used by the machine for deposition is of form G1 command followed by destination coordinates (X, Y, and Z), filament length extruded (E), and also the speed (F) in which this movement must take place.To control the printing speed, the F parameter value, which is the motor movement speed in millimeters per second (mm/s), would get updated based on the history and the controller commands.To ensure machine safety and printability, the F parameter was bounded to 8000 -100 (mm/s).To control the extrusion rate, the M221 command has been utilized.This command gets a percentage as input, indicating the multiplier to the extrusion rate defined in the G-code.Due to machine limits, this feature was bounded to [2 -0.2].The updated G-codes would then be saved, and, in the end, the updated G-code would be transferred over the cloud to print.The controller's goal is determined based on the previous layer's print.The next layer's goal signal is generated based on previously printed geometry.The goal signal has 100% normal or 100% empty signals depending on the area of the part that maps to the regions.The threshold for this goal generation is set to 5%, the same threshold is used for the data set labeling, and the regions containing less than 5% printing area are considered empty.After this step, a goal signal with a dimension of 9 × 9 × 4 would be available to the controller.If the region aims to get a normal print, the controller aims to increase the density in the under-printed areas and decrease the density in the over-printed areas.The controller manipulates the process parameters based on the probabilities observed in the top surface to achieve this goal.If the region is meant to be empty, the controller maximizes the empty signal by minimizing the deposition in that section.This way, by minimizing the Over and Under-printing signals, the controller maximizes the printing outcome to get as close as possible to the generated goal.The summary of the controller's logic is depicted in Fig. 13.The advantage of the goal map is to be dynamically flexible based on the part's cross-section.This algorithm doesn't rely on the geometry or features present in print since a new goal map would be generated depending on the new geometry on each layer.
The controller manipulates the parameters depending on the probability of the observed state in the specific region.Based on our observations, to reduce the print's density, we must lower the feed rate and increase the speed and vice versa to increase the density.Therefore, based on the values observed in each grid, the controller changes the print parameters accordingly.In some cases, we simultaneously observe underprinting and Overprinting situations in the same region.To address this scenario, a small bias towards increasing the print speed is implemented to break the tie between the signals and get a faster and more efficient print.
The controller starts with a predefined step size, limiting the maximum effect on the process parameters.By every iteration, if needed, the step size of the controller would dynamically increase or decrease.If the region is under a serious defective print, such as a strong Under or Overprinting situation, the step size will increase to expedite the recovery time.On the other hand, if the region has achieved a strong satisfactory improved result, to help the convergence, the step size decreases.The steps would be reset to the default value if the controller is saturated.The summary of the steps taken by the controller is depicted in Fig. 14.
A surface roughness measurement criterion is needed to measure the effectiveness of the developed algorithm.Therefore, having the processed heightmap data as the output of the LSP module, a dimensionless number f(Ra) is defined to describe the evenness of the artifact's top surface.The f(Ra) value of the surface can be obtained by taking the mean of absolute heightmap values.Equation ( 5

Results
The theoretical side of this experiment heavily relies on the controllability and stability of the environmental factors during the tests.The FFF machines face numerous random deviations due to ambient temperature gradients, machine vibrations and degradation, and G-code incompatibility.To that aim, an online monitoring and control algorithm can help with the necessary customized adjustments to successfully execute the process even under random errors.
The HCAE model were trained over 11,000 unlabeled images and 1400, 9*9 labeled images.To increase the training size and more generalization, the data were augmented by shifts, rotations, and flips.After sufficient training, to test out the HCAE model, first the model's prediction accuracy on unseen data inputs labeled by experts are measured.Thereafter, in practice test of the controller capabilities on 2 parts sliced with inefficient and different parameters is mentioned.The artifacts under study started with a severe quality problem and the algorithm successfully mitigates the inefficiencies in the G-code and the machine and converges to a high-quality part after a few iterations.The breakdown time of each processing step is summarized in Table 2.The total processing time to generate the updated G-code takes a maximum of 18 seconds as shown in the table.This maximum time is small enough compared to the average layer print time of 120 seconds, which will assure print continuity in a real-time setting.When the scanner and processors work, the extruder head is parked on the side of the printing area.Before continuing to the next layer, a wiper would clean the extruder to ensure the idle time won't affect the print.

HCAE results
The HCAE model is designed based on the relaxed labeling approach to achieve a descriptive result rather than traditional decisive models.The network's outputs are mutually dependent and include the most dominant aspects of the data rather than suggesting the strongest feature.This difference makes the performance assessment using traditional methods based on completely independent labels incapable of capturing the suggested network's capabilities.To better illustrate, let's pick a region manually labeled as a fully under-printing situation, and HCAE output suggests under and overprinting, which includes underprinting and over-printing regions in the grid.Traditional categorical assessment methods created based on hard labeling would penalize the network's accuracy due to dissimilarities of the labels and their corresponding prediction.However, the HCAE model should still get credit for suggesting half underprinting situation, which was also observed by the label.The extra half-overprinting information would be beneficial for the controller but misleading for categorical accuracy assessment methods.It is suggested that a Multi-Label Confusion Matrix would be helpful for multi-output models (Heydarian et al., 2022).However, their proposed method penalizes the network with the same weight as an utterly wrong classification while, in a descriptive task, for a partial correct description the network must earn a reward.Therefore, a new accuracy measurement metric was developed to address this problem, based on the partial penalty and rewarding policy for multi-output descriptive models.This new assessment method fully credits the network if the label and the prediction are fully aligned, while it partially credits and penalizes if the estimated description is partially correct, like the previously mentioned example, and penalizes in full if the prediction is completely off.We will name this method as the HCAE metric for future reference .Fig. 15 has summarized the credit policy for the HCAE metric.A confusion matrix is created based on the HCAE metric discussed above to summarize the network's output accurate results.This table shows how the network has performed for each specific label in the dataset.The HCAE network has shown strong capabilities in detecting Under/Over/Okay labels with more than 90% accuracy.In cases with two labels assigned to the grid, the network showed above 80% accuracy.
The drop in the accuracy compared to single-label regions is mostly due to the detection of only one of the two labels.After reviewing those mislabeled sections, the network suggested the correct label in some cases, and the manual labeling due to human errors was not perfect.The implemented HCAE model's confusion matrix is depicted in Fig. 16.
To better compare the HCAE model's prediction accuracy and repeatability, the F1-score criteria have been utilized.This scoring method combines each label's precision and recall values to show how well the model performs in that category.The mathematical representation of recall, precision, and F1-score has been demonstrated below: Once the F1 score for each label is available, the categorizer's F1 score, in general, can be calculated by simple or weighted averaging of each label's scores.The resulting F1 scores for the HCAE model are summarized in Fig. 16.

Controller Results
For testing out the implemented controller, two parts with different geometries have been selected and sliced into 20 layers with different inefficient manually selected initial parameters, as in severe lack of extrusion and severe over-extrusion.The controller is tasked with compensation for these undesired printing situations.The first three layers showed a significantly low quality; right after controller engagement, the quality increased drastically.
In Fig. 17 and Fig. 18, the results for both experiments are depicted.In the first experiment, part #1, with a square shape, regions in the middle that faced severe under-printing (lack of material) and low density are densified, and corners with over extrusion situation are lightened.In the second experiment, part #2, with a custom shape and different parameter sets, the quality of the part in various regions needed densification or de-densification; in both cases, the controller managed to capture and compensate for the under-optimal parameters in the G-code.In testing, the controller showed great capacities in capturing overshooting situations and reversing the change rates to stabilize.An example of overshooting can be shown in the left boundary of Fig. 17.The left boundary at first started as a severe under-printing; in layer seven, the layer got too high , surpassed the desired point, and became over-printing.But after two more iterations, the left section stabilized in a dense green region.

Conclusion
In this paper, the point cloud data represents the top surface of printed parts for each layer.Starting from the generated point clouds and subsequent processing, the height map of the part for each layer is obtained.In this process, the computations use the RANSAC method to best fit the reference plain to the point cloud data.Once the data is processed, the HCAE's structure allows unsupervised training capability over unlabeled data.This approach significantly reduced the reliance on a manually labeled dataset which is time and labor-intensive.Then, by introducing a small amount of manually labeled data, generalization and classification accuracy were significantly improved.The HCAE model's design, not only considers the neighboring correlation of input data segments but also statistically represents each grid over the possible labels.The relaxed labeling, prediction, and statistical representation allowed the HCAE network to overcome the conventional hard labeling limits and describe the observed data in more depth and detail.
The resulting information generated by the HCAE model is then used to perform an in-situ process adjustment.The controller implemented for this study aims to densify the part surface and create an even surface finish.The effectiveness of this approach was tested by printing two manually defective parts.Both parts started with a significantly deviated state and within 4 iterations, the controller managed to converge the quality to an acceptable state and prevent loss of material and print failure.

Future Work
The proposed approach in this study can be further expanded to include various sensory inputs to fuse various aspects of the ongoing process.Such as a co-axial camera or a side-mounted camera for geometrical verification and registration.The deep model can easily expand to include image inputs as well.On top of all, the proposed control method can expand to include even more process parameters that are controllable within the G-code file.This study is not limited to the additive manufacturing domain and easily can be integrated into various industrial applications.

Fig. 2 :
Fig. 2: Experimental setup (a) CAD model (b) overview of the in-situ quality assessment system.

Fig. 3 :
Fig. 3: Point cloud noise from shallow effect (a) CAD model of the scanned part (b) Scan process (c) Point cloud after scanning (the normal point is shown as white, and noise is shown as red).

Fig. 6 :
Fig. 6: Selection of regions with multiple labels present, a) okay and Under-printing b) okay and Overprinting c) Over and Under-printing.

Fig. 8 :
Fig. 8: The architecture of the multi-region HCAE network.On the left batch of scans is provided as input, and on the top right batch of Labeled inputs and bottom right, a batch of reconstructed scans is achieved as outputs.

Fig. 8 .
To better analyze the data aggregated from the laser scanner, multiple layers of CNN layers and Inception modules have been stacked on top of each other to create a wide and deep network to study the input and extract the possible features.This structure eases the feature extraction and dimensionality reduction process by unsupervised training of the Convolutional Auto Encoder-decoder section.This technique has been utilized in the previous study, showing a significant improvement in accuracy and training costs.

Fig. 12 :
Fig. 12: Multi-region HCAE Height maps as inputs and RBG representation of output labels for various shapes.
Fig. 13: a) Scheme of the Controller logic.The top left shows two examples of labels seen while printing, and on the right, their corresponding goal is generated based on the G-code.b) Controller's simplified policies.

Fig. 14 :
Fig. 14: Steps were taken to generate a new G-code.A) G-code processing and registration of the 9*9 grid, B) Generation of Goal map, C) performing the print and gathering a height map from the top surface, D) Creation of Labels based on the Heightmap by Multi-region E) Grid-wise manipulated G-code.

Fig. 15 :
Fig. 15: Custom HCAE metric flowchart developed to implement partial grading policy for multi-output models.

Fig. 17 :
Fig. 17: Experiment results for part #1.a) The starting quality and final quality of the part.b) Five iterations of the controlled manufacturing.

Fig. 18 :
Fig. 18: Experiment results for part #2.a) The starting quality and final quality of the part.b) Five iterations of controlled manufacturing.

Table 1 :
Summary of Inception module design parameters.

Table 2 :
Breakdown of the Processing times