This section presents how experimental data was generated and how each model was created and trained. We also explain the tests conducted to assess the performance of the models and the verification done on the results.
4.1 Problem Definition for Case Study Area
The case study organisation for this research is an aircraft manufacturer and the studied processes are drawn from the assembly stage for the wings. The current installation process for components on the inside of aircraft wings requires workers to place and secure them in defined places, as indicated by the wing frame suppliers. A component of difficulty for the case study organisation engineers is the wing brackets which secure electrical and hydraulic lines, seen in Fig 5. This requires the engineer to adhere the bracket to the wing, within the marked boundary, with only limited visibility. The conditions for a correct application of the bracket are that it must be within the boundary within a 7mm tolerance, and that there must be flow of adhesive around the whole circumference of the bracket. Fig 5 shows these conditions for installation.
In the case study organisation’s drive to automate the verification of aircraft wing assembly, the installation of these brackets will be a key task. The automatic verification of the bracket installation process will have to respect the criteria that the human operators adhere to. Therefore, the solution proposed must be able to determine whether a 7mm tolerance is maintained around the edge of the bracket, and whether there is complete adhesion around the whole circumference of the base. If these criteria are met, then the bracket can be considered installed correctly. If either of these criteria are violated, then the solution will have to identify this bracket for second inspection by a human operator, followed by a corrective procedure if necessary.
4.2 Component Representation and Data Generation
A decision was taken to generate a custom dataset for this project because of the lack of open source databases with the brackets needed for aircraft installation. A suitable mock setup of the bracket installation process was developed. The replica bracket was created using LEGO® bricks to represent the bracket and modelling compound to represent the adhesive applied to the bracket base. Modelling compound was chosen over a true adhesive for the ease of removal and reapplication. A solid blank background with a slight shine and texture was used to represent the wing surface, and a suitable boundary marking applied to locate the placement of the bracket (see Fig 5).
Experimental data was generated by taking images of the bracket installed both correctly and incorrectly. Consideration was given to the subtleties in each of these cases, to identify the sub-cases that might exist within both, and to ensure that data was gathered on as many variations of the installation as possible and supporting generalization of the trained model.
For the correct installation case, this involved all those installations which met the criteria as discussed previously. Special attention was paid to the edge cases, where the bracket was just within or outside the tolerance range from the marking. It was expected that these would be the most difficult for the models to distinguish between correct and incorrect installation, so sufficient images of these were obtained to support any conclusions drawn from the results. In the incorrect installation case, there were several sub-cases identified that the model would have to learn as belonging to the incorrect distribution of images. These were as follows: (1) Inside guide, no adhesion; (2) Outside guide, correct adhesion; (3) Outside guide, incorrect adhesion in one location; (4) Outside guide, incorrect adhesion in many locations; (5) Outside guide, no adhesion (Fig 6).
Images were obtained for each of these sub-cases, again with attention given to including images of the edge cases. By ensuring that all these cases were covered in the data generation, the chances of inherent bias in the dataset were mitigated. Hence, any results obtained in testing would be representative of what would be expected given a true dataset. This dataset was validated by an Aerospace expert as what would be expected on a true production line. The images taken were RGB images of shape (256, 256, 3). For training, there were 70 images of correctly installed brackets and 70 images of incorrectly installed brackets, totalling 140 training images.
In order to replicate a low data scenario, 70 images of incorrectly installed brackets corresponding to 10 images of each of the 7 sub-classes identified previously were collected. For validation, there were 100 images of correctly installed brackets and 100 images of incorrectly installed brackets, totalling 200 validation images. Furthermore, a framework to procedurally load the generated images and perform random augmentations was created. This included affine transforms of translation, rotation, scaling, shear, horizontal and vertical flipping, and the colour space transform of scaling image brightness. The images were also normalised to aid the gradient descent algorithm in minimising the loss function.
4.3 Baseline CNN Model
A baseline CNN architecture was implemented (Fig 7) and trained as a binary classifer. Each convolutional layer was followed by batch normalisation, a ReLU activation function, 30% dropout, and a Keras max-pooling layer with default settings. Optimisation of the model was achieved using the ADAM algorithm with a learning rate of 0.0001 and a binary cross-entropy loss function. Training was performed over 25 epochs with a batch size of 16. Images of either correctly installed brackets or incorrectly installed brackets were passed to the network during training, along with their class labels. The model predicted the probability of the image belonging to the positive class, which in this case was of the image showing an incorrectly installed bracket. The default threshold of 0.5 was applied to determine the binary class prediction.
4.4 SNN Model
The SNN architecture was implemented with two baseline CNNs as seen in Fig 7. The dense feature layers of each twin layer were followed by 30% dropout also, however, no dropout was added to the L1 distance layer. There were no learnable parameters in this layer and hence no chance of overfitting. Optimisation of the model was achieved using the same parameters as the baseline CNN model.
The SNN was trained by passing pairs of images to the twin CNNs (see Section III.D for further detail on the selection of this pair), along with the label of whether they were of the same class or not. The identical CNN networks with shared weights would then perform the same feature extraction on the
images, producing feature vectors to be compared in the L1 distance layer. The distance between the feature vectors would be calculated as in Equation 1, where p and q are the two n-dimensional feature vectors. This would then be converted to a similarity score at the output layer by passing it through a sigmoid function, where a value closer to 1 denoted a higher confidence that the images were of the same class. A default threshold of 0.5 was used to determine whether the final decision was that the images were of the same class or not.
![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAnkAAABGCAYAAACwuoLwAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAFiUAABYlAUlSJPAAAABfaVRYdFNuaXBNZXRhZGF0YQAAAAAAeyJjbGlwUG9pbnRzIjpbeyJ4IjowLCJ5IjowfSx7IngiOjYzMywieSI6MH0seyJ4Ijo2MzMsInkiOjcxfSx7IngiOjAsInkiOjcxfV19o2gzSgAAD7RJREFUeF7t3TuP41QbB/Bn3m+AoKRAkCCxGgQFIOTAtijZJjTDUoVi5Ug0M81ssQwV2WZXSEkDSrQFabgMzTSTaFtEUgAFiNEijQ0VLZ8h73mOjy9x7PiS4yT2/H+ScXBmk/h2zuNzPVgIBAAAAACV8j+1BgAAAIAKQZAHAAAAUEEI8gAAAAAqCEEeAAAAQAUhyAMAAADI4Y8//qBHjx6p/8vms88+k/++SOhdCwAAAJDR999/Tz/99BN99dVXasuyn3/+mb799lt66aWX6P79+2qr77///qPPP/+cbt++TXfv3lVb9UJJHgAAAEAGHOB98803kQEeB3dcSvf+++/T119/rbauev755+mLL76gL7/8Un5eERDkAQAAAKTEVawff/xxbAnee++9Rw8fPqQHDx6oLfE40Hvy5In8vCKqbhHkAQAAAKR07949+uijj+jll19WW6K9/vrr6tV6b7zxBn3wwQfyc3VDkAcAAACQAler/vbbb9Rut9WWeC+++KJ6lezDDz+Un6u72hZBHgAAAEAK3A6PvfPOO3Kty7vvvivXFxcXcq0LgjwAAACABNwb9unTp/J1UlVtVlxly3744Qf5PbogyAMAAABI8Ndff8n1W2+9Jde6Pffcc3Ltfo8OCPIAAAAAEsznc7nmHrFFcKuA3e/RAUEeAAAAQAUhyAMAAACoIAR5AAAAABWEIA8AAAAggWEYcm3btlzr5vaqdb9HBwR5AAAAAAlee+01uf7777/lOsm///6rXqXDgyEz93t0QJAHAAAAkIB71fJ0ZizNPLPuwMY//vhj4th3//zzj1zz9GY6e+8iyAMAAABIwZ3ObN1Ydo8ePaKDgwM5sDHjEroXXnhBbo/zyy+/yPUnn3wi17ocLAT1GgAAAADWePvtt+mVV17ROs/s3bt3ZTXwr7/+qrbogSAPAAAAICWuqn3zzTfp999/96Yj2wRX1XLQqOvzglBdCwAAAJASB2Lfffcd3bt3T23ZzIMHD+Tn6Q7wGEryAAAAADLi6to///yTHj58qLZk9+mnn9Lt27dldW0REOQBAAAA5MBVt0+fPqX79++rLelxRwzuTVtECZ4LQR4AAABABaFNHgAAAEAFIcgDAAAAqCAEeQAAAAAVhCAPAAAAoILQ8QIAbiib7KlFFtWp2aypTTZNLUu8qFNdbFNbAQBKqeQleZxId6nRaFB3qjaVkT2gxoHYh8FU7FE1TLsHcu6+g9CJsQeNyO2Q7GYf0yl1eR83XRoD5x6zxec1OvT4oketVt1JPzgt6XTooudsq+/geLrnsjGoSkqw33C8oeoKCfK8zMhdCkgsZXB3IBLi1hUdno3ptKneKKPaMY0nh3R13qI6B3tTJDgAy5o0nJjqtcOcLIgrItYu1oT6hvoHAfblNbXHMxqeHpF8+3pA3Ys2zWZi2+yM5DddXVfmoQsAbqZCgrzmUCSuIkF201bjVl290oOfvuqtEc3NCVkLkShXoFql1hyKDMYizsdGrTo1Kl0qA5CDuEeCcd6o1aXEu6TWpOPZxAnaAmrHxyRraK1nNBer0TnR6TD0pHj4KqprAaDUiquurd9SLzit1JdUygDvRCTLIsBbiES5WolwTeRjM5mRzUctVCEAhDSHwYBtRK1UD0NNOo0qzhPs6yu5Ns+O/bTEvibeqvvhFABg27bQJs+ktq6qVHtAHQ7wxGdOwk/dFdIcWrKKaX6i2goBgBKqthUPQ2nukdodVS0bYj1z0pNgGmVfntNc/PXRHZTjAUC5FRbkOQmlYNwiPc/DNg06J/Izjf6pSOqrrEbHZ05GNuqphuIA4MhVbXuHjlaivCldjMTKbC+lJ07gd0gaKyAAAHaisCDPSSgFXe1a7Es6lx9p0tnxDUh9m22nWmp+Qo9RmgewJHu1rXhwmi1oMUuqlrVJ1uCGAj8AgDIqKMhTCaVgaqqrnT52SvFSJb72lAZdNawEd4/vhkrD5PAJ/vsHjS7p7tBqTwfL3yGXBjXEdw0G3DOY/1+8jv3eJrVVLja6uEFR3h6cu6Jtfm1AVLVt1jasbm3Dcpthi9znU3ktlvD6AgDwLDZhWYuJaSwM8TH8UXIxjIXZN9U2Y9G31N9uxFr0DefzjaQPtPrLv0ct7r+z+sbKe3Ix+uJbNLAmC1P9Vl783zvx9sFfTLF1jYmZ7u/2kMh/nd9uLv9y7/iHtku7PndF2/DayHVMK847JnLJlt44/zZ8nP20hgxTU/qVnnsu49I5y0tb+fctX/eWSC/897b/2wsVm9dM1H7ny2uSjjdA2eUO8vwERdxcE/8G4e3eTagrOPEy/4QbWf2dwTd+MLHmhTNA/s0i8XN+LicagffFsnEeGQpSwgnHSpCS9IVp9ztSaP9zLnkTv8wBya7PXdE0XBuZj2lhdnttLROBc/BzyxLwx4gPOviYi6BG3gD+8XdO+er94Cz5Ap99U2ReE3+8AaohX3XttOuMU8e9XBczOnanBBJqblsypqtdixrLKo1Dc0Kz44ihVa561OgRnc2GzvhY4i+abn2ocnW9Sb2M3zFEMvo0Tmg7mDhEQ+1VOpQv5vSMZ1qquN2du6IVcG2AEqq2nZ9Qp4L13Pb0MT07GssxQcOm3Tq1rvok4h85ALQIftQ7czq/LPmxSJvXaOvgB1AtOYK8KXXFTcfMich05ato+jOqhB5vtWMaekOrBNrWMPH6aLz+925EJMJydBfFOLqzEqx4nVGULOMHZg9iVEPz4Oj/OZbZtjq57PLcFa3ga2P79uzaCvW2nZ90KteekQdLH3rHy70/DLp13aWeCPCsmRrcWQgGP/NSPx1myGsi7ikAyBHk2YMeydvO6EdPJaZ6rLHkjMqmaYpWze6ApZlML5zf6TKOKDzsVa7PjWTToBf8tqgewH5nFIfG8QOrptBzJ85VI9jhId+SvpH/vl0b6e65slnubVtxXho7p5MTLuEO9BiW6nTLUC8j2LYtroKstn3fiG/UkNfk21eA6sgY5E3psSqSiH1y8qpW12dU7tyzrd5lITdhOAhYGtFe0VZ64g3vokRVHaT5G5C2eu6KtkfXRtH33L4wJzPKWkhYfDCgL7j2xiAVkkq4VnD1Z71egmGZNOQ1pdlXgOJkC/K8J6f40eCncnRRIS6jkkNgHFCnx+0s0qm96rRMy2I5CDBoteY4XHoS9TcphdsMRo0NmOZvQCr23G25qnEfro0c91zZTLstp9THnFDmyXAKDgZ0B9fe/RFXwhVo7hAewoofoAxDBEWZ07ot3zca8pr8+wpQHZmCPP8JMqZtnD0gr2YqKqMS73c7F9Qei5s9YtLwZFeUrmmaGsneFVHdF0wIpci/SSdc8rTaFjFcZbea+CbJXlK1/eoVPbZ77oq2jWtjrY3vuSj7dW3xfNay6ZYIeqwc0x0WFgwUElz7DzixJVxec4fVh5/a8UxcB0Ov/d6+2jivEcqyrwBFytHxIo5I+AM9CCMzKm5cn+emq98SyRVL2cs00FZDikgEvPYeSlSVYF4rAVmo4X1U4hsp8DSbvqSq5HZ87oqm7dpIK+89VxZcCicPoEH9cb7roJBgoJDgWghU7cc9+HklXOaZX21tc3Xx1FnE63JLyGsqta8Am8kZ5K2WqNmDDp0f9VVipjuj8uedTNPLNNhmhUWVnlwGG0HFVnvo4PcQ86WcF9Orxsszj2bJetcq+3XuirbBtbFT+3Jt+ccvTzu8QoOBooJrL02ISWO9Ei4R9AZvDP4d1z1qtVrUu3Q2lUNMXnNoij1kEcehtPsKoF+mIM9vGxccf0kklN0G1c+PaHyHb0kWyKjEk3Z342qZGrlfnWZIgHCj/DBOJPzSk7gSAJGBuFVLCfNihtsM8jRkco9ldU2LRqabICkpxw/0n8hvzjyaes7d/ijq2gCuMnba4Rl9K3s7PJYUDIhz5AWBaZZNk7kUwtX/y/wSLqM/DgW9NdVuLb6N2z5JzGva/I4Q2VSjXPsKUKRsJXnB8ZdO6qptDQ/EeSTHaQrfTrLBcY+oreFGa572ncxwdCHCr3XCjfLFbz33Gzzzb3KqdwTDpImVowQgrHlK/WBOPWpRnY9NvUVXhxOyTm+pNxzpxg/026ZpbaO113Zw7opWyLUBXrBvJA8s7Zp2Ob1qBMbQWx8M2JdOAJh26RU+8HCwFJuDH/XAwORDQ10eEw56o0pJnQeoMpQUCwl5jeUmjjHt8Uq1rwAFylhd26ShNSHTy7QMkdeKjMoN8GrHdCbfHFFLJKadizaNdVVZeFW2I+qtKxkMD0fB5idOxioWHj2dG1n3JxYttFWn1Oh4HDwugvqOGRcxhKogU3WicBtPl7o6MqOdnLuiFXBt3HResC8C/YiHy2irDxBsXTDAbfWiqpnjluKbNviDIJt9EQKdqwcGXuo98dDgzHoR/TvU/pempHg1rzH7bl4T6HwSXWddsn0FKJBInHZEzTmZZa5Jb/7PNfMULs1nKJbc83n6c2JuNq9heJ7PNHMs+t+9telINcs1z6q2c1cW2a6NXMd0SY57bu/kvDci0w51/Au/ztIf97VzqXr3R455Wr25oct75j3euYyZmzfDvmLuWqg6jb1rt6B2TGNZ9zWiVjdQVRGQPGRFSoGStLTVQdHCw30kD3TrjvklEp58bY1KStu5K43s18bNlrMdHldl1gNzB7tUyXHsdbZnbfK8+yPPdaI6bFSipDipQ1qV9hVgQ+UK8gSuQhFPX07bppVAL9TzMqatTTKRKXCvPW73lbo6KEZoiq71cyw6DYudr+bJ+m9SIqXr3JVIpmsDeCJ+t0mm304rxVJ3AsMVCcHAvrXJ8zolxbRDW8cJEKsxFFNSh7Qq7SvApnYX5IXHQ8tABnoTkwzZiL1BXfEI7SSvoZKRvA1vReZ7ZfKk38ON23SES6diM5TpQDac5gDPdNtr3Siazl2JpL02tNngntu5adcZ8HhTgVKwpGBgv9rk+R2x8pRwB9se8uDR2x/kXJek9nhV2leAzW09yOOSqkajIZ6uVfWJaljP27IMtVJrDmm2sESwd0hXvY4zJVGoZCT9DBkh/NnDDUvwpHDpVMw4f/aAOq1zoqMJWYsZDcvRo0AvXeeuNFJeGxrouud2ySu90WhrwYCO4HrDz2i2TfFf7hB3QB0al7eWIKpzVkhl9hVAB/EEWhleA/TQsrP2+14D4dCysx+0v/bu3BUN18buBTr6FNHwfmIaC8NYva55m5n1+0KdkgwzT+eZCnQuCHfOiu3MUoF9BdDggP8jbhYAgBuIS+9Q0gMA1YQgDwAAAKCCSte7FgAAAACSIcgDAAAAqCAEeQAAAAAVhCAPAAAAoIIQ5AEAAABUEII8AAAAgApCkAcAAABQQQjyAAAAACqH6P9Ru0M41Jw+AAAAAABJRU5ErkJggg==)
4.5 Model Training with Transfer Learning
The VGG16 model was selected for use in this study, due to its proven ability to work well in transfer learning. The VGG16 architecture (shown in Fig. 8) is simple when compared to other state-of-the-art CNNs, such as ResNet [22] and Inception [23]. This enabled us to focus on evaluating the overarching SNN architecture instead of dealing with various aspects of the network. The input layer of the VGG16 model was replaced with a new input layer suitable for taking the images in the bracket installation dataset. The earlier layers of the VGG16 model were then set to remain fixed throughout training (i.e., a learning rate of 0), but the last block of convolutional layers was set to be trainable. This would allow specific high-level features of the brackets and adhesive to be learned. Finally, a randomly initialised dense layer of 128 units was added to the end of the VGG16 model to create the feature vector, and the same output layer in Section III.D was added. The dense layer after the VGG16 model had a dropout of 50%, and the model was optimised using Adam with a learning rate of 0.0001 and a binary cross-entropy loss function. The model was trained over 25 epochs with a batch size of 16. For the SNN, each of the twin CNNs were replaced with the VGG16 model and the input layer of each modified as above. The outputs of each VGG16 model were then passed to separate dense layers of 1024 units with 30% dropout to create the feature vectors. The remainder of the model then stayed the same as for the base SNN. The model was optimised using the same optimiser and training options as before.
4.6 SNN Input Pair Consideration
The two images selected at each training step for an SNN are typically randomly sampled from the available classes. In this study, the image pairs could be one of: (correctly installed, correctly installed), (correctly installed, incorrectly installed), (incorrectly installed, correctly installed), or (incorrectly installed, incorrectly installed). Initially, this default method of selecting the images was used.
However, following on from our experimental results, the method of passing in the data was revisited. As discussed previously, the incorrectly installed bracket case had several sub-cases where each sub-case could differ significantly from the others. It was therefore hypothesized that attempting to label images from different incorrect sub-cases as being similar would cause optimisation problems for the SNN. An additional test was therefore conducted, looking to investigate the performance impact of the input image pair choice. For this, one of the input images was always presented as a reference image of a correctly installed bracket, and always to the same input of the network. The other input image could be of either a correctly installed bracket or any case of incorrectly installed bracket. This method of passing the input images is referenced by “custom input image pair”. The method was justified by the nature in which the SNN would be used in deployment. One image would be a reference image of a correctly installed bracket and the second would be of a newly installed bracket. The model would then verify whether the bracket had been installed correctly by comparing the images.
4.7 Edge Case Consideration
The edge case images were of particular interest in this study. It was expected that these would be where most misclassifications would be made. The edge cases are those where the bracket is placed just within or outside the tolerance of the marked boundary and with correct adhesion. All the edge case images in the original dataset were taken into a separate dataset, and then additional images were taken and added to create a suitably sized dataset specifically for the edge case images. The new edge case dataset had 50 images each of correctly and incorrectly installed brackets for training, resulting in 100 images. For validation, there were 40 images of each case, resulting in 80 images in total.
4.8 SNN Similarity Voting
Ensemble methods involve training multiple models on subsections of the training data and then voting on the same set of testing data. When verifying whether a new test image is the same as a known reference image, the similarity score will depend on the reference image used. This means that the resulting decision of whether the images are the same or not could change depending on which reference image is used and how similar it is to the test image provided. In deployment into aircraft assembly lines, verification will be done by providing a reference image to the SNN of a correctly installed bracket, and then passing it an image taken of the newly installed bracket. It is expected that following training of the model, if the test image and the similar reference image were passed through the SNN then these would be identified as being similar, and the newly installed bracket would be verified as having been installed correctly.
If however an incorrectly installed bracket image were used as the reference image, the similarity score generated may be more uncertain and lie around the threshold value of 0.5. The bracket may then be identified as being installed incorrectly. Furthermore, using the bagged model voting as a reference, the hypothesis here is that the dependency of the verification result on the reference image used would be reduced if the test image were compared to multiple reference images. Given the limited distribution of possible reference images, the test image should be similar to more reference images than not. Hence, when compared to enough reference images, it would be expected to see the majority of the outcomes being that the images are similar. A majority voting rule would then produce the correct result. Fig 9 shows the method used at testing to implement this voting scheme. Compared to the typical bagging method, multiple test input pairs are evaluated on a single model and a vote had on the final outcome, rather than evaluating multiple models on a single test input.
4.9 Testing and Evaluation
To evaluate how the changes made improved or hindered performance, suitable metrics were required to assess the models. Though the SNN performed image verification rather than classification, the typical classification performance metrics still applied. In particular, the accuracy, precision, and recall metrics were selected. The accuracy metric was selected as an initial indicator of model performance, which allowed for the training progress to be monitored and simple comparisons to be made between models. The accuracy by itself was not a sufficient indicator of performance however, and so the precision and recall metrics were also observed.
The chosen metrics enabled quantitative assessment of the model performances and also held important industrial implications. The precision of any model proposed would indicate what proportion of brackets identified as being incorrectly installed actually were incorrectly installed. From another perspective, it could be inferred from this value how often a bracket identified as being incorrectly installed was in fact correctly installed.
This metric, known as the False Discovery Rate (FDR), is calculated from the precision metric. Aerospace companies would be interested in these two metrics, as they would indicate how often a human operator would have to give a second opinion on the automated test. If the precision was too low, or conversely the FDR was too high, then the solution would be returning many false positives (FPs). It would then not be cost effective, as human operators would still be a common necessity in the procedure. Perhaps the most important metric in safety critical sector like Aerospace is the recall. Practically, this metric shows what proportion of incorrectly installed brackets are identified as such. From a different perspective, the number of incorrectly installed brackets falsely identified as being correctly installed can be evaluated by another metric, the False Negative Rate (FNR). A low recall or high FNR value would indicate many incorrectly installed brackets were passing through undetected, which could have severe consequences should aircraft be put through to operation with these onboard.
4.10 Results Verification on a Comparable Dataset
In order to give confidence in the results obtained using the custom dataset generated, the decision was made to repeat the experiments on a well-known baseline dataset. It could then be seen whether the same trends were present in the results. This was an important additional test for this study, where the data used was created specifically for the task. It would prove that the previous results observed on the hand-crafted dataset were not obtained through any bias or simplicity of the data. The Omniglot dataset (see Fig 10) was selected for this, which is a popular benchmark for SNNs [12].
Modifications to the dataset were necessary to formulate the problem in the same way as for the bracket verification task. To make the dataset sufficiently similar to that used in the bracket verification task, two of the alphabets were chosen that were similar in nature to each other. Namely, these were the Latin and Greek alphabets. This maintained the similarity that was present between the correctly and incorrectly installed brackets. Examples of similar and dissimilar images for this task are seen in Fig 10, where in this case similarity refers to whether the character is from the same alphabet or not. With the new dataset, the task was to perform alphabet verification at the alphabet level because each alphabet was analogous to the correctly or incorrectly installed bracket classes, and each character was analogous to the different sub-classes of brackets as described previously.