An Evaluation Framework Combining Real-Time Transmission Electron Microscopy and Integrated Machine Learning-Particle Filter Estimation Enables Detection and Quantitative Tracking of Nanoscale Defects During Plastic Deformation Processes

14 Observation of dynamic processes by transmission electron microscopy (TEM) is 15 an attractive technique to experimentally analyze materials’ nanoscale phenomena and 16 understand the microstructure-properties relationships in nanoscale. Even if spatial and 17 temporal resolutions of real-time TEM increase significantly, it is still difficult to say 18 that the researchers quantitatively evaluate the dynamic behavior of defects. Images in 19 TEM video are a two-dimensional projection of three-dimensional space phenomena, 20 thus missing information must be existed that makes image’s uniquely accurate 21 interpretation challenging. Therefore, even though they are still a clustering high- 22 dimensional data and can be compressed to two-dimensional, conventional statistical methods for analyzing images may not be powerful enough to track nanoscale behavior by removing various artifacts associated with experiment; and automated and unbiased processing tools for such big-data are becoming mission-critical to discover knowledge about unforeseen behavior. We have developed a method to quantitative image analysis 27 framework to resolve these problems, in which machine learning and particle filter 28 estimation are uniquely combined. The quantitative and automated measurement of the 29 dislocation velocity in an Fe-31Mn-3Al-3Si autunitic steel subjected to the tensile 30 deformation was performed to validate the framework, and an intermittent motion of the 31 dislocations was quantitatively analyzed. The framework is successfully classifying, 32 identifying and tracking nanoscale objects; these are not able to be accurately 33 implemented by the conventional mean-path based analysis.


Introduction 38
In recent years, researchers have been trying to implement machine learning (ML) 39 based approaches in a wide range of scientific fields, and it has attracted considerable 40 attention [1]. ML has demonstrated its capability to implement semantic segmentation, 41 which classifies objects in an image pixel by pixel, and has been applied to practical 42 applications for example, automated driving technology and the medical field 43 individual objects must be separately tracked throughout the TEM video with being 80 distinguished from others. Especially in the case that the objects repeat unexpected 81 behaviors such as sudden move-and-stop and irregular change of own shape, tracking 82 the objects becomes highly challenging. The unexpected behaviors are often caused by 83 atomic to nanoscale local environment, which is closely related the inhomogeneity of 84 material. Thus, developing a model to predict such behaviors for data analysis would be 85 nearly impossible. 86 In this study, we developed a ML-based framework for quantitative analysis of 87 nanoscale objects' dynamic behavior based on the information obtained by detecting the 88 objects in a video using machine learning and tracking the detected objects with particle 89 filters. We confirmed that if a video presents a single experiment, the number of data is 90 sufficient for machine learning to detect dislocations in that video. We then applied the 91 developed ML-based framework to a video in which the dislocation gliding under 92 applied external tensile stresses in a metal was observed using TEM. By detecting and 93 tracking dislocations in the TEM video singly and as a whole using the framework, we 94 were able to calculate the time history of dislocation velocity and quantitatively 95 analyzed its behavior. In particular, we employed the particle filter to the quantitative 96 analysis part of the framework. Thanks to the probabilistic prediction of the particle 97 filter, we successfully captured the unexpected behaviors of individual dislocations. 98

Results 99
When a metallic material is plastically deformed by applying the stress , a slip 100 deformation occurs along a specific crystal direction (slip direction) on certain crystal 101 planes (slip plane). Slip deformation is localized by the movement of dislocations, 102 indicated by the "" symbol, on the slip plane as schematically shown in Supplemental 103  Differentiating both sides by time , the strain rate of the crystal ̇ can be written by the 108 average migration velocity of the dislocations . 109

̇=
(1) 110 For the dislocation velocity measurement by TEM observation, Johnston et al. 111 reported one of the first successful cases that measured the average dislocation velocity 112 [25]. They measured the average velocity of the dislocations by dividing the 113 displacement of the dislocations by the time that the stress was applied. However, since 114 the actual dislocation motion is intermittent, a continuous velocity measurement 115 providing the chronological changes is necessary to understand the intrinsic dislocation 116 behavior. Therefore, the overreaching goal of the framework development is to assess 117 the traverse speed of nanoscale objects such as dislocations without compromising the 118 original data's temporal and spatial resolutions. In this study, we attempt to archive 119 the10 nm/s order temporal and spatial resolutions by applying a U-Net based ML and 120 particle filter integrated method to in situ TEM deformation videos. 121 The actual validation of the framework proposed in this study was implemented by the 122 following steps described in the rest of this section. The experimental data, TEM videos, 123 were taken during in-situ TEM deformation experiments, in which an high-manganese 124 austenitic steel (Fe-31Mn-3Al-3Si) was subjected to a forced displacement with a 125 tensile rate of 100 nm/s. as shown in Fig.1 (a). In Fig.1 (b), a group of dislocation lines 126 like arcs moved to the left. Since TEM images represent a 2D projection of a 3D object, 127 the real space geometry of dislocations in the crystalline grain needs to be retrieved to 128 evaluate the stress condition in the observed area. The crystal orientation of the material 129 in the movie is shown in Fig.1 (b). In this particular case, the dislocations observed in 130 the movie are moving on the ABC plane and the incident electron beam is transmitted in 131 the direction of CD ⃗⃗⃗⃗⃗ in Fig.1 (b). Table 1 summarizes the Schmid factors for the ABC 132 and ABD planes, which indicate the contribution fraction of the load stress to the 133 resolved shear force acting on the slip system. There are two advantages to use ML for this task. The first one is that the detection 165 process is efficient and objective. ML detects dislocations in every frame of the video 166 after learning from a training data which is a correct image set created by the operators. 167 The detection of dislocations in the video will be conducted on the same criteria as the 168 one of the correct image set. The second one is that ML is more robust than numerical 169 filtering. ML is able to detect dislocations without being misled by non-dislocation lines 170 in a TEM image. For these reasons, we thought it is the best to use a ML method to 171 frames as training data and 101 -170 frames as test data. Fig.2 (b) shows the output 179 from U-Net. We were able to obtain the same output as the correct image for the test 180 data. 181 In the last step, in order to track down the same dislocation in the video, we used a 182 particle filter, which is one of object tracking methods in videos. Other methods such as 183 optical flow are commonly used for the object tracking. Optical flow, however, cannot 184 track dislocations accurately. Optical flow cannot track the point which moves quickly 185 and it is difficult to specify the feature points in a line with shape changes, although 186 movements of dislocations may be unpredictable and the shapes of dislocations may 187 change. In this study, we thought that a particle filter approach [33][34] is more suitable 188 for tracking dislocations. Since particle filter tracks objects using probability 189 distributions, it can retake and keep tracking individual dislocations even if the exact 190 location of more than one dislocations was temporary lost due to a sudden and unforeseen 191 movement. Particle filter is a better fit for this case as the dislocations' shape change likely 192 occur and the movement of dislocations may be unpredictable. 193 For the use of particle filter, it is necessary to identify individual dislocations in each 194 of video frames. We adopted a method to identify dislocations based on the spatial 195 continuity of pixels belonging to the dislocations. tracking dislocations that meet these conditions. 206 In here, the results of successful tracking four targeted dislocations are shown. The 207 dislocations (i)-(iv) are shown in Fig.3 (a), and the tracking of dislocation (i) is shown in 208 Fig.3 (b). In Fig.3 (b), the blue dots represent the particles distributed on the field, the 209 red dots represent the center of gravity of the blue dots, and the green dots represent the 210 midpoints of the dislocations closest to the red dots, i.e., the coordinates of the 211 dislocations being tracked. We confirmed that the green point stayed on a single 212 dislocation across frames. 213 We will show the results of the dislocation velocities measured by the above tools. Schmid factor between that direction and [1 ̅ 2 ̅ 2] is 0.136, which is the largest. Then we 236 calculated the strain rate in the tensile direction to be 43.5 µ/s. The strain rate in the 237 tensile direction at the experimental conditions is 100 µ/s, which is a reasonable value 238 considering the wide range of dislocation density values. 239 In Fig.4, we can observe intermittent dislocation motion. The reason for this may 240 be that the dislocations are stationary due to localized crystal defects in the sample, 241 which inhibit their motion, and they move when they gain an energy to overcome the 242 obstacles and advance due to external stress. It is also possible that the elastic field from

Discussion 248
In this study, we developed a Framework to detect dislocations in videos captured 249 using TEM using U-Net and measure their migration velocity using particle filters by 250 taking their intermittent motion and shape changes into account. The dislocation 251 velocities were measured and confirmed to be theoretically valid, and their intermittent 252 motions could be quantitatively evaluated. 253 This method has possibility to be applied not only to dislocation videos like the one 254 used in this study, but also to videos of TEM in situ experiments (dynamic observation) 255 on other phenomena. For example, immediate applications would be dynamically 256 measure the velocity and analyze the shape changes of dislocations in various 257 dislocation reactions including but not limited to Orowan mechanism (particle 258 dispersion strengthening mechanism), grain boundary migration, and deformation 259 twinning behavior induced by external stimuli such as magnetic field, heat or stress 260 field. It is also possible to chase the velocity, motion and shape change of nanoparticles 261 during an oriented-attachment reaction where dynamics in particles, translational and 262 rotational accelerations, is critical to gain the mechanistic understanding (e.g., DOI:

Method 280
The configuration of the dislocation velocity measurement tool developed in this 281 study is shown in Fig.5. With the developed velocity measurement tool, is capable of 282 automatic measurement of the velocity of each dislocation in the TEM video. 283

Optical Flow 284
In experimental TEM videos, we cannot accurately measure the velocity of 285 dislocation movement because the FOV moves. Therefore, we create a static coordinate (2) 295 Considering that the motion of the object is small, the Taylor expansion of the right-296 hand side yields 297 ( + ∆ , + ∆ , + ∆ ) = ( , , ) + ∆ + ∆ + ∆ .

Identification of each dislocation 322
In the binarized image, the dislocation pixels are distinguished from the 323 background pixels, but the dislocations are not distinguished from each other. The 324 particle filter needs to identify the dislocations in a frame because it needs to set the 325 target to be tracked. Therefore, we developed a program to search around the 326 dislocation pixels and identify them as the same dislocation if they are continuous, as 327 shown in Fig.7. 328

Particle filter to track dislocation 329
Particle filter [33][34] is a method for estimating the position of an object by 330 distributing a large number of particles on the screen and using the prediction from 331 the previous state and the current observation information. The particle filter 332 approximates the probability distribution of the object to be tracked in the entire state 333 space by a large number of particles with state quantities and weights (likelihoods), 334 which enables robust tracking against noise and environmental variations. The 335 particle filter algorithm is shown below (see Fig.8).  3. Obtain information necessary for likelihood calculation for each particle. 340 4. Calculate the likelihood for each particle based on the particle information. 341 The likelihood is computed by the brightness of the pixel where the particle 342 is located, and the similarity between the image of the region around each 343 particle and the image of the region around the dislocation in the previous 344 frame. 345 5. Calculate the weighted average with the likelihood of each particle as a 346 weight. 347 6. Re-spread particles with a probability proportional to the high likelihood 348 of each particle. 349 7. Move to the next state, and repeat from procedure 2. 350 351 By performing the above processes in each frame, particles are able to track the 352 target object. When implementing a particle filter, it is important to design the 353 prediction (Procedure 2) and the likelihood (Procedure 4). appropriately based on 354 information such as the motion and shape of the target object, in order to track the 355 target object accurately. In this study, we used the information that dislocations 356 moved only in one direction for prediction. We also used the information that    513  514  515  516  517  518  519  520  521  522  523  524  525