Morphology And Status Transition Monitoring For Mouse Embryonic Stem Cell Colonies In Virto By LSTM Networks With Progressive Training Using Fluorescence Microscopy Images

Purpose Embryonic stem (ES) cells represent as a cellular resource for basic biological studies and for their uses as medically relevant cells in in vitro studies. Fluorescence microscopy images taken during cell culture are frequently used to manually monitor time-series morphology changes and status transitions of ES cell (ESC) colonies, and to study dynamical pattern formation and heterogeneity distribution within ESC colonies, intrinsic fluctuation and cell-cell cooperativity. Therefore, tracking and furthermore predicting morphology changes and status transitions of ESC colonies is an effective method to monitor culture medium for maintaining ES cells in undifferentiated or early differentiated stage. Methods A P-LSTM (Progressive Long Short-Term Memory) structure is proposed to incorporate some new time-lapse images real-time taken from incubators for a new RNN (Recurrent Neural Networks) training. The P-LSTM can achieve adaptive long- and short- term memories to generate accurate predicted images. On the time-lapse images, entropy and bi-lateral filtering are used to extract the range of every colony to calculate colony morphology. Colony status transitions between consecutive images are calculated by mapping the calculated colony centers and ranges. Results Accuracies for the colony status transition, area and roundness for the 15 predicted (five-hour) future frames calculated from 1500-2500 colonies for respective frames show the effectiveness of the proposed method. Conclusion We proposed an efficient and automatic method to predict and monitor status transitions and morphology changes of mouse ESC colonies in culture using time-lapse fluorescence microscopy images. The results show our new machine learning method is effective in generating predicted fluorescence images. However, the prediction for the frames over 15 is usually not precise according our tests (not shown in the paper). The reason can be considered as no further new real images were learned. Besides, in some cases especially in status transitions, shapes of convex-hull colonies may be not similar to their corresponding the mouse ESC colonies on the real or predicted images. The method for calculating the colony morphology and centers should be improved to acutely evaluate morphology changes and status transitions.


Introduction
Embryonic stem cells (ESCs) are gaining attention as a cellular resource for basic biological studies and for their uses as medically relevant cells in in vitro studies. ESCs are known as undifferentiated and self-renewing stem cells. Upon induction of differentiation, they are going to differentiate into multiple cell types of distinct cell lineages. ESCs in a colony are not a group of homogenous cells, in which cells with different cell differentiation status co-exist and shift between the statuses. Although the nature of such heterogeneities of ESC population remains elusive, different types and levels of heterogeneity were revealed under different culture conditions [1,2]. ESC colonies that have an internal structure with respect to cell size, marker expression or biomechanical properties were usually calculated using microscopy images. For example, heterogeneities were represented by entropy [3] and differentiation was by the colony shape [4] on the microscopy images.
Monitoring of stem cell pluripotency and differentiation has been considered as important, where spectroscopy or electrochemical sensing [5], or fluorescence markers [6] were used. Time-lapse fluorescence microscopy images taken during culture were frequently used to track and thus monitor morphology changes (entropy, area, roundness, etc.) and status transitions (moving, merging, splitting, etc.) of an ESC colony or between colonies [7,8]. The tracking results assist study of timedevelopment of the cell-state transition of undifferentiated or early differentiated ES cells in colonylevels. For example, the morphology change of a mouse ESC colony could be used to calculate dynamical pattern formation and heterogeneity distribution at the colony, thus reveal the interaction of both stochastics and deterministic components, namely, intrinsic fluctuation and cell-cell cooperativity [9].
Recently, a machine learning method, RNN (Recurrent Neural Networks) learns sequential data of recurrent hidden (or internal) states at each time frame (such as consecutive time-lapse images) to predict future time-series states for efficiently monitoring, such as for tool wearing [10] or for early characterization of drug-induced cardiotoxicity [11]. Many RNN structures have been proposed to improve the predicted accuracy and speed or to accommodate the application. For example, LSTM (Long Short-Term Memory) networks [12] were proposed to solve vanishing and exploding gradient problems in capturing long-term dependencies in classical RNNs [13]. LSTM achieved successes in many fields and applications such as handwriting recognition [14] and medical care predictions [15].
In some applications, not the whole images, but simplified pixels representing probability vectors or position vectors (classified by CNNs (convolution neural networks)) were used as LSTM inputs to predict probability of future hiPS reprogramming [16] or cell motility [17].
The simplified pixels or other signals such as spectroscopy or electrochemical sensing, or fluorescence markers; however, cannot represent morphological features or patterns of cell colonies at the original fluorescence microscopy images. This study, therefore, processes whole time-lapse fluorescence images to predict status transitions and shape changes of ESC colonies. However, the morphology usually changes apparently and the status transits abruptly (finishes the status transition in a few consecutive frames of time-lapse images). We propose a progressive LSTM (P-LSTM)) structure that incorporates few new images and excludes the same number of the earliest images for a new repetition of training to deal with the colony short-term changes on transitions. A prediction can be made after any new training. However, too many repetitions cause retained long-term memory that brings image retention or disturbance in predicted images. Therefore, a P-LSTM network restarts after certain repetitions and multiple P-LSTM networks are implemented concurrently to keep there is a P-LSTM network with optimal times of repetitions to solve retained long-term-memory problems.
The P-LSTM thus includes adaptive long-and short-term memories to generate predicted images with accurate status transitions and morphology changes of respective ESC colonies. Cell colony tracking is then implemented to calculate the colony morphology changes and status transitions for monitoring future behavior of these colonies. The P-LSTM network structure is introduced in Section 2.2. Section 2.3 introduce the method of evaluating morphology of each ES colony on the predicted and their corresponding real time-lapse images. Section 2.4 introduces the method of tracking status transitions of ES colonies based on the convex-hull colonies on consecutive real and predicted images.
The calculated morphology and status transitions for ES colonies on real time-lapse images are used to evaluate the accuracy of the morphology changes and status transitions on the predicted images.

fluorescence microscopy images of mouse ES cell cultures
Time-lapse fluorescence microscopy images of mES cells were acquired by using a confocal microscope, CV1000-EH with built-in incubator (Yokogawa Electric Corp., Tokyo, Japan). The timelapse images with 20 min interval were taken by the equipped EMCCD camera (512512 pixels, 16  Meanwhile, the fewer sliding images S the better effect of LSTM short-term memory but take more computation. S is currently set as 3 to achieve high prediction accuracy and quick computation. 6

Concurrent progressive LSTM network
We implement two P-LSTM networks concurrently with the interval of 5 repetitions to keep there exist P-LSTM networks with further 5-9 times of repetitions after the first training. The prediction from any of these repetitions was tested as optimal to solve retained long-term-memory problems and possess sufficient long-term memory to obtain accurate predicted images. A P-LSTM network should be restart from the first training if the repetition of progressive training already reached nine.   Figure 2 shows the colony region extraction for a real fluorescence microscopy image ( Fig. 2(a)) taken from the incubator. For imaging live cells, the fluorescence image has relatively low signal-tonoise ratio. The image is converted as a gray-level image to calculate the entropy by the following equation [18].  their corresponding colonies on the real image ( Fig. 2(a)) except the one during status transiting (as the merging colony labeled as "1" in the right-bottom quadrant). Figure 3 shows the colony region extraction for an image predicted by the P-LSTM network ( Fig. 3(a)) corresponding to the real image ( Fig. 2(a)) it predicts. The images after the entropy and

Morphology calculation for colonies on predicted fluorescence microscopy image
the followed convex-hull computation are Fig. 3(b) and Fig. (c), respectively. Because of heterogeneous inherence of the predicted image ( Fig. 3(a)), the obtained convex-hull colonies are generally larger than the ones on the original predicted image (Fig. 3(c)). Therefore, we binarize the predicted image (Fig. 3(d)), then the bi-later filtering [20] to delete the noises and finally the convexhull algorithm to obtain the convex-hull colonies as shown in Fig. 3(e).
These convex-hull colonies are generally similar than to the ones on the original image ( Fig.   3(a)). However, some small colonies become too small by the bi-later filtering to be dealt as noises and deleted. For example, the colony at the right-top part is too small to be judged as noise, and the area in-between the opposite parts of a merging colony is deleted so that the merged colony became two ( Fig. 3(c)). Therefore, we use the colony centers to check if there are not labeled or multiply labeled colonies by the bi-lateral filtering but merged (as "1" colony in Fig. 3(c)) or labelled (as "2" and "3" colonies in Fig. 3(c)) by the entropy processing. These colonies are then used to replace the ones on the bi-later filtered images as shown in Fig. 3(f)).

Status transition tracking for mouse ESC colonies using convex-hull colonies on consecutive time-lapse images
Two mappings are implemented to track the status transition of every mouse ESC colony between two (previous and current) real or predicted consecutive images. The recorded colony centers at the previous frame image are mapped to the current frame image at the first mapping, and the colony centers at the current frame are mapped onto the previous frame at the second mapping.
The status transition of every colony is then determined as either of the following cases.
Moving: one-to-one mapping. A colony region on the current image is mapped by only one center of a colony on the previous image, and conversely this colony region on the previous image is also mapped by only one colony center of the current image. This case is defined as no status transition.
Moving in: one case of Moving where the colony center of the previous image is located near the image edge but the colony center of the current image was not. This case is defined as in a status transition.
Moving out: one case of Moving where the colony center of the current image is located near the image edge but the colony center of the previous image was not. This case is also defined as in a status transition.
Disappearance: none mapping. It is a colony region on the previous image that was mapped by no colony centers of the current image. This case is in a status transition.
Appearance: none mapped. It is a colony on the current image that was mapped by no colony centers of the previous image. This case is in a status transition.
Merging: Multiple to one mapping. Two or more centers of the previous image were mapped into the region of the same colony on the current image. This case is a status transition.

Splitting:
One mapped by multiple. Two or more centers of the current image were mapped into the region of the same colony on the previous image. This case is also a status transition.
Disappearance not followed by a Moving out, or Appearance not followed by a Moving in is considered as an error caused the area of a convex-hull colony is near the threshold for dealing with this colony as a noise. Such Disappearance or Appearance occurs when a small colony is judged as noise and deleted but not in its neighboring frames.

Real-time evaluation for P-LSTM structure
We developed the system on the PCs with Intel Core i9-

Method evaluation by comparing predicted accuracies colony status transition and morphology
We have implemented 110 times of real-time P-LSTM training and prediction for respective 5 sets of time-lapse images taken in mouse ESC culture. Totally, 550 times of P-LSTM training and prediction were implemented. Each prediction generated 15 frames of predicted images. Table I shows, at each predicted frame, the mean and standard deviation of the accuracies for the colony status transition, area and roundness (the ratio of the inscribed radius to circumradius) calculated the colonies on the 550 predicted images comparing to their corresponding the colonies on the real images.
A predicted frame includes totally 1500-2500 colonies from the 550 predictions. Meanwhile, a colony predicted as Moving (no status transition) is counted as correct only its corresponding colony at the corresponding frame calculated from real time-lapse images is also Moving. Meanwhile, a colony predicted as a status transition (Merging, Splitting, Appearance, Disappearance, Merging in, and Merging out) is counted as correct if either of its corresponding colony at the corresponding frame together with its previous or next three frames (currently as ±3) is the same type of status transition calculated from real time-lapse images. Figure 4 shows the accuracies for the status transition, area and roundness are the later predicted frames the lower, but all acceptable in the 15 frames (about 75% accuracy for status transition and 65% for area prediction, and over 80% for shape (roundness)). The result shows the P-LSTM structure helps monitoring mouse ESC colony status transition and shape changes in culture for five-hour future. Figure 5 shows an example with a prediction for colony Merging. Figure 5  selected frames of the real images, the labelled convex-hull colonies with their circumcircles on the real images, predicted images, and the labelled convex-hull colonies with their circumcircles on the predicted images, respectively. The convex-hull colonies shown in Fig. 6(b) are similar to the corresponding real fluorescence images ( Fig. 6(a)). The system monitored (predicted) a Moving in at the sixth frame as shown in Figure 6(d), that really happen at the eighth frame after the prediction as shown in Fig.6(b). Therefore, the predicted Moving in is two-frames ahead of the real images, thus is considered as correct. Figure 7 shows an example with an Appearance status transition. Figure 7(a) show three continuous frames of predicted images, and Figure 7(b) shows their labelled convex-hull colonies circumcircles calculated from the three respective predicted images. A convex-hull colony at the right-button part of the image shown on the second frame is recognized as a colony but not recognized on its neighboring (previous and next) frames because of small area. Therefore, it is a considered as an Appearance error caused by the image processing when determining the labelled colonies. Such Appearance and Disappearance errors did not appear in the convex-hull colonies processed from the real fluorescence images, but were about 0.25% of all convex-hull colonies processed from the predicted fluorescence images.

Conclusion and discussion
In this paper, we proposed an efficient method for monitoring status transitions and morphology changes of mouse ESC colonies in culture using time-lapse fluorescence images real-time taken from an incubator. To accurately learn apparent short-term changes of mouse ESC colonies on time-lapse fluorescence images, a new machine learning method, progressive LSTM structure was proposed that incorporates several non-simplified time-lapse fluorescence images to implement a prediction of new training. Meanwhile, the LSTM structure restarts to avoid retained long-term-memory problems.
Besides, we developed entropy and bi-lateral filtering methods to extract the range and calculate morphology of every cell colony on a predicted image, and to track colony transitions using the colony centers on consecutive predicted images. The tested accuracy shows the prediction for (15 frames) five-hour future are reliable by comparing the original predicted and real fluorescence images, and cell colony morphology and status transition calculated based on extracted convex-hull cell colonies from the predicted and real fluorescence images. 13 The results show our new machine learning method is effective in generating predicted fluorescence images. However, the prediction for the frames over 15 is usually not precise according our tests (not shown in the paper). The reason can be considered as no further new real images were learned. Besides, in some cases especially in status transitions, shapes of convex-hull colonies may be not similar to their corresponding the mouse ESC colonies on the real or predicted images. The method for calculating the colony morphology and centers should be improved to acutely evaluate morphology changes and status transitions.

Declaration of Competing Interest
The authors declare no competing financial or personal relationships that could be viewed as influencing the work reported in this paper.    Table 1 prediction accuracy vs 15 predicted frames   Predicted  frame  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15 Roundness