With phenotypic heterogeneity in whole cell populations widely recognised, the demand for quantitative and temporal analysis approaches to characterise single cell morphology and dynamics has increased. We present CellPhe, a pattern recognition toolkit for the characterisation of cellular phenotypes within time-lapse videos. To maximise data quality for downstream analysis, our toolkit includes automated recognition and removal of erroneous cell boundaries induced by inaccurate tracking and segmentation. We provide an extensive list of features extracted from individual cell time series, with custom feature selection to identify variables that provide greatest discrimination for the analysis in question. We demonstrate the use of ensemble classification for accurate prediction of cellular phenotype and clustering algorithms for the characterisation of heterogeneous subsets. We validate and prove adaptability using different cell types and experimental conditions. Our methods could be extended to other imaging modalities, such as fluorescence, and would be suitable for all time-lapse studies including clinical applications.