In this section, we will discuss the taxonomy related to anomaly detection which is confined to the scope of this work, reviewing the related literature to find the research gap thereby to elucidate the major motivation of this work. Techniques for Anomaly detection can be widely classified as global technique and local technique based on the size of the selected reference set, supervised technique and unsupervised technique based on the raw data islabeled or unlabeled and as full space technique or subspace technique based on attribute count [10–13] considered for defining anomalies as shown in Fig. 1.
Theoritically these techniques of anomaly detection can be classified as statistical method, classification based anomaly detection, nearest neighbor based detection, spectral based detection, clustering based detection etc. In this work, we focus on continuous data of high dimensionality from social media in a local sense hence unsupervised subspace anomaly detection method is selected. Supervised algorithm is applicable for ordinary classification of binary data and requires huge normal and abnormal data in order to learn mechanisms underlying for different data sets but it becomes worsen when as there is increase in dimensionality [14–15]. When the reference object scope differs, anomaly detection varies as either local or global. In order to achieve accuracy local technique is preferred over global technique for data sets of high dimensionality such as from social media [16].
Data deviation is covered or obscured in high dimensional spaces by irrelevant attributes leading to deterioration of full space anamoly detection, therey subspace anamoly detection is preferred in this work for outlier detection. Generally unsupervised anomaly detection are either density based or distance based. In models based on distance, any object at an distance greater than d from the host object is considered to be outlier but these approaches based on distance are not able to detect outliers effectively from datasets of different densities. Another approach is to measure local point density, among which local outlier factor approach is famous which computes density of local rechability ration with respect to nearest neighbor point [17–18].
Social media involves high dimensional spaces, hence models based on distance are not applicable whose drawbacks are overcome by replacing distance by angle for anomaly detection. The angle variation is measured in angle based anomaly detection where the data point of social media is considered with respect to other points. In a particular cluster, variance between normal objects will be a large values of angle, whereas the variance between the outliers will be typically small value of angles [19–20]. Social media has the curse of dimensionality but still angle based outlier detection is able to provide satisfactory performance for datasets of high dimensionality, but for full space even angle based anomaly detection shows deterioration [21]. Hence subspace based anomaly detection was opted for clustering method based on equi depth grid, where threat detection was done only in sparse cuboids of data rather than denser ones[22–23].
When ‘n’ dimensional space is considered, there occurs possibility of 2nattribute subsets termed as subspace, these subspaces increase in number exponentially as the dimensionality increases [24–25]. Hence to detect threat is such a large subset is exhautive hence these subspaces are not applicable over high dimensional dataset involved in social media. Selecting meaningful subspace based anomaly detection for machine learning applications is the major motivation for this work.