A survey on artificial intelligence techniques for security event correlation: models, challenges, and opportunities

Information systems need to process a large amount of event monitoring data. The process of finding the relationships between events is called correlation, which creates a context between independent events and previously collected information in real time and normalizes it for subsequent processing. In cybersecurity, events can determine the steps of attackers and can be analyzed as part of a specific attack strategy. In this survey, we present the systematization of security event correlation models in terms of their representation in AI-based monitoring systems as: rule-based, semantic, graphical and machine learning based-models. We define the main directions of current research in the field of AI-based security event correlation and the methods used for the correlation of both single events and their sequences in attack scenarios. We also describe the prospects for the development of hybrid correlation models. In conclusion, we identify the existing problems in the field and possible ways to overcome them.


Introduction
Modern security tools based on the artificial intelligence (AI) monitor a large number of system events and must identify those that may pose a potential threat or indicate an attack. A security event is a detected occurrence of a system or process state that indicates a possible security threat, a failure of safeguards, or a previously unknown situation that may be relevant to security (ISO 2022;Barrett 2018;Force 2018 generated by security systems, such as intrusion detection system (IDS), notify that a specific event or series of security events have occurred (Johnson et al. 2016;Wood and Erlinger 2007). An essential element of situational awareness is the security event correlation (ISO 2015a; Vlahakis et al. 2018;Kent and Souppaya 2006), which makes it possible to identify the relationships between events. Security event correlation contributes to a better understanding of the attack development by steps, determination of the source and goal of attack. The main purpose of correlation is to combine disparate security events into a single sequence or pattern that reflects the scenario of system behavior (Shittu et al. 2015;Li et al. 2021a). In predictive analytics, security event correlation techniques are capable of analyzing both historical data and real-time events, and automatically detect changing thresholds (Husák et al. 2018). This allows detecting anomalous events and preventing cyberattacks on early stages. The primary purpose of alert correlation is to identify the most significant events in the security dataset. In this case, prioritization and filtering of events implement the selection of individual events from a large data set, ranking and aggregation of events (Dwivedi and Tripathi 2015;Nasir et al. 2020;Cinque et al. 2020). Also, security event correlation has applications in digital forensics when it is necessary to investigate the source of an attack and trigger events.
There are some related surveys devoted to security event correlation approaches (Salah et al. 2013;Mirheidari et al. 2013;Yu Beng et al. 2014). Often, security event correlation techniques match the steps of an attacker as a multi-step or target attack for detection and prediction (Ramaki et al. 2018;Husák et al. 2018;Kovačević et al. 2020;Albasheer et al. 2022). In our last review (Kotenko et al. 2022), we consider a number of publications on the correlation of security events, including in the field of attack detection.
Most researchers consider approaches to event correlation in terms of implemented methods. As a rule, they distinguish three main groups of event correlation methods: similarity-based, sequence-based and case-based or knowledge-based (Salah et al. 2013;Ramaki et al. 2018). Similarity-based methods analyze the event proximity based on the calculation of a certain similarity measure of event attributes or fields. Sequence-based methods correlate events based on their causal relationship. There are also statistics-base correlation methods (Mirheidari et al. 2013;Albasheer et al. 2022), which can be both a subcategory of methods based on similarity and causal methods. Case-or knowledge-based correlation methods rely on a knowledge base system used to represent well-defined scenarios of events. Sometimes researchers separate data mining as a category of alert correlation techniques (Yu Beng et al. 2014;Husák et al.;Kovačević et al. 2020). Such methods include a stage of learning from historical events, which creates an intelligent model for further predictions and searches for event patterns.
In our previous work (Kotenko et al. 2022), we present an extended classification of security event correlation approaches based on correlation methods, knowledge extraction, detection type, number of sources, level of analysis, and architecture. We divide correlation methods into three main classes: similarity-based, step-based, and mixed. According to the event knowledge extraction, we separate manual, supervised and unsupervised methods. The type of detection depends on whether the approach examines intrusion detection or anomaly detection. The event data source can be either single or multiple. Depending on the stage, security information can be processed at the raw data, events, and report levels. Architecture of the security event correlation system can be centralized, distributed or hierarchical. In this review, we focus on correlation methods using artificial intelligence. We introduce as a new classification criterion for the knowledge representation in intelligent correlation approaches.
The process of representing cause-and-effect relationships of input data in artificial intelligence is closely related to knowledge representation models. As canonical models of knowledge representation, researchers consider rules, logical representation, semantic networks, and frames (Stephan et al. 2007;Tanwar et al. 2010). Knowledge representation in cybersecurity problems can be one of the following types: neural network training weights, rules derived from fuzzy logic, conditional probabilities of Markov models, events from monitoring logs, decision trees, or signature rules (Hamed et al. 2018;Sarker et al. 2021). Models for representing knowledge about security events are most often described in the literature in the form of ontologies (Sikos 2021) and attack graphs (Zeng et al. 2019;Lallie et al. 2020).
In this survey, we suggest and analyze the taxonomy of security event correlation models based on the ways of event knowledge representation in AI-based monitoring systems. To our best knowledge, our paper is one of the first surveys that focuses on the AI-based security event correlation according to the ways of knowledge representation and usage. Our review contains many recently proposed approaches that were not included in the papers, that have been published in the scientific literature in the last years. At the same time, we include a large number of new scientific publications of recent years that have not been reviewed in our previous work. In our survey, the methods from related fields of event detection are mentioned to demonstrate the existing methods for intelligent analysis of event sequences. As a result, we also outline the prospects of developing combined correlation models.
The paper is organized as follows. Section 2 enumerates and explains the terms and notation used in the paper. Section 3 considers AI-based security event correlation models such as rule-based, semantic, graphical, shallow and deep learning, and hybrid. We analyze how artificial intelligence methods are applied for each category and provide examples of using intelligent event correlation models. Section 4 contains a summary and discussion of the models under consideration. Section 5 offers some promising ways to create combined event correlation models. Finally, Sect. 6 discusses a number of open challenges and considers perspectives for future research that could inspire researchers and developers in this field. Section 7 concludes the paper.

Background and notations for security event correlation
First, we define a few important notations for better reading and understanding of the review. Table 1 contains a description of common abbreviations used in the paper. A number of specific abbreviations (names of correlation methods or techniques) are marked with an asterisk (*). We also give explanations of abbreviations directly in the text of the paper. Table 2 contains a description of the main notations and symbols used in the paper.
Let us determine the definitions of event, event type, security event and event correlation, as they are important terms for the area of event correlation.
Definition 1 Event is any observable occurrence in a system or network (Cichonski et al. 2012;Stouffer et al. 2017;Force 2018).
Examples of events can be a router access control lists updating, a user sending email, a server receiving a request, or a firewall policy changing. System activity is recorded as events based on predefined configuration policies, including security policies. Each event contains a number of characteristics that define a particular state. The state change information typically includes a timestamp and a topological label identifying the location of the occurrence. Also, an event can have a unique identifier and point to a specific user.
Definition 2 Event type is a specification for a set of events that have a similar purpose and the same structure.
For example, there are five event types that can be logged in Microsoft Windows: error, warning, information, success audit and failure audit (Microsoft 2021). Error is an event indicating a serious problem, such as loss of data or functionality. Warning is an event that is not necessarily serious, but may indicate a possible future problem. Information is an event that notifies about the successful operation of an application or service. Successful audit is an event about a verified successful attempt to access the security system. Failure audit is an event about a verified failed attempt to access the security system. Definition 3 Security event is "an identified occurrence of a system, service or network state indicating a possible breach of information security policy or failure of safeguards, or a previously unknown situation that may be security relevant". This definition is given  (Barrett 2018), the cybersecurity event is "a cybersecurity change that may have an impact on organizational operations".
Thus, we can say that a security event describes a change in the security state of a system or a violation of a security policy that has potential security implications for the system or its environment. This information includes device performance data, software and hardware failures, unsuccessful login attempts, network transfers, evidence of malicious code, a known attack or suspicious activity. Examples of changes in system security states include changes in discretionary access controls, changes in the security level of the subject, and changes in the user password. Examples of security policy violations are increased login attempts or attempts to violate mandatory access control (CNSS 2022). As a rule, security information and event management (SIEM) systems monitor and correlate security events in real time, as well as process and store data, analyze and generate reports.

Definition 4
Alert is a notification that a particular security event has occurred, containing information about the specifics of the occurrence. It is notification regarding current vulnerabilities, exploits, and other security issues (Johnson et al. 2016).
Alerts are usually generated by systems such as IDS, antiviruses, firewalls, and others. For example, intrusion detection system alerts usually contain information about detected unusual activity, as well as the specifics of occurrence (Wood and Erlinger 2007). The fields that are usually displayed in alerts include a brief security event description, the affected platforms (application, operating system, etc.), the expected impact (download failure, data exfiltration, etc.), and metadata (creation date and time, duration, etc.). Multiple alerts can contain information about a single event, or vice versa, one alert includes information about a series of events.
Definition 5 Incident is a single or a series of unwanted or unexpected security events that have a significant probability of compromising confidentiality, integrity, or availability (CIA) of a system or the information in a way that could negative impact core business processes and threatening information security (Scarfone et al. 2008;ISO 2022).
Sometimes an incident refers to any abnormal or unexpected event or set of events at any time during the life cycle of the system (ISO 2015b). Examples of negative impacts from incidents include system crashes, unauthorized use of system privileges, unauthorized access to sensitive data, and the execution of data-destroying malware (Cichonski et al. 2012). A cybersecurity incident have an impact on the organization, prompting the need for response and recovery (Barrett 2018).
We summarize briefly: events are identified changes in the environment, alerts are notifications that certain security events have occurred, and incidents are special secirity events that negatively affect the CIA and have an impact on important security processes. An event does not always have to cause an incident. An event may be irregular or unexpected but does not seriously impact a business. Security events are analyzed to determine if they are classified as security incidents (Fig. 1).

3
Definition 6 Multi-step attack is the set of interrelation intrusion activities of one or more attackers who pursue a specific goal of intrusion (Navarro et al. 2018;Zhang et al. 2019;Liu 2020).
If an attacker follows a step-by-step method in their attempts to attack a system, then we are talking about a multi-step or multi-stage attack. The steps are not isolated, but are interconnected by some logical relationships. The attacker gains knowledge about the target system at each stage to prepare for the next stage of the attack (Katipally et al. 2011). For example, the attacker first tries to identify potential vulnerabilities by scanning the target network, then hacks into vulnerable hosts, then installs malware, and eventually initiates a DDoS attack on the target server.
Multi-step attack researchers are also exploring different life cycles to describe Advanced Persistent Threats (APTs). According to NIST (Ross et al. 2011), an APT is an adversary that possesses sophisticated levels of expertise and significant resources, which allow it to create opportunities to achieve its objectives by using multiple attack vectors. Exploring the paths of various APTs, researchers can distinguish a multitude (3 and more) different stages in the life cycle (Quintero-Bonilla and Martín del Rey 2020). For example, Alshamrani et al. (2019) describe APT steps as reconnaissance, establishing foothold, lateral movement, data exfiltration and post-exfiltration. It is not necessary that these stages are found in every APT attack. We note that in the review we consider only those publications using the term APT attack that relate to the strategy and detection of stages of this type of attacks.
Definition 7 Event correlation is the creation of context between independent events (ISO 2015a; Vlahakis et al. 2018). NIST (Kent and Souppaya 2006) defines event correlation as the search for relationships between two or more log entries.
Event correlation is also defined as the process of consolidating events to improve the quality of their information while reducing the number of events (Limmer and Dressler 2008;Dwivedi and Tripathi 2015). The main purpose of correlation is to combine disparate security events into a single sequence or pattern that reflects the scenario of system behavior. Security event correlation can also be seen as one way to incident detection (Li et al. 2021a). Security event correlation uses to multi-step attack detection by finding multiple network events and activities with similar properties (Shittu et al. 2015). Similarly, the alert correlation is the definition of the individual alert relationships. The result is usually meta-alerts (Cinque et al. 2020). Meta alerts are objects that consist of all the common attributes of the correlated alerts, a list of data sources, and link to the original alerts (Kushwah et al. 2019;Nasir et al. 2020).
As mentioned earlier, there are three main categories of correlation methods: similaritybased, step-based, and mixed. Mixed correlation methods use a combination of different correlation algorithms without the obvious predominance of one over the other. Figure 2 shows the interaction of technologies that support the security event. Gray lines indicate the processes of collecting and monitoring events, orange -processing and analysis. The SIEM system monitors events through agents or sensors that have a centralized or distributed structure. Sources of security events can be system logs, network, IDS, antivirus, etc. Further, the SIEM system performs event correlation, vulnerability assessment, generation of countermeasures, reports, etc.
Similarity-based correlation methods compare multiple events based on their attribute similarity. A measure of similarity between attributes can be calculated using Euclidean, Mahalanobis or Manhattan distance functions, correlation coefficients and other mathematical tools. In cybersecurity, the basic principle of this correlation type is that a group of similar events can correspond to the same type of attack.
Step-based correlation methods create chains of events, reconstruct a user's actions, and analyze connections between several events. These types of approaches can both match security events based on specific sequence signatures, and define event chains based on their statistical relationships without predetermined knowledge. In cybersecurity, step-based correlation methods often use attack scenarios and vulnerability knowledge bases as sources of knowledge.
Step-based correlation methods, in turn, we can divide into causal-based methods and data mining methods. Cause-based correlation methods analyze the causal structure of events and obtain a sequence where previous steps determine subsequent ones. Data mining correlation methods search for patterns in event datasets using statistical analysis. Note that in this study, we distinguish between the terms "data mining" and "machine learning" not as equivalent. Data mining is a broader field of artificial intelligence research that includes mechanisms such as sequence mining, time series analysis, Bayesian networks, classification, regression analysis and others. So machine learning is one of the areas of data mining. In general, the task of event correlation can be defined in the following three stages: (1) calculation of pairwise similarity of events; (2) compilation of sequences of events as steps of a  Figure 3 shows these stages for security events. Independent events describe system behavior, including anomalies. Anomalous behavior also can include attacking actions. The first two stages determine the relationship of events and compose their possible sequences. The third stage classifies or predicts event sequences using the knowledge base. So from a security point of view, certain event sequences may correspond to normal system behavior, anomalous behavior, or a specific type of attack. These categories, in turn, determine the target system state in the security assessment.
Let the set of events, for example in log or capture file, is denoted as: where e is a single event, N is the number of events in the record, n = 1...N.
Traditionally, a set of events is viewed as sequences ordered in time: Otherwise, for any event e ∈ E , the subset of previous events e pre and the subset of subsequent events e sub are defined as: The set of events attributes (or features) is denoted as: where f is a single feature name, K is a number of features, k = 1...K. where L is the number of possible k feature values, k = 1...K, l = 1...L.
The event feature is defined as a mapping: As a result, each event e ∈ E can be represented as a feature vector: where the pair ( f i , d i ) corresponds to the i-th feature of the event ( f i ∈ F ) with the value d i ∈ D E k . Let us introduce general notation for the event correlation process. We denote the sequence of events as a pair: where E is the event set, and R E is the set of relationships between these events. So, the set R E can describe several causal conditions for event pairs.
Event correlation can be represented as a mapping: where the symbol ( → ) denotes a functional relationship between events e 1 ∈ E and e 2 ∈ E , r ∈ R E . In this case, the e 2 is called correlating event for the event e 1 if e 2 = r(e 1 ) . A pair of events can belong both to the same sequence of events ( e 1 , e 2 ∈ E ) and to different ones ( e 1 ∈ E 1 , e 2 ∈ E 2 ). A correspondence of events between two subsequences eS 1 = (E 1 , R 1 ) and eS 2 = (E 2 , R 2 ) is a mapping: From the point of view of cybersecurity, a sequence of certain events eS a can be considered as an attack scenario, where each event is an attacker step. In this case, the path of the attacker can be described as: where i=1...n, a 0 is a source of the attack, a n is a target of the attack, n is a length of the path.
Then matching the current sequence of eS c events with the sequence of attacking actions eS a as Corr ∶ eS c → eS a is important to multi-step attack detection. In this case, we are talking about the so-called intrusion or misuse detection. If we only know the sequence of events in normal operation eS n , the mapping is calculated as Corr ∶ eS c → eS n . This is called anomaly detection.

Artificial intelligence models for security event correlation: the state-of-the-art
We will take the canonical methods as the basis for knowledge representation models: rules, logical representation, semantic networks, and frames. Various semantic models can be used as a logical representation of knowledge about security events, and graph models can be used as a semantic networks representation. We also consider frames as a set of features and labels used in machine and deep learning models. Thus, we highlight the following AI models for event correlation: • Rule-based models (knowledge representation is a set of conditions to compare and aggregate security events); • Semantic models (knowledge representation is some language with specific syntax and semantics for describing security events); • Graphical models (knowledge representation is in the form of graphical networks); • Machine learning models (knowledge representation is a data structure that consists of a collection of security event features and their values). Figure 4 shows the classification of AI correlation models by event knowledge representation. Further, in this section, we will consider AI-based event correlation approaches, systematizing them by knowledge representation. We also note the existence of hybrid models that combine several forms of knowledge representation.
In this section, we consider approaches to the security event correlation grouped according to the proposed classification of AI-based models. In addition to event correlation, we will also include alert correlation in the review as many researchers consider events based on system notifications. The security event correlation approaches also contain methods for multistep attack detection, since they consist in finding relationships between events belonging to Fig. 4 Classification of AI correlation models by event knowledge representation the same attack scenario. The general principle of each method is given using the notations described in Sect. 2.

Rule-based models
Rule-based correlation models most often use knowledge about the causal relationships of security events, which are presented in the form of conditional sentences. In this case, event correlation approaches often use a knowledge base that contains rules for matching event attributes.
We distinguish two subcategories of rules: similarity rules and causal rules. Similarity rules describe the conditions for the similarity of events in terms of their attributes (features), they often use the threshold values of correlation coefficients and similarity measures sim: Figure 5 shows an example of calculating a binary correlation matrix of security events. First, for each pair of events e i and e j , their attribute-based similarity is determined and placed in cell (i, j) of the event similarity matrix. Further, each element of this matrix is compared with a similarity threshold. The elements of the event correlation matrix are boolean objects of the set R, where r(e i , e j ) ∈ [True, False], r ∈ R . A true value is assigned if the similarity of an event pair is above the threshold, a false value-if it is below. A threshold can be set as a fixed value of the correlation coefficient, or calculated from the average value of the feature correlation (Hostiadi et al. 2019). The security event similarity approach can also take into account event attribute weights assigned depending on the attack class (Sun et al. 2020). In addition, similarity can be defined both between event attributes of the same type, and between attributes of different types (Kotenko et al. 2018a(Kotenko et al. , 2020. Causal rules describe the conditions for the causal relationship of events, and they often use knowledge bases. These rules of event correlation can be described using models of prerequisites and consequences. Those models connect events in such a way that the consequences of early events coincide with the prerequisites of subsequent events. For event e the prerequisite set is denoted as Pr(e) and the consequence set as Cs(e). Then: Figure 6 contains an example of using such rules to build the event sequence. The knowledge base contains information about the prerequisites and consequences of events. The causal rules describe the conditions when the premise of one event is the consequence of another. As another example, let the event be described as a triple e = (time, type, host) . Then the conditions for linking events e 1 and e 2 can be as follows (Khosravi and Ladani 2020): 1. The prerequisites-consequences events occur on the same host: host 1 = host 2 , host 1 ∈ Cs(e 1 ) , host 2 ∈ Pr(e 2 ); 2. The prerequisite event and the consequence event belong to the same class of events: type 1 = type 2 , type 1 ∈ Cs(e 1 ) , type 2 ∈ Pr(e 2 ); 3. The prerequisite event precedes the consequence event: time 1 < time 2 , time 1 ∈ Cs(e 1 ) , time 2 ∈ Pr(e 2 ).
Knowledge of prerequisites and consequences can be also represented in the form of a codebook as a binary matrix, where "1" is the presence of a causal relationship between events, and "0" is the absence. In RACC (Mahdavi et al. 2020) (Real-time Alert Correlation based on Codebooks) codebooks correspond to attack scenarios that are mapped to

Fig. 6
Simple prerequisites and consequences correlation method incoming real-time alerts using matrix operations. TempoCode-IoT (Siddiqui and Boukerche 2021) uses a flow function representation based on unsupervised learning of a temporal codebook that captures key patterns in data across different time windows. Cluster centers from each time window data are stored as codewords.
The two subcategories of rules listed above can be combined into composite rules. The approach by Xl et al. (2021) uses data affinity propagation (AP) clustering, identifying similar alerts, and then a prerequisite and consequence method to recover the full attack process in IoT networks.
Traditionally, security event analysis is provided by default by expert rules, such as rules provided by SIEM systems like Open Source SIEM or Sigma, and programmed rules, for example in IDS Bro. AI methods allow the use of automatic extraction of correlation rules, and reducing the cost of manual specification. Thus, the correlation can be built without requiring predetermined knowledge, so that the system allows finding new correlations between events. In ABE (Lanoe et al. 2018) (Automaton Based Engine) the correlation rules are first represented by a correlation tree based on historical data analysis. In this tree, the nodes are logical nodes (operators AND and OR), and the leafs are the attacker's actions. The correlation tree is then transformed into an automaton that is able to recognize sequences of security events.
An example of rule extraction methods is association rule mining (ARM) based on the frequent rule mining (FRM) paradigm. ARM algorithms allow finding relationships between event attributes in the form of consequence rules. In this case, possible event sets ES = {E 1 , ..., E h } are considered, where h is the number of sets. A repeating ordered subset of events, or a vector of events, is denoted as E ′ . Let this vector have a set Correlation in this case is a mapping of two consecutive event vectors Corr ∶ E � i → E � j . To assess the correlation, two metrics are introduced -support and confidence. Denote the frequency of occurrence of E ′ as support count (E � ) =| ES � | , where || is the size of the set. The support measure means the proportion of event sets containing the E ′ subset. Then for vector Confidence measures how often events of E ′ j appear in the vector E ′ i : Figure 7 shows the calculation example with these metrics for five event sets. Frequent itemset mining (FIM) consists of defining support for each item in a set. A frequent event vector is the vector whose support and confidence are greater than or equal to the given thresholds min support and min conf : Examples of event rule mining are presented in a number of publications. SIRUS (Bénard et al. 2021) (Stable and Interpretable RUle Set) extracts interpretable rules from the random forest classifier by searching for frequent patterns in trees. Case-crossover APriori algorithm (Dhaou et al. 2021) provides association and causal rules explaining the occurrences of flooding events. The extraction of association rules can be based on temporal event characteristics, when event segmentation is performed using a sliding window. This analysis is based on the calculation of the frequency characteristics of attributes in accordance with the minimum support threshold (Xie et al. 2018).

Semantics models
Semantics models use languages with some specific rules, syntax and semantics to establish a relationship between input and output. In this case, events can be represented by sequences of characters that can be considered as "words" of a formal language, specified by some formal grammar. Denote the semantic model of the event as: where 1. The event set in a sequence is defined as a set E = (e 0 , E i , E f ) consisting of a sequence, including a start event e 0 , a set of intermediate state events E i , and a set of final events E f ; 2. Σ is the model input alphabet (non-empty symbol set); 3.
is the state transition function ∶ E × Σ → E.
In cybersecurity, the signature language allows describing system penetrations as sequences of actions that an attacker performs to compromise. Examples of such languages are STATL (Eckmann et al. 2002) (State Transition Analysis Technique Language) or ASTD (Tidjon et al. 2020) (Algebraic State Transition Diagram), which represent an event sequence in the form of state machines with actions and state variables. Other languages like SHEDEL (Meier et al. 2002) (Simple Hierarchical Event Description Language) and EDL (Jaeger et al. 2015) (Event Description Language) introduce a colored Petri net where nodes are system states, and transitions are current system events.  Figure 8 shows an example of a simple signature for a brute-force attack with access to "/ etc/passwd" in EDL (Jaeger et al. 2015). Start represents the initial state and suspicious access represents the final state. Failed and successful login attempts are intermediate states. The transition between these states is triggered by the occurrence of account login and file access events ( e 1 -e 4 ). Intra-event conditions compare properties of the current event and constant values. Inter-event conditions are conditions between the current event and the previously checked event. Mappings make the properties of the currently mapped event available for future checks.
Transitions between system states in attack scenarios can also be defined using a fuzzy declarative language. Thus, in a fuzzy state machine, the events are represented as fuzzy sets. Transitions from one state to another are described by a fuzzy transition function (Almseidin et al. 2019). Possible input values are determined by the types of attacks in the form of sets of events, and transitions from one state to another are described by fuzzy rules.
For semantic models of security event correlation, researchers often use natural language processing methods. So, by analogy with word embedding, correlation approaches include event embedding. Event embedding is a learned representation for raw event logs as text, where the events are words and words with the same meaning have a similar representation. There are research papers on analyzing log events using word embedding methods. Event embedding can be represented as a parametrized function f ∶ E → , where is the embedding event space. The embedding algorithm f learns the space to preserve the linguistic structure in the reference event "text" corpus D(E) based on the vocabulary V . The structure in D(E) is analyzed in terms of the relationship between events caused by their co-occurrence, according to the context definition. So this function maps each event e i to a vector v(e i ) ∈ ℝ d .
The most widely used for word embedding models are Word2Vec (Mikolov et al. 2013b, a) and GloVe (Pennington et al. 2014) both of which are based on unsupervised learning. LogEvent2vec (Wang et al. 2020) and LogUAD (Wang et al. 2022) use Word2Vec to generate word vectors and generates weighted log sequence feature vectors. Doc2Vec (Le and Mikolov 2014) is similar to the Word2vec algorithm, but instead of vectorizing words, it creates a vector embedding of text snippets. For approach by Liu et al. (2020), the training corpus consists of rows that are transformed from the raw events of security logs. Each row is treated as a paragraph in training the Doc2Vec model. LogTransfer (Chen et al. 2020b) represents each event log template using Glove, which takes into account both global word matching and local context information. Thus, the presentation of templates minimizes the impact of word order (i.e. syntax) while preserving semantic information. This helps to solve the problem that the log syntax of systems of various types is different, while the log semantics should be reserved.
The big disadvantage of Word2Vec and GloVe is the inability to encode unknown events and attributes. To solve this, the fastText model (Bojanowski et al. 2017), an extension of Word2Vec, splits words into several sub-words (or n-grams), and then transmits them to a neural network. So the improved LogEvent2Vec model by Ryciak et al. (2022) uses the fastText algorithm instead of Word2Vec. Ring et al. (2021) analyze four different approaches to presenting audit log data: one-hot-ecoding, Word2Vec, fastText, and GloVe. As a result, the study recommends using FastText, which showed the most significant latent space, has the ability to generalize previously unknown values.
Sentence embedding is similar to word embedding, but instead of words, they encode the whole sentence into a vector representation. Some of the most modern models for sentence embedding are ELMo (Peters et al. 2018) and BERT (Devlin et al. 2019). Such models create context-sensitive representations of a word instead of creating one value for each word. ELMo (embeddings from language model) considers the context in which words are used, rather than creating a dictionary of words with its vector form. One of the experiments by Zhan and Haddadi (2019) compares how differently the Word2Vec embedding and the ELMo representation perform in prediction network. ELMo representation is used to define each event embedding matrix. The results illustrate that the ELMo representation can better illustrate the contextual-temporal dynamics in event prediction. BERT (Bidirectional Encoder Representations from Transformers) is actively used not only in the processing of natural, but also synthetic languages, such as HTTP/HTTPS for attack detection in network traffic (Seyyar et al. 2022). BERT models also allow learning the context of event log keys in anomaly detection systems LAnoBERT (Lee et al. 2021) and LogBERT (Guo et al. 2021). Figure 9 illustrates the example of using embeddings to event log classification, as described by Ryciak et al. (2022). The input data segment in the example is six logs from the BGL dataset (Oliner and Stearley 2007). Parsing and analysis of input logs reveals templates of log lines and possible event sequences. Sequences are encoded into a matrix using template embeddings. Next, the average vector of the sequence is calculated and sent to the input of the classifier.
Another common formal language for describing events is the ontology language. We can represent a formal event ontology as: where E is event instances, C is the concept set (event types, etc.), F E is the property set, R E is the relation set, and D is the ontology domain. Ontology learning (OL) allows one to automatically or semi-automatically create ontologies by extracting terms for describing events. During event detection, the OL system attempts to extract complex relationships from the sequence of events by representing them as a natural language text. Examples of such linguistic tools for detecting events and matching their types with target events in the ontology are ZSEE (Huang et al. 2018) (Zero-Shot transfer learning for Event Extraction) and OntoED . We can say, that event ontology learning aims to get event ontology embedding with the correlation of events, based on the relations among event types.
Within the security framework, the ontology can be based on a hierarchy of concepts that determine the actions of attackers to implement attacks of various classes with varying degrees of detail. Intrusion ontology by Barzegar and Shajari (2018) is based on the Intrusion Detection Messaging Format (IDMEF). IDMEF is a data model for representing information exported by an IDS. An ontology for attack detection can be created using neural networks to learn text embedding as a latent representation of raw security event logs . The attack patterns for the ontology by Wang et al. (2018) are extracted from the normalized datasets using an Attribute-Oriented Induction-based Frequent-Item Mining algorithm (AOI-FIM). This algorithm includes event aggregation and Fig. 9 Simple example of using word embeddings to event log classification pattern search using data mining. This correlation approach is based on a machine learning paradigm, such as learning by example, which extracts generalized data and frequently occurring items.

Graphical models
Graphical models allow one to represent knowledge about events in the form of graphical networks. This network consists of nodes depicting objects and arcs describing those object's relationships. Graphical models (GM) allow one to represent the sequence of events in the form of directed graphs: G = (E, R E , w) , where an event set E is a set of vertices or nodes, relationships R E is a set of edges R E ⊂ E × E , and w is a function mapping edges to their weights, w ∶ E → ℝ . If the steps of the attacker are considered as events, then such a graph is called an attack graph.
Intelligent methods carry out automated construction of graphs based on event data. NoDoze (Hassan et al. 2019), OmegaLog (Hassan et al. 2020), UNICORN , HOLMES (Milajerdi et al. 2019), and WATSON (Zeng et al. 2021) analyze the semantic information of the logs and model event knowledge provenance graphs (KPG). Figure 10 shows the example of provenance graph from motivation by (Zeng et al. 2021). The attack scenario reflects the actions of a tester who wants to exfiltrate a secret document (secret.txt ), while imitating their normal behavior (using Github, etc.). The graph nodes are system objects: the rectangles are processes, the ovals are files, the diamonds are sockets, and the edges between nodes represent system calls. High-level behaviors are highlighted with colored boxes.
NoDoze (Hassan et al. 2019) is based on the understanding that the suspiciousness of each event on the provenance graph must be adjusted based on the suspiciousness of neighboring events on the graph. To assign anomaly scores to events, NoDoze creates an event frequency Fig. 10 Simple provenance graph example Source: Adapted from (Zeng et al. 2021) database and then aggregates the integral anomaly score across neighboring graph events. OmegaLog (Hassan et al. 2020) performs static analysis on log message strings (LMS) and determines their timing relationships, creating a set of all valid LMS control flow paths that may occur at run time. Once the attack is investigated, OmegaLog can use the LMS control flow paths to analyze the flow of events in a cause-and-effect manner.
UNICORN ) creates a block graph representing the entire history of system calls and builds a normal evolutionary model of system behavior to detect anomalous actions without knowledge of attacks. HOLMES (Milajerdi et al. 2019) compares tactics, methods and procedures that can be used to perform each stage of APT and creates a high-level graph that summarizes the actions of the attacker in real-time. Host-based intrusion detection system WATSON (Zeng et al. 2021) abstracts behaviors as embeddings (numeric vectors) based on contextual information and provides a vector representation of behavior semantics.
The uncertainty of event values in a graph can be expressed using hierarchical fuzzy situational networks (HFSN) (Kotenko et al. 2019) based on fuzzy inference and multi-agent implementation. This approach allows one to make decisions quickly in dynamic operating conditions.
Graph embedding, or graph representation learning, is a machine learning approach, capable to convert nodes (log entries) in the heterogeneous graph into low-dimension vectors. Approaches like CoRelatE (Huang et al. 2021) study correlations between entities, facts, and relationships from instances in sequences in the form of natural language, and then build knowledge graphs.
Probabilistic graphical model (PGM) is a model in which dependencies between random variables are represented as a graph. In addition to vertices and edges, Bayesian networks (BN) contain a quantitative assessment of relationships based on the conditional probability distributions of each node in the context of its parents. Probability of event e 1 , provided that event e 2 has occurred (posterior probability) as: where P(e 1 | e 2 ) is the probability of e 1 , provided that e 2 has occurred, P(e 1 ) and P(e 2 ) are the probabilities of e 1 and e 2 .
Then the probability of a certain event sequence is: where e pa(i) are parents of node i. This model can be used to calculate the probability of a certain security violation or an attacker's action (Kim et al. 2020). The Bayesian graph can reflect possible attack paths with the probability of transition between the attacker's steps. So at any moment, the attacker is in a node e i and moves on to the next node e j only if a certain vulnerability exists and can be exploited. The probability of this state transition is given by: where R ij is the state transition between events e i and e j , P V is the probability presented by the vulnerability j, and P V exp is the probability that such a vulnerability is exploitable if present. Algorithms for finding the paths of an attacker with the highest transition probability (6) P(e 2 | e 1 ) = P(e 1 | e 2 )P(e 2 ) P(e 1 ) (7) P(e 1 ...e n ) = n ∏ i=1 p(e i | e pa(i) ), 1 3 allow us to detect possible multi-step attacks, including APT attacks (Zimba et al. 2019) and zero-day attacks (Sun et al. 2018).
To reflect the impact of events on state variables, ECTBN (Bhattacharjya et al. 2020) (Event-driven Continuous-Time Bayesian Networks) can be used, in which, in addition to state variables, a history of events with a timestamp can affect the time and probability of transition of state variables. DOMCA (Sen et al. 2022) (Detection Of Multistage Coordinated Attacks) presents attack scenarios using the Dempster-Schafer Theory (DST) (Dempster 2008). DST is a generalization of traditional Bayesian probability that allows you to assign probabilities to sets of statements. This allows you to combine events from several sources without a priori knowledge, i.e. prior probability distributions about the states of the system.
The Markov model (MM) or chain is similar to the Bayesian network in terms of dependencies. The difference is that Bayesian networks are directional and acyclic, whereas Markov chain are undirected and can be circular. Approaches such as RTMA (Zhang et al. 2019b) (Real Time Mining Algorithm) and Third Eye (Hossain and Xie 2020) monitor how well the observed sequence of events corresponds to the established model of normal or malicious behavior. For multi-step attack scenario reconstruction, the RTMA (Zhang et al. 2019b) uses the concept of MM to facilitate alerts analysis. In this case, the correlation is carried out between event types and event attributes. The Markov state in Third Eye (Hossain and Xie 2020) denotes the state of the node under test (NUT) for the current operating IoT channel at the end of a timeslot. The state transition diagram of the proposed Markov model depicts the interaction between the primary user, the NUT, and the external node (an external terminal used by an attacker).
In a hidden Markov model (HMM), states are not observable, and we can only keep track of the variables (or symbols) that are affected by the state: For a given model, the observation probability of O is determined as: An important step in model training is to tune the model parameters to maximize the probability of observation. Figure 11 contains an illustration of a hidden Markov model over time. Each circle q i , i = 1..N , represents a state and O j , j = 1..T , represents observations. The highlighted path is the path of maximum probability. Parameter optimization can be achieved using the Baum-Welch algorithm (Welch 2003), as well as others. In (Khan and Abuhasel 2021), a hidden Markov model is explored to model serial data that is generated by IoT devices. To optimize the HMM parameters, a genetic algorithm is used that maps the search space to the genetic space. Each gene has a mean and variance for each state in the HMM.
Often, hidden Markov models are trained for each event sequence type. So for cybersecurity the HMM is created for each type of attack, for example using a training set of alerts generated by an IDS (Zegeye et al. 2018;Shawly et al. 2019) or based on common vulnerabilities and impacts (CVE) (Holgado et al. 2017;Ma et al. 2022). In the first case, the HMM parameters are extracted from the IDS alert training dataset, and can be adjusted online. In the second case, the possible observations (V) are based on different tags in the CVE repository and the severity of the alerts. For a multi-stage attack pattern k, the HMM includes the number of attack stages (Markov chain states), the number of associated observations, and the above probability matrices A and B. Subsequently, given observations associated with attack k, the HMM estimates the probability of being in each state of the model using Viterbi algorithm (Viterbi 1967).

Machine learning models
Machine learning models, such as shallow and deep, use frames as a representation of data. A frame is the AI data structure that includes a collection of attributes and values. It consists of a collection of slots and slot values of any type and size. This structure allows one to use large amounts of knowledge about events and analyze them using intelligent methods such as cluster analysis and machine learning.
If there is a knowledge base about all normal ( eS n ) and abnormal ( eS a ) events and their attributes, the problem of event detection and prediction of event sequences ( eS out ) can be reduced to the problem of multi-class classification in the form of some function  . The detection of anomalies in the event sequence is considered as an unsupervised or semi-supervised learning task eS out = (eS n ) . In both cases, the following main stages of solving the problem are distinguished: (1) identification of informative event features; (2) selection and training of a model (algorithm) capable of assigning the current event sequence to a certain class; (3) calculation of reliability and accuracy. As a rule, all three steps are repeated iteratively until a set of features and a model are found that meet the specified reliability and accuracy criteria. Shallow learning models are traditional machine learning techniques which can be used for alert correlation by mapping alert features, such as alert attribute values, event rates, etc. The number of alerts per day, the frequency of event occurrence, relational functions obtained from the social graph analysis are used as features for training. Chang and Wang (2016) profile malware data to extract attack scenarios using k nearest neighbor (k-NN), decision tree (DT) and support vector machine (SVM) algorithms. For Big Data monitoring in IoT system, the approach by Kotenko et al. (2018b) uses a structure involving principal component analysis (PCA), DT, SVM, k-NN, the Gaussian naïve Bayes (GNB) and the artificial neural network (ANN). The SMOTE-RF model (Li et al. 2021b) combines SMOTE and random forest (RF) algorithms to solve the problem of unbalanced classification and multi-classification in APT datasets. The SMOTE increases the number of minority samples through k nearest neighbor interpolation to improve the distribution of an imbalanced dataset. Then, multi-class learning is performed based on the RF.
Recurrent neural network (RNN) allows the analysis of sequential data such as time series or natural language texts. In the latter case, tools like SAM-Net (Lv et al. 2019) are common, which model relationships between events in a text corpus and represent them as scripts. Approaches such as Tiresias (Shen et al. 2018), DeepLog (Du et al. 2017), OC4Seq (Wang et al. 2021b) use RNN to predict future events based on previous observations to track anomaly behavior. Tiresias (Shen et al. 2018) calculates a probability distribution of possible events e pre = {e k+1 ...e n } given historical observed events e sub = {e 1 ...e k } , where k refers to the rollback window size, to predict the specific steps that will be taken by an adversary when performing an attack. Similarly, OC4Seq (Wang et al. 2021b) uses Gated Recurrent Units (GRU) for anomaly detection in event sequences. DeepLog (Du et al. 2017) uses Long Short-Term Memory (LSTM) and learns the correlations and patterns embedded in a sequence of log entries produced by normal system execution paths.
Convolutional neural network (CNN) is often used to process arrays of input data to look for specific patterns. For example, Chen et al. (2020a) encode the system call sequence into a two-dimensional fixed-length "picture", which is very suitable for CNN analysis. For this, N-Gram is used, the main idea of which is to cut the sliding window and get the sequence segments of length N. CNN is used in DeepCorr (Nasr et al. 2018) to study the stream correlation function adapted to the complex Tor network.
Autoencoder (AE) is a symmetric neural network and usually studies the features of events in an unsupervised manner. Abdullayeva (2021) uses a deep autoencoder model to automatically extract event features and encode APT attack vectors. Network Anomaly Detection (Min et al. 2021) with MemAE (Memory-augmented deep Auto-Encoder) solves the problem of over-generalization when normal sampling and attack sampling have common features. So APT signature templates have problems with low generalization performance. MemAE approximates the attack input reconstruction to a normal pattern by using a memory module.
Generative Adversarial Network (GAN) uses an adversarial mechanism to extract implicit relationships between events. The teacher network by Liu et al. (2019b) encodes event data into vectorized knowledge representations for feature learning. The student network processes raw texts for event detection and requires no extra toolkits, naturally eliminating the error propagation problem faced by pipeline approaches.
Model ensembles combine several models of learning, including both supervised and unsupervised, or shallow and deep. The approach by Oki et al. (2018) uses an ensemble of RF, logistic regression (LR) and AE models to detect and predict mobile network outages using multiple sets of user activity data. Ghafouri et al. (2018) describe an ensemble predictor that contains a deep neural network (DNN) and a linear regression model to detect anomalous cyber-physical sensor readings, where each sensor's measurement is predicted as a function of other sensors. Model combination studies often involve evaluating and determining the optimal combinations of AI models and their parameters to most effectively achieve results. Joloudari et al. (2020) use three AI-based classification models to early detect and classify APT attacks, including Bayesian network, C5.0 decision tree, and multilayer perceptron (MLP). Also, a combination of CNN and LSTM models is often used to detect and predict APT (Cheng et al. 2019;Do Xuan and Dao 2021). The advantage of the CNN-LSTM model in security event analysis is that such an architecture works well with tasks where the raw data has an explicit structure and has temporal properties.

Hybrid models
Some approaches and systems combine several event correlation methods, without the obvious predominance of one over the other. As a rule, systems that use a combination of similarity-based and casual-based correlation assume that the most similar events may be associated with the same attack scenario. Multi-step attack scenario reconstruction consists of three main aspects: (1) identifying related security events, (2) matching a subset to the appropriate scenario, and (3) ordering the sequence of events.
Correlation can be based on the similarity of events by parameters (for example, source and destination IP addresses and ports), and the scenario can be represented as a graph (Haas and Fischer 2019;Bajtoš et al. 2020). So SOAAPR (Heigl et al. 2021) (Streaming Outlier Analysis and Attack Pattern Recognition) matches and groups alerts in streaming mode, and the resulting clusters are converted into a graphical representation. The result is an attack signature that represents the attack scenario in terms of communication behavior, cause in data features, and time sequence of associated alerts.
Another way of correlation is the analysis of semantically similar events. MAAC (Wang et al. 2021a) (Multi-step Attack detection by Alert Correlation) uses Doc2vec to get the semantic representation of the alert description and calculates the cosine distance of the generated vector. MAAC matches the alerts and creates a graph first for alerts generated on the same host and then between hosts. The approach by Zhang et al. (2022a) uses the Word2Vec model to convert alerts to low-dimensional continuous values and match semantically similar alerts. Then the distance of the alert vector to each attack stage is converted into the probability of generating alerts at each attack stage, replacing the initial Baum-Welch value, to build the attack HMM. The Log2Vec method (Liu et al. 2019a) uses Log-Specific Word Embedding (LSWE) Word2Vec for word representation that enhances domain-specific semantic and relational information. LSWE uses two methods of word embedding: Lexical information Word Embedding (LWE) and Semantic Word Embedding (SWE). LWE predicts the target word so that its vector representation distance is as close as possible to its synonyms and as far as possible from its antonyms. SWE defines associative word relationships.
The extraction of security event attributes can also be performed using machine learning methods. MIF (Mao et al. 2021) (Multi-Information Fusion system) extracts anomalous alert streams using a CNN called Convolution and agent decision Tree network (CTnet) and then reconstructs the attack scenario using a graph-based fusion module. CTnet evaluates attack risks, which, together with information about attack nodes and attack time, are used to build a graph-based fusion module. The high-risk attack chain is retrieved using a Time-Weighted Depth-First Search (TW-DFS) algorithm. Weight information determines the path through nodes with a higher risk of attack, and time information helps to remove non-temporal correlations of attacks.
Recently, AI-based methods are often used to analyze event graphs. REGNN (Luo et al. 2020) (Real-time Event Graph Neural Network) is used to embed and predict real-time events by building dynamic heterogeneous graphs. This model creates event provenance graphs for user behavior and then uses recurrent neural networks to model the time dependence of past events and embed real-time events. In turn, CNN can be applied to graph convolutional network (GCN) (Nguyen and Grishman 2018), in which the convolution vector for each node is computed from nearest neighbor representation vectors. To detect and predict events, the GNN uses the current event vector in the graph. The GDN (Deng and Hooi 2021) (Graphical Deviation Network based approach) examines the graph of relationships between sensors and detects deviations from these patterns. This approach makes it possible to detect anomalies without preliminary data on the structure of graphs.

Summary of AI-based security event correlation models
The main part of the described security event correlation approaches has a common goal: to detect and predict security breaches that are step-by-step in nature -multi-step or targeted attacks or cause-and-effect violations of the system stability. In this section, we provide a summary of the review results. We classify the considered correlation approaches according to the following main criteria: application, AI-model used, and correlation method.
We can select the following main directions for application of AI methods to correlate security events: • Clustering of similar events to reduce the volume of processed information and classify security events to event detection (ED) (Liu et al. 2019b;Deng and Hooi 2021), event grouping (EG) (Hostiadi et al. 2019;Sun et al. 2020), and event pattern extraction (EPE) (Dhaou et al. 2021;Zeng et al. 2021). • Intrusion detection (ID), which deals with multi-stage and targeted attacks (Joloudari et al. 2020;Sen et al. 2022), or anomaly detection (AD) Wang et al. 2022) to notify the security administrator about misuses and deviations from normal behavior, respectively. • Intrusion prediction (IP) (Holgado et al. 2017;Oki et al. 2018) based on incoming events, which allows early detection of intruder targets.
We will also highlight three main areas of event correlation methods: • Similarity-based (SB) methods are based on the idea that similar events can have the same root cause or the same type, and the found links depend on the inherent similarity between attributes of each event (Kotenko et al. 2018a;Heigl et al. 2021). • Causal-based (CB) methods focus on the causal structure of a event sequence, when previous steps determine the ones that follow (Zegeye et al. 2018;Hossain and Xie 2020). • Data mining (DM) is a process of discovering significant patterns, especially in a large amount of data (Abdullayeva 2021;Zhang et al. 2022a).
Tables 3, 4, 5, 6 and 7 provide an overview of the considered security event correlation approaches. For each approach (the name of the approach, if any, is indicated), we also define the main application area (App.), the AI-model (Model) and their basis (Basis), the correlation method (Corr.), the type of data under study (Data type), and used dataset (Dataset). For rule-based correlation models, we can see a clear correspondence between rule type and correlation method. Semantic correlation models and machine learning models use data mining, while graphical models explore causal correlation. Event grouping and pattern searching are characteristic of rule-based and graphical models, while intrusion and anomaly detection and prediction are characteristic of graphical and machine learning models.
We can distinguish the following main types of data for security event correlation: event logs, IDS alerts, network traffic, vulnerability databases (CVE), and malware. We designate the datasets created by researchers and not made publicly available as Generated. The following open datasets are used as in the reviewed studies: • as system logs BlueGene/L Supercomputer System (BGL) and Thunderbird (Oliner and Stearley 2007 Researchers also use a number of metrics and formal approaches to assess the quality of correlation of security events. The performance of the correlation method is defined as the ability to correctly identify interrelated security events, as well as to make correct predictions about security conditions. As a formal approach, the authors compare the reconstructed attack scenarios with the description of the data set (Barzegar and Shajari 2018; Sun et al. 2018;Zhang et al. 2019b;Hassan et al. 2020). This evaluation method is often used for approaches using graph models.
The researchers also calculate indicators corresponding to the correctness of the AI model decision. Often the model makes a decision based on a binary classification: an event is part of an attack or not, an event correlates with another or not, etc. Here and below, we use the term "decision" to generalize the application of correlation. In this case, the "predicted decision" is the value adopted by the model in the experiment. The "real decision" is the value that corresponds to the actual condition or characteristic of the security event.
The AI model decision can correctly indicate the presence or absence of a certain condition or characteristic of a security event -true positive (TP) and true negative (TN), respectively. Also, the decision may incorrectly indicate the presence or absence of a certain condition or characteristic of a security event -a false positive (FP) and a false negative (FN), respectively.
The short description of the metrics used by researchers in the review is presented in Table 8. We should note that research authors may use different names for metrics that at the same time have the same context, for example precision (Ryciak et al. 2022;Deng and Hooi 2021) and soundness (Barzegar and Shajari 2018).
Metrics of accuracy, precision, recall, and F-measure are classical quality indicators for intelligent methods. They demonstrate how well attacks (Khosravi and Ladani 2020;Siddiqui and Boukerche 2021;Seyyar et al. 2022) or anomalies Chen et al. (2020b); Min et al. (2021) are detected, and event sequences are reconstructed (Dhaou et al. 2021;Zeng et al. 2021). Also, authors can use the term correlation ratio (Xl et al. 2021), the calculation method of which is similar to accuracy. In addition, metrics are calculated based on the confusion matrix, such as TNR and FNR (Joloudari et al. 2020). Metrics called error rate (Shawly et al. 2019) and false alarm rate (Xl et al. 2021) determine the proportion of false positives among all alerts or alarms.
The correlation quality is also calculated using the AUC ROC metric (Wang et al. 2020;Lee et al. 2021;Ryciak et al. 2022). It displays the ratio between the proportion of correctly predicted decisions (TPR) and the proportion of incorrectly predicted decisions (FPR) when the threshold of the decision rule is varied.  (Cheng et al. 2019) or MSE (Ghafouri et al. 2018). Such metrics show the difference between real and predicted event sequences.
If the goal of correlation is to reduce the number of incoming events, then reduction rate (Zhang et al. 2019b) or aggregation rate (Sun et al. 2020) metrics are additionally used. Also, a number of authors evaluate the performance of their approaches in terms of computational resources, such as time, processor load, and memory consumption (Lanoe et al. 2018;Milajerdi et al. 2019;Mahdavi et al. 2020).
Thus, the quality metrics used directly depend on the application of security event correlation methods. Improvement in the performance of the correlation method is achieved by increasing accuracy, precision, recall, F-measure, TNR and AUC ROC. At the same time, FPR, FNR, ER, MAE and MSE should be minimized.

Prospects for combining AI-based security event correlation models
Recent trends in security event correlation are leading to an increasing use of hybrid AImodels (Haas and Fischer 2019;Bajtoš et al. 2020;Deng and Hooi 2021). Such models allow researchers to use the advantages of various correlation methods and offset the disadvantages.
Similarity-based methods have the simplicity of implementation to determine the relationship between a pair of events. However, the difficulty consists in choosing the most efficient way to calculate the event connection (Hostiadi et al. 2019;Bajtoš et al. 2020). Simple matching of event attributes can give a lot of false positives. At the same time, complex functions can be too specific, which makes similarity-based methods less flexible and poorly adapted for detecting a wide range of events.
Causal-based correlation methods make it easy to interpret the results of correlation by the operator. Therefore, this category of methods is well suited for visualizing the sequence of events (Heigl et al. 2021;Wang et al. 2021a). The simplicity of implementing such models is reduced, and also requires more computing resources to process a large amount of data.
In turn, data mining correlation methods are easier to deal with. Data mining methods allow one to automatically extract event features for correlation (Nguyen and Grishman 2018;Liu et al. 2019a). The interpretability of the results, in turn, decreases. In addition, the performance of trained models strongly depends on the training dataset.
Based on the results of the review, we can describe the functional requirements for the most complete event correlation model as follows: • the model should be able to define sequences or clusters of related events; • the model should take into account the correlation between the features of events; • the model should take into account the semantic properties of events; • the model should be able to automatically process a large amount of event data; • the model should be able to visually interpret the results of event correlation. Figure 12 shows the proposed diagram of the application of the hybrid model for the correlation of security events. This model meets all the requirements by combining correlation methods with the corresponding functions. The hybrid model accepts a stream of security events as input. The output of the model is the reconstructed security event scenario (attack graph, anomalous sequence, etc.), as well as the type of scenario and the security events predicted according to the scenario. The semantic model transforms the input data into context-aware security event vectors. The rule-based model demonstrates the proximity of vectors of similar events in a clear way, and then separates them into clusters. Interpretability and visualization of the results of the correlation model is supported by building dependency graphs of security events. Automation of the process of searching for patterns of security events, their classification, and prediction is achieved through a machine learning model. At the same time, when implementing supervised learning, the hybrid security event correlation model is able to detect known multi-step attacks as scenarios. When implementing unsupervised learning, the model is able to detect anomalies in normal event scenarios. In this way, a wide application coverage is achieved in terms of intelligent models and correlation methods, as well as applications.
It should be noted that this hybrid model can be adjusted depending on the characteristics of the target system and non-functional requirements for this system, such as resource consumption, time costs, and others.

Challenges and opportunities
Below we will present several potential topics in the correlation of security events that seem to us promising areas of research based on AI-based methods and existing challenges.
• Hiding malicious patterns Compared to the tasks of analyzing natural language texts (Huang et al. 2018) and building knowledge bases based on event detection (Deng and Hooi 2021), the detection of multi-step attacks is complicated by the attacker's intent to hide his trail. In particular, even if attack alerts are detected, it is often difficult to define the entire attacker logic without additional knowledge. So, security event cor-

Fig. 12
Diagram of a hybrid correlation model relation methods can encounter missing data in probabilistic models. An attacker can also trick unsupervised anomaly detection methods by disguising some of the events as normal. A suitable solution is to study the attacker's behavior given the semantics and context of security events. Profiling of different types of attackers is also an important and promising area (Oki et al. 2018). • Explainability of event semantics This challenge follows directly from the previous one.
Although AI-based methods such as deep learning methods are quite effective at detecting multi-step attacks, many of them operate in a "black box" manner (Du et al. 2017).
In this case, it is more difficult for security operators to understand the semantics of the detected events. In addition, cyber-analysts are faced with a semantic gap between lowlevel audit events and high-level system behavior (Zeng et al. 2021). In this case, the use of interpreted intelligent algorithms will be useful additions, including in forensic problems. • Analysis of highly cardinal events In this case, we are talking about the high uniqueness of several significant categorical event features. Correlation methods based on ontologies and knowledge graphs cope well with the analysis of such features. However, as a rule, shallow and deep learning models use encoding of such functions, for example, one-hot encoding, which, in turn, reduces the efficiency of processing high cardinal values.The development of algorithms for learning ontologies and event databases are a good help in processing of such data (Huang et al. 2021). • Analysis of large and/or heterogeneous data In this case, we are faced with the need to process events from different sources in various formats and with different semantics. Especially in distributed and complex systems, it becomes necessary to analyze a very large amount of data, which requires large computing resources. This task is particularly difficult when detecting attacks affecting multiple sources. Here there is also the problem of choosing the architecture of the correlation system: distributed, centralized or hierarchical. In the case of a large amount of event data, a good solution is to support parallel computing and use big data processing tools (Kotenko et al. 2018b). Heterogeneous event processing may include event unification algorithms or the use of correlation methods without a clear dependence on the event format. • Event knowledge base support As knowledge about events, there can be both correlation rules and patterns of known multi-step attacks or patterns of normal behavior. In this direction, we can also add the development of event ontologies. Creating such knowledge bases manually with the involvement of experts is quite laborious. In this case, an important direction is the development of adaptive event correlation methods or online learning algorithms for timely updating of pattern databases and knowledge graphs ). • Few publicly available datasets to assess the correlation of cybersecurity events. Any learning methods largely depend on the availability of a reliable model-building dataset. Most researchers use private datasets that are not shared to reproduce experiments. For this reason, it is important not only to develop new event correlation mechanisms, but also to publish generated security event datasets with the appropriate use license and citation rules.
We hope that this survey and discussion will provide researchers in related fields with an understanding of the latest approaches to the correlation of security events, as well as interest in the development of intelligent event analysis methods applicable in cybersecurity.

Conclusion
This paper provided a review of the security event correlation literature over the last years. The review focuses on how AI-based techniques are applied to detect causal security issues, such as the attack scenario detection and prognosis. We presented the systematization of security event correlation models based on AI knowledge representation such as: rule-based models, semantic models, graphical models, machine learning-based models, and hybrid models. We provided comparison tables of the described AI-based correlation models by application, model basis, correlation method, and security event data used. One of the results of the paper is also a description of the prospects for the development of hybrid correlation models. We also highlighted the challenges that researchers face when developing security event correlation approaches to stimulate future research. In general, they relate to the complexity of defining the logic of a multi-step attack and the complexity of processing a large amount of heterogeneous data. Hence, there is a need to improve approaches to event correlation, both in terms of semantics and learning capabilities.