The SMART approach to develop good cyber security metrics

doi:10.21203/rs.3.rs-3125926/v1

Download PDF

Research Article

The SMART approach to develop good cyber security metrics

https://doi.org/10.21203/rs.3.rs-3125926/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

When it comes to the need to manage cyber security, identifying and utilising good cyber security metrics is essential. This allows organisations to manage their cyber risk more effectively. However, the literature lacks consensus on properties and characteristics of good metrics. We join the current efforts that aim at closing this gap. Hence, the main two objectives of this work are firstly to explore and identify relevant cyber security metrics proposed by researchers in the cyber security domain, and then to assess them against the SMART (Specific, Measurable, Actionable, Relevant, and Timely) criteria to determine their feasibility and improve the quality of the selected security metrics.

We identified 100 relevant metrics, of which 22 were able to be assessed against the SMART criteria. The resulting set of metrics can be considered as feasible set of metrics to implement. Additionally, we have identified the properties that a good metric should possess, most of which can be regarded as variants of the SMART criteria.

Consequently, we extend the subcategories proposed by [1] to enhance the categorisation of metrics. The proposed subcategories are user, interface-induced, and software vulnerabilities; preventative, reactive, proactive defence strength; zero-day, targeted, botnet, malware, and evasion techniques; and security state, incidents, and investment. We propose to include the following: configuration management, access control management, backup and restore, security audit, security testing, and security training. Additionally, we recommend including two additional elements when assessing metrics wherein the metrics should be inexpensive to gather and independently verifiable via an outside reference.

Cyber Security Metrics

SMART Criteria

Properties

Attributes

Categorisation

Peter Drucker, the often cited author of business management approaches, once said “if you can’t measure it, you can’t manage it.” [2]. According to [3]⁠, to make sound decisions about protecting our infrastructure, we need effective cyber security metrics. The authors in [4] recommend that to gauge the success of a cyber security programme we need to be able to provide shareholders with measurable solutions to facilitate decision making.

“Security metrics can provide organisations with a mean to measure and understand the security status of their systems” [5]⁠. According to [6]⁠, metrics can facilitate decision making and improve performance. Similarly, “Effective cybersecurity decision-making demands cybersecurity quantification” [7]⁠. According to [8]⁠, “metrics are tools that help in decision making”. Therefore, we recognise the need to develop effective and suitable cyber security metrics.

The authors in [9] propose that cyber security metrics provide insight, and they can be utilised to measure the cyber security posture of an organisation. The authors in [10]⁠ argue that to objectively evaluate the security of systems we need to devise suitable cyber security metrics. The authors in [11] suggest that cyber security metrics are useful in the evaluation of systems’ security, loss quantification, and in the assessment of cyber security solutions. On the other hand, “lack of security metrics exacerbates the challenge of measuring risk objectively” [12]⁠. The authors in [13] discuss that metrics can be used to provide reproducible and repeatable measures that reflect the security protection level. Hence, metrics can be used not just to evaluate the security level, but also for measuring risk.

According to [14]⁠, organisations should assess their cyber risk before they address security measurement. The authors in [15] suggest that cyber risk should be quantified within predictive analytics. The authors add that cyber security metrics can be used to measure systems’ vulnerabilities. Although, “there is a lack of cyber security metrics for vulnerability assessments, and minimal integration of cyber security metrics within risk assessment” [12]. The authors in [16]⁠ discuss that cyber security risk assessment is usually performed with the help of cyber security metrics. This clearly calls for the integration of cyber security metrics within cyber risk management programs.

However, organisations need to agree to a definition of cyber security metrics that suits their business first and foremost before they begin selecting their cyber security metrics. “Without a restrictive definition, the term metric degenerates to a buzzword, which can be dangerous in terms of suggested comparability” [17]⁠. Although “There is no standard definition of what constitutes a cybersecurity metric” [16]. The authors in [6] define a risk metric as the unit by which risk is measured. The authors in [9] define a security metric as “a collection of several measurements taken at different points in time, compared against baseline and interpreted to reveal an understanding”. The authors in [16] define cyber security metric as something of a value that results from measuring security properties of a system. Thus, we recognise that the process of developing metrics should begin with a suitable definition for the desired metrics as well as stating the properties or attributes that good metrics should possess.

Moreover, according to [1]⁠, several organisations in the United States recognise that developing effective and suitable cyber security metrics is a hard problem. The authors add that there is a gap between the existing metrics and the suitable metrics. The author in [18] discusses that security metrics pertain to things that can be evaluated and used to measure the security of an information system, which has emerged as an indispensable difficulty in the field of information systems. According to [19]⁠, collection of detailed cyber security metrics is a very difficult task to accomplish. The authors in [20] argue that despite the enormous efforts made, cyber security metrics remains an open problem.

Furthermore, “there are no direct methods of measuring strength of cyber security” [8]⁠. Even though, the author in [21]⁠ argues that solving the problem of quantifying cyber security would contribute to answering decision making and risk management related questions. The author adds that to solve such a problem, practitioners need to define a set of cyber security metrics that are effective and suitable to achieve quantitative cyber risk management and thus improve decision making. The authors in [20] suggest that cyber security metrics research can be categorised as follows: first, define metrics to measure, then design procedures to measure the well-defined metrics.

This paper is organised as follows: in section two, we highlight the related work on the topic of security metrics. In section three, we introduce the methodology we used to survey the scholarly knowledge on this topic. This is followed by the discussion section where we explain how we identified and assessed the metrics. Finally, the conclusion section summarises the main outcomes of this work and indicate future research directions.

In this section, we discuss the works that have researched cyber security metrics. We particularly focus on the importance of working with metrics as well as the properties that a good metric should possess.

According to [22]⁠, there is no clear definition for the notion of more secure, and thus cyber security metrics should be leveraged and integrated in the security assessment. The authors in [23] discuss that traditional approaches normally take a snapshot of an organisation at a given time, at which point cyber security metrics would be generated. Hence, the author in [24] argues that cyber security metrics can be used to benchmark various systems if the conditions of the assessment are maintained. Although, the author assessed the proposed metrics with the help of subjective methods only. Furthermore, according to [22], cyber security metrics can be used to provide an up-to-date information regarding the security state of defence and attack sides. The authors add that cyber security metrics allow comparing security systems with each other. The authors in [8]⁠ argue that security metrics can be used to leverage resources and determine whether a security solution has succeeded. The authors add to respond effectively to the ever-changing cyber threat landscape, the strength of cyber systems need to be quantified. Moreover, the authors in [5] suggest that cyber security metrics can be used to measure the performance of an organisation against its peers. The authors point out that quantifying the proposed metrics will be addressed in a future work.

The author in [25] developed risk-based metrics that can detect anomalous behaviour to mitigate the effects of attacks on the critical infrastructure of the United States. Although too many metrics are expensive to implement and may be too difficult to manage. Additionally, “there are many metrics that may not be directly measurable (e.g., attacker capabilities)” [21]⁠. Furthermore, “There are several reasons why a cybersecurity metric solution may not be functioning as it should. This can relate to problems with the metrics themselves, such as missing data” [16]⁠. Additionally, the authors in [22] highlight that there is no formal model for metrics whereby we can conduct a rigorous analysis. The authors in [26] identified a set of security metrics to assess network security when there exist a limited number of vulnerabilities. However, it is not clear how the proposed metrics change should the configuration changes. Furthermore, the authors in [27]⁠ found that when the number of vulnerabilities become large, the cyber security metrics’ values become static. The authors in [23]⁠ argue that a static cyber risk management approach may become obsolete very quickly because many parameters used in the analysis constantly change over time.

According to [1]⁠, there is a lack of discussion about how cyber security metrics can be used as parameters in cyber security modelling. The authors in [22] argue that to use cyber security metrics to determine in advance the required preventative measures against cyber threats, organisations need to be able to process and analyse measurements in a way that allow them to extrapolate their values. Hence, the author in [24] highlights that data analytics can be leveraged to analyse cyber security metrics. Furthermore, the author in [7] argues that cyber security datasets can be leveraged to define cyber security metrics at the following abstraction levels: data, knowledge, and application. The authors in [8] point out that cost computation may become too difficult without data driven cyber security metrics and thus decision making with regard to cyber security spending becomes harder. Therefore, looking at leveraging data science advances in cyber security metrics research is worth looking into.

Moreover, the authors in [8]⁠ argue that a metric is usually a product of one or more measures. Hence, the author in [7] suggests that a cyber security metric can be a function that maps from a set of objects to a set of values with a scale. The authors in [3] discuss that a technical security metric is the output of a mathematical model whereby we utilise measurements of a technical object. Furthermore, the authors in [28] argue that measuring the security of a system would enable security engineers to improve decision making with regards to how to best design their system. The authors proposed a model whereby the probability of failure for different security components can be quantified to determine the critical ones. Moreover, according to [29]⁠, measurements have something in common which is an aspect of something that has a descriptor to allow comparison. Similarly. “Metrics are descriptive and measure the current properties and performance of a system” [15]⁠. Hence, the authors in [13] argue that security metrics can be used to compare different evaluations of the same hardware or software over time. The authors add that security metrics can be used to evaluate level of compliance with a standard.

Moreover, the authors in [30] argue that it is important to develop specific cyber security metrics for the purpose of securing systems while conducting penetration testing. Although, the authors have not provided a definition of the desired metrics or what properties such metrics should possess. According to [17]⁠, security metrics are classified according to their properties. Thus, choosing the right properties or characteristics that a metric should possess is as important as developing a definition for the metric.

In general, the authors in [31] discuss that, when it comes to properties of a good metrics, we should be able to consistently measure them, collect them by automated means, and express them as percentage or cardinal number. The authors in [14]⁠ suggest that “values of metrics should be related to the following qualities: efficiency, objectivity, and consistency”. According to [18], the attributes of good metrics are as follows: they need to be clear, cheap, and can be collected by automated means, as well as they should be measured objectively. We notice that the authors stress the importance of being able to systematically measure metrics and to not rely on subjective criteria.

However, in the context of security metrics, the authors in [3]⁠ discuss that the attributes of technical security metrics are that the number needs to be small so that they can be managed effectively, they need to be easy to understand, and they need to be measured objectively. According to [32]⁠, good security metrics should be able to measure properties for decision making, be measured in a repeatable manner, and be verifiable independently. The authors in [1]⁠ discuss that effective security metrics need to be easy to understand, measured in units, and expressed in numbers. Thus, we see that the common properties proposed by [3]⁠, [32]⁠ and [1] are being easy to understand, measured objectively, and verifiable via an outside reference. Even though, “there is a lack of a comprehensive set of security metrics capable of being computed in real-time” [12].

Moreover, the author in [21]⁠ calls for establishing a set of criteria that can distinguish between good and poor metrics because “good metrics lead to good decisions and bad metrics lead to bad decisions” [3]⁠, although, there seems to be lack of research into addressing such a problem in the literature. Moreover, when it comes to the scale of metrics, the authors in [13]⁠ argue that several security metrics proposed in the literature lack adequate description. Therefore, there is a need for further research to avoid developing poor metrics unnecessarily, as well as to get insight into the description and scale of good metrics.

In the next section, we present the methodology used in this work in which we explain our approach in extracting the required information from the relevant sources.

To survey the scholarly knowledge on the topic of security metrics, we performed the literature search on the following five academic databases: IEEE Xplore, ACM Digital, Library, ScienceDirect, Scopus, and SpringerLink. We used the advanced search features on all engines to facilitate the process. We used double quotation to search for the exact phrase (search string). After we had applied the inclusion/exclusion criteria, we then examined the results to remove irrelevant and duplicate sources.

Table 1

Highlights of the searches
Search string	Search engine	Date	Hits	I/E	Eligible	Remarks
cybersecurity metrics	IEEE Xplore	10/10/22	19	6	3	Search: in document title OR publication title OR author keywords OR abstract. Filter: conferences, journals
cyber security metrics		10/10/22	7	5	4
systems security metrics		08/11/22	6	4	3
cybersecurity metrics	ACM Digital Library	10/10/22	8	1	1	Search: title OR publication title OR author keyword OR abstract
cyber security metrics		10/10/22	8	2	0	Query: Title:("search string") OR ContentGroupTitle:("search string") OR Keyword:("search string") OR Abstract:("search string")
systems security metrics		08/11/22	3	1	1
cybersecurity metrics	ScienceDirect	10/10/22	23	1	1	Search: title, abstract or author-specified keywords. Filter by article type: research/review
cyber security metrics		10/10/22	13	1	1
systems security metrics		08/11/22	27	2	1
cybersecurity metrics	Scopus	17/10/22	132	32	9	Search: article title, abstract, keywords. Filter by document types: conference paper, article
cyber security metrics		17/10/22	84	25	2	Query: TITLE-ABS-KEY("search string") AND (LIMIT-TO (DOCTYPE,"cp") OR LIMIT-TO (DOCTYPE,"ar"))
systems security metrics		08/11/22	218	13	1
cybersecurity metrics	SpringerLink	18/10/22	49	2	2	Manually: Search with the exact string. Filter by content type: journal article, conference paper. And then search in abstract and keywords
cyber security metrics		18/10/22	30	6	2
systems security metrics		08/11/22	23	13	1

With respect to the inclusion and exclusion criteria, we included studies that primarily discuss cyber security metrics, written in English, and whose type are either journal articles or conference proceedings. While the exclusion criteria include studies that do not specifically address cyber security metrics (including definition, properties, etc.), non-peer reviewed sources, and full text is inaccessible. Table 1 highlights the searches conducted at the beginning of this work. We used three different strings to search for relevant sources in each database. The US literature manly uses ‘cybersecurity’ keyword, while in the UK it is generally defined as ‘cyber security’, in addition to ‘systems security metrics’ that some researchers use to refer to the field of security metrics. The search string column is followed by the search engine name. This is followed by the date of the search. This is followed by the number of hits. This is in turn followed by the number of remaining sources after applying the inclusion/exclusion criteria. This is followed by the eligible sources after removing irrelevant and duplicates. Lastly, the final column contains remarks about how the advanced search feature was utilised in each engine. We note that we did the search manually in the SpringerLink engine because it does not have an advanced feature to do so automatically as the other four engines.

As shown in Table 1, we see that the total number of hits is 650, whereas after applying the inclusion/exclusion criteria, the total number is 114. Lastly, after removing false/positives and duplicates, the final number is 32.

In the next section, we discuss how we extracted the required information from the identified sources.

This section has two subsections dedicated to presenting information regarding the synthesis and evaluation of information gathered from the sources identified in the previous section.

4.1 Information synthesis

Throughout the review process, we maintained our database in the form of a spreadsheet that comprises the following sheets:

Search highlights sheet that contains the search strings, search engine, date of search, overall hits as well as hits after applying the inclusion and exclusion criteria.
Highlights sheet with the following headings: title of study, name of authors, publication date, summary, conclusion, recommendations, properties of cyber security metrics. This is followed by the proposed cyber security metrics.

Cyber security metrics sheet with the following headings: study reference number, publication date and type. This is followed by the proposed cyber security metric along with its description. This is followed by category and subcategory. The categories and subcategories are adopted from [1] (see Table 2). In addition to these categories, we have included one more category named other, whereby we list any metrics that do not fall under the main four categories. Additionally, to enhance the categorisation of metrics, we extended the subcategories of all categories to include additional ones that were elicited from some of the selected sources.

Table 2

Summary of the main categories and subcategories
Category	Subcategories	Remarks
Vulnerability	User, Interface-induced, and Software	pertains to measuring the level of system vulnerability
Defence	Preventive, Reactive, and Proactive	aims to measure the strength of cyber defence
Attack	Zero-day, Targeted, Botnet, Malware, and Evasion techniques	aims to measure the strength of cyber attacks
Situation	Security state, Incidents, and Investment	reflects the development of attack-defence interaction

In addition to these categories, we have included one more category named other, whereby we list any metrics that do not fall under the main four categories. Additionally, to enhance the categorisation of metrics, we extended the subcategories to include additional ones that were elicited from the following sources: 1, 3, 4, 8, 9, 18, 21, 24, and 26. They are as follows: configuration management, access control management, backup and restore, security audit, security testing, and security training. In addition to probability-based, time-based, ideal-based, and design-based metrics.

SMART assessment sheet lists all the identified cyber security metrics in the previous stage in addition to five columns that represent the SMART criteria so that we could assess each metric accordingly.
Properties of cyber security metrics sheet contains the key attributes that a metric should possess.

This approach allowed us to gather and keep track of all the relevant information in a way that enabled us not just to get insights into the direction and trends of the research in this area, but also to find gaps in the existing metrics, their properties and classification.

4.2 Information analysis of SMART metrics

In this subsection, we discuss the approach we followed to analyse the information gathered in the previous subsection. In terms of source types, 70 percent of the selected sources are conference proceedings, 13 percent of which used case study as their method, whereas four percent of the journal articles are survey research papers.

We analysed the selected sources by studying their contribution to security metrics research in two key aspects, one of which is proposal of metrics and/or proposal of properties that a good metric should have. Moreover, when gaps and/or unanswered questions found, we added them under the problems heading of the sheet. In addition to recommendations for future research directions.

The main three headings of the cyber security metrics sheet are the metrics, categories, and subcategories, which are the main elements of this work. Most of the sources proposed security metrics. Nonetheless, the ones that did not propose any metrics explicitly they either addressed the definition or properties of metrics. We recorded all the proposed metrics accompanied by their proposed description whenever provided, although some sources did not include a description with their proposed metrics and, hence, we noted them with no description.

Moreover, the authors in [33] recommend that “metrics should be SMART (Specific, Measurable, Actionable, Relevant, and Time-dependent)”. According to [13]⁠, the SMART criteria has been used widely for developing good security metrics. Thus, we assessed all the identified metrics against the SMART criteria. Each metric needs to satisfy the five elements of the criteria. Therefore, the total number of security metrics identified from 24 sources are 100, all of which fall under five categories and 26 subcategories. While the total number of the security metrics after applying the SMART criteria is 22, all of which fall under four categories and 10 subcategories.

The statistics of security metrics count (C) can be broken down further, to count how many times a metric is identified (i.e., per source) per category/subcategory as outlined in Table 3.

Table 3

Count of sources from which security metrics were identified per category/subcategory
Category	Subcategory	Count
System Vulnerabilities	Interface-Induced Vulnerabilities	13
System Vulnerabilities	Software Vulnerabilities	8
System Vulnerabilities	Overall Vulnerabilities	5
System Vulnerabilities	User Vulnerabilities	5
Situation	Security Incidents	9
Situation	Security State	5
Situation	Cost	4
Situation	Resilience Metrics	3
Situation	Agility Metrics	1
Other	Backup and Restore	4
Other	Access Control Management	3
Other	Configuration Management	3
Other	Security Training	3
Other	Probability-Based Metrics	2
Other	Compliance	1

Similarly, Table 4 shows the counts of sources per category after applying the SMART criteria.

Table 4

Count of sources from which SMART security metrics were identified
Category	Subcategory	Count	Sources
System Vulnerabilities	Software Vulnerabilities	6	2, 3, 5, 8, 19, 25, 30
System Vulnerabilities	Interface-Induced Vulnerabilities	5	3, 9, 25 (2), 27
System Vulnerabilities	Overall Vulnerabilities	4	4, 9, 15, 35
System Vulnerabilities	User Vulnerabilities	5	2, 3, 8 (2), 18
Other	Configuration Management	3	3, 8, 29
Other	Security Training	3	9, 15, 18
Defence Power	Intrusion Detection	4	3, 15, 19, 29
Defence Power	Proactive Defences Strength	2	3, 20
Defence Power	Preventative Defences Strength	1	19
Situation	Security Incidents	6	5 (2), 8, 9, 30, 31

Moreover, we view the security metrics per category or subcategory. Hence, to present the data visually, we first imported the two datasets (total cyber security metrics and the SMART security metrics) to Elasticsearch - “the world’s leading free and open search and analytics solution” [34]⁠, and then we used Kibana to create various visualisations, such as data table, tag cloud, pie chart, etc. For instance, we started creating visualisations for the first dataset to show statistics in relation to the percentage of metrics that fall under each category and subcategory so that we can gain insight into where most metrics are categorised. Additionally, we created visualisations to show security metrics per category. This is useful when we look at the metrics from the categorisation perspective. Therefore, we began by creating a tag cloud visualisation to show all the identified metrics as shown in Fig. 1.

We note that vulnerability count metric is the top metric followed by attack surface and mean time between incidents.

Moreover, some visualisations could not show all data due to the size constraint that yield in missing labels from pie charts; this happens when there are many slices (i.e., it is not possible to visualise all the labels in one figure). To solve this problem, we created a visualisation for each category so that we could show all the security metrics and subcategories within each category and then add it to the main dashboard. However, this was not necessary for the attack/threat severity category because there is a total of three security metrics identified all of which did not satisfy the SMART criteria. Hence, since the focus of this work is on the assessed security metrics, we present the visualisations that address the SMART security metrics.

We start with the category named other, Fig. 2 shows a pie chart of the assessed security metrics (i.e., after applying the SMART criteria) that fall under this category accompanied by the percentage of how many times a metric was identified out of the total number of the identified metrics within the category. To do this, under slice by, we add the category field first, and then add the subcategory field as a second slice, aggregated by the security metric field. Hence, the inner circle represents the category, the middle circle shows the subcategories, and the outer circle shows the security metrics.

We see that the top security metric in this category is the trained personnel count, identified in three sources. The authors in [9]⁠ describe this metric as “percentage of information system security personnel that have received security training”.

In Fig. 3, we present the security metrics that fall under the situation category.

The top security metric in this category is the unauthorised access attempts, identified in three sources. The authors in [5]⁠ describe this metric as “high number of failed authentication attempts”.

In Fig. 4, we present the security metrics that fall under the defence power category.

The top security metric in this category is the detection performance, identified in four sources. The authors in [3]⁠ describe this metric as “a measure of the effectiveness of the detection mechanisms (intrusion detection system, anti-virus software, etc.)”.

In Fig. 5, we present the security metrics that fall under the system vulnerabilities category.

The top security metric in this category is the vulnerability count, identified in nine sources. The authors in [1]⁠ describe this metric as the number of systems that haven’t been patched yet. The authors add that it normally takes a while for all the required patches to be applied successfully.

We created one more visualisation that shows all the SMART security metrics as shown in Fig. 6.

We can see that majority of the identified metrics fall under system vulnerability category with 51 percent, 18 percent of which fall under software vulnerability subcategory, while 13 percent fall under interface-induced vulnerability and 11 and 9 percent fall under user and overall vulnerability subcategory respectively. Hence, the top security metrics is the vulnerability count with 16 percent. Similarly, the second top category is defence power with 20 percent that comprises three subcategories intrusion detection 13 percent, proactive defences strength four percent, and preventative defences strength 2 percent. Hence, the top security metric is detection performance with nine percent.

In Table 5, we summarise the security metrics that satisfied the SMART criteria. This is followed by a column that contains the category and subcategory the security metrics fall under. This is followed by a column that contains the number of times a metric was identified from the selected sources.

Table 5

Count of sources from which SMART security metrics were identified
Metric	Category	Subcategory	Count	Sources
Vulnerability Count	System Vulnerabilities	Software Vulnerabilities	6	2, 3, 5, 8, 19, 30
Vulnerability Count	System Vulnerabilities	Overall Vulnerabilities	3	4, 15, 35
Detection Performance	Defence Power	Intrusion Detection	4	3, 15, 19, 29
Password Guessability	System Vulnerabilities	User Vulnerabilities	4	2, 3, 8, 18
Access Points Count	System Vulnerabilities	Interface-Induced Vulnerabilities	3	3, 9, 25
Trained Personnel Count	Other	Security Training	3	9, 15, 18
Unauthorised Access Attempts	Situation	Security Incidents	3	5, 8, 30
Detection Mechanism Deficiency Count	Defence Power	Intrusion Detection	2	3, 15
Patch Count	Other	Configuration Management	2	8, 29
Access Retries Count	System Vulnerabilities	Interface-Induced Vulnerabilities	1	31
Administrative Privileges Users Count	Other	Configuration Management	1	3
Critical Network Nodes	System Vulnerabilities	Interface-Induced Vulnerabilities	1	25
Defence in Depth Layers Count	Defence Power	Proactive Defences Strength	1	3
Events Assignment Delay	Situation	Security Incidents	1	31
Malware Detector Effectiveness	Defence Power	Proactive Defences Strength	1	20
Non-authorised Devices Count	System Vulnerabilities	User Vulnerabilities	1	8
Percentage of Severe Systems	System Vulnerabilities	Interface-Induced Vulnerabilities	1	27
Physical Incidents Percentage	Situation	Security Incidents	1	9
Prevention Performance	Defence Power	Preventative Defences Strength	1	19
Reported Incidents Percentage	Situation	Security Incidents	1	9
Reported Security Holes	System Vulnerabilities	Software Vulnerabilities	1	15
Simultaneous Logins Count	Situation	Security Incidents	1	5
Vulnerabilities Mitigation Count	System Vulnerabilities	Overall Vulnerabilities	1	9

We note that vulnerability count metric is categorised under two subcategories software and overall respectively. This is due to the different description given by the authors from which we identified the metrics. For instance, the authors in [3]⁠ describe vulnerability count as “the sum of known and unpatched vulnerabilities, each multiplied by their exposure time interval” which falls under software vulnerability subcategory. While the authors in [35]⁠ describe vulnerability count as “measures the number of vulnerabilities present in a system” which falls under overall vulnerability subcategory. That is why it is important to develop an appropriate definition or description for the desired metrics right from the start.

4.3 Properties of good metrics

In this subsection, we discuss some of the attributes that a good metric should possess.

According to [17]⁠, security metrics are classified according to their properties. The author discusses that security metrics need to be objective and verifiable. The author [29]⁠ argues that top level security metrics should be dynamic. The authors in [32]⁠ recommend that a good metric should be repeatable and verifiable. The authors in [15] discuss that in addition to repeatable, metrics should be feasible, quantifiable, and objective. The authors in [1]⁠ agree that metrics should be quantifiable, and they add that they should also be easy to understand. The authors in [12]⁠ argue that a good metric needs to be efficient and cost effective.

Although, the authors in [9]⁠ argue that cost effective characteristic is difficult to achieve. According to [18]⁠, in addition to meaningfulness, objectiveness and cost effectiveness, it is important that we should be able to collect metrics by automatic means. The authors in [13]⁠ recommend that metrics should be comparable and reproducible. In addition to reproducible, the authors in [22]⁠ recommend that metrics should be time-bound. The authors in [9]⁠ argue that security metrics should incorporate time.

Moreover, we identified all the relevant properties from the selected sources and then compared them to each element of the SMART criteria (Specific, Measurable, Actionable, Relevant, and Timely). In this way, on one hand, we would verify the elements of the SMART criteria, and on the other hand we would be able to enhance the criteria by extending it to include properties identified in the literature that can add value in relation to assessing security metrics.

Table 6 summarises the identified properties accompanied by the number of times a property was identified, followed by the element of the SMART criteria that can be matched with or equivalent to such a property, this is followed by some remarks to denote whether a property matches, equivalent to, or can be considered as a variant of one of the five elements of the criteria.

Table 6

Summary of the properties that a good security metric should have.
Property	Count	SMART	Sources	Remarks
Objective	7	S	3, 9, 14, 15, 17, 18, 22	equivalent to Specific
Quantifiable	5	M	2, 3, 13, 15, 31	equivalent to Measurable
Cost effective	4	No match	9, 12, 13, 18	inexpensive to gather
Repeatable	3	~M	13, 15, 32	considered as a variant of Measurable
Reproducible	3	~M	9, 13, 22	considered as a variant of Measurable
Verifiable	3	No match	9, 17, 32	independently verifiable via an outside reference
Meaningful	3	R	9, 18, 31	equivalent to Relevant
Auto collected	2	A	18, 31	i.e., it can be collected by automated means
Consistent	2	~T	14, 31	considered as a variant of Timely
Easy to understand	2	~S	2, 3	considered as a variant of Specific
Efficient	2	~R	12, 14	considered as a variant of Relevant
Comparable	1	~S	13	considered as a variant of Specific
Feasible	1	A	15	1) equivalent to Actionable
Timely	1	T	22	Same

We see that most of the identified properties are equivalent to an element of the SMART criteria, some of which can be considered as variants. The two unique attributes are cost effective and verifiable. Thus, the SMART criteria should be extended to include these two attributes.

Finally, in terms of any future efforts to implement a subset of the identified security metrics, there are two additional aspects that need to be looked at, one of which is the scale and the other is scope. The author [29] discusses that security metrics should have a proper scale that corresponds with the analysis type. The authors in [13]⁠ argue that several security metrics lack from an appropriate description of scale. The authors suggest that scale can be nominal, ordinal, interval, ratio, or absolute. While scope can be user, software, hardware, or organisation.

We have identified 100 relevant security metrics, 22 of which satisfied the SMART criteria. The final set of metrics can be regarded as feasible to implement and organisations should select the ones that would best suit their requirements and preferences.

Moreover, to enhance the categorisation of metrics, we extended the subcategories proposed by [1]⁠ to include the following: configuration management, access control management, backup and restore, security audit, security testing, security training.

Furthermore, we have identified the properties that a good metric should possess, most of which can be considered as variants of the SMART criteria except two, cost efficiency and verifiability. Hence, the SMART criteria should be extended to include these two elements. Thus, this is particularly useful to facilitate the assessment of metrics.

The outcome of this work will feed into future research that aims to look at the role that effective cyber security metrics can play in measuring cyber security posture of organisations. We envisage that metrics will be selected depending on the organisation’s requirements and preferences, whereby the organisation could measure cyber risk in a robust and reliable manner.

Competing interests:

the authors did not receive any funding for performing this work. Additionally, they do not have interests to disclose. Finally, the authors do not have any competing interests to declare that are of relevance to this article.

Pendleton, M., Garcia-Lebron, R., Cho, J.H., Xu, S.: A survey on systems security metrics. ACM Computing Surveys. 49, (2016). https://doi.org/10.1145/3005714
Patrinos, H.: You Can’t Manage What You Don’t Measure. Available at https://blogs.worldbank.org/education/you-can-t-manage-what-you-don-t-measure (2014). Accessed 24 May 2023
Boyer, W., McQueen, M.: Ideal based cyber security technical metrics for control systems. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp. 246–260 (2008)
Abraham, S., Nair, S.: Exploitability analysis using predictive cybersecurity framework. In: Proceedings - 2015 IEEE 2nd International Conference on Cybernetics, CYBCONF 2015. pp. 317–323 (2015)
Ahmed, Y., Naqvi, S., Josephs, M.: Cybersecurity Metrics for Enhanced Protection of Healthcare IT Systems. (2019)
Aziz, B., Malik, A., Jung, J.: Check your blind spot: A new cyber-security metric for measuring incident response readiness. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp. 19–33 (2017)
Xu, S.: SARR: A Cybersecurity Metrics and Quantification Framework (Keynote). (2021)
Gupta Bhol, S., Mohanty, J., Kumar Pattnaik, P.: Taxonomy of cyber security metrics to measure strength of cyber security. Mater Today Proc. (2021). https://doi.org/10.1016/j.matpr.2021.06.228
Jafari, S., Mtenzi, F., Fitzpatrick, R., O’shea, B.: Security Metrics for e-Healthcare Information Systems: A Domain Specific Metrics Approach. (2010)
Stolfo, S., Bellovin, S.M., Evans, D.: Measuring security. IEEE Secur Priv. 9, 60–65 (2011). https://doi.org/10.1109/MSP.2011.56
Rabai, L.B.A., Jouini, M., Nafati, M., Aissa, A. Ben, Mili, A.: An economic model of security threats for cloud computing systems. In: Proceedings 2012 International Conference on Cyber Security, Cyber Warfare and Digital Forensic, CyberSec 2012. pp. 100–105 (2012)
Knowles, W., Prince, D., Hutchison, D., Disso, J.F.P., Jones, K.: A survey of cyber security management in industrial control systems. International Journal of Critical Infrastructure Protection. 9, 52–80 (2015). https://doi.org/10.1016/j.ijcip.2015.02.002
Longueira-Romerc, A., Iglesias, R., Gonzalez, D., Garitano, I.: How to Quantify the Security Level of Embedded Systems? A Taxonomy of Security Metrics. In: IEEE International Conference on Industrial Informatics (INDIN). pp. 153–158 (2020)
Kowalski, S., Barabanov, R., Hoffmann, R.: Cyber security alert warning system: A socio-techinal coordinate system proposal. In: Proceedings - 2011 3rd International Workshop on Security Measurements and Metrics, Metrisec 2011. pp. 21–24 (2011)
Scala, N.M., Goethals, P.: A Review of and Agenda for Cybersecurity Policy Models. (2016)
Van Haastrecht, M., Ozkan, B.Y., Brinkhuis, M., Spruit, M.: Respite for smes: A systematic review of socio-technical cybersecurity metrics. Applied Sciences (Switzerland). 11, (2021). https://doi.org/10.3390/app11156909
Hecker, A.: On system security metrics and the definition approaches. In: Proceedings - 2nd Int. Conf. Emerging Security Inf., Systems and Technologies, SECURWARE 2008, Includes DEPEND 2008: 1st Int. Workshop on Dependability and Security in Complex and Critical Inf. Sys. pp. 412–419 (2008)
Geleta, R.: Cyber security metrics for performance measurement in e-business. In: Proceedings of the International Conference on Smart Systems and Inventive Technology, ICSSIT 2018. pp. 220–222 (2018)
Zhao, X., Zhao, J., Jiang, X., Zhang, X., Zhang, W.: Construction and Security Measurement of Cybersecurity Metrics Framework Based on Network Behavior. In: Journal of Physics: Conference Series. Institute of Physics Publishing (2019)
Charlton, J., Xu, S.: A New Method for Inferring Ground-Truth Labels and Malware Detector Effectiveness Metrics. (2021)
Xu, S.: The Cybersecurity Dynamics Way of Thinking and Landscape. In: MTD 2020 - Proceedings of the 7th ACM Workshop on Moving Target Defense. pp. 69–80 (2020)
Yevseiev, S., Milov, O., Opirskyy, I., Dunaievska, O., Huk, O., Pogorelov, V., Bondarenko, K., Zviertseva, N., Melenti, Y., Tomashevsky, B.: DEVELOPMENT OF A CONCEPT FOR CYBERSECURITY METRICS CLASSIFICATION. Eastern-European Journal of Enterprise Technologies. 4, 6–18 (2022). https://doi.org/10.15587/1729-4061.2022.263416
Vega-Barbas, M., Villagrá, V.A., Monje, F., Riesco, R., Larriva-Novo, X., Berrocal, J.: Ontology-based system for dynamic risk management in administrative domains. Applied Sciences (Switzerland). 9, (2019). https://doi.org/10.3390/app9214547
Mizanoor Rahman, S.M.: Cybersecurity metrics for human-robot collaborative automotive manufacturing. In: 2021 IEEE International Workshop on Metrology for Automotive, MetroAutomotive 2021 - Proceedings. pp. 254–259 (2021)
Schneidewind, N.: Metrics for mitigating cybersecurity threats to networks. IEEE Internet Comput. 14, 64–71 (2010). https://doi.org/10.1109/MIC.2010.14
Yusuf, S.E., Ge, M., Hong, J.B., Kang Kim, H., Kim, P., Kim, D.S.: Security Modelling and Analysis of Dynamic Enterprise Networks. (2016)
Enoch, S.Y., Ge, M., Hong, J.B., Alzaid, H., Kim, D.S.: A systematic evaluation of cybersecurity metrics for dynamic networks. Computer Networks. 144, 216–229 (2018). https://doi.org/10.1016/j.comnet.2018.07.028
Aissa, A. Ben, Abdalla, I., Hussein, L.F., Elhadad, A.: A Novel Stochastic Model For Cybersecurity Metric Inspired By Markov Chain Model And Attack Graphs. (2020)
Pfleeger, S.L.: Useful cybersecurity metrics. IT Prof. 11, 38–45 (2009). https://doi.org/10.1109/MITP.2009.63
Al-Shiha, R., Alghowinem, S.: Security metrics for ethical hacking. In: Advances in Intelligent Systems and Computing. pp. 1154–1165 (2019)
Holstein, D.K., Stouffer, K.: Trust but verify critical infrastructure cyber security solutions. In: Proceedings of the Annual Hawaii International Conference on System Sciences (2010)
Abercrombie, R.K., Sheldon, F.T., Hauser, K.R., Lantz, M.W., Mili, A.: Risk assessment methodology based on the NISTIR 7628 guidelines. In: Proceedings of the Annual Hawaii International Conference on System Sciences. pp. 1802–1811 (2013)
Le, N.T., Hoang, D.B.: Capability maturity model and metrics framework for cyber cloud security. Scalable Computing. 18, 277–290 (2017). https://doi.org/10.12694/scpe.v18i4.1329
Elasticsearch, Elasticsearch. Available at https://www.elastic.co/. Accessed 7 May 2023
Yağdereli, E., Gemci, C., Aktaş, A.Z.: A study on cyber-security of autonomous and unmanned vehicles. Journal of Defense Modeling and Simulation. 12, 369–381 (2015). https://doi.org/10.1177/1548512915575803

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

The SMART approach to develop good cyber security metrics

Status:

Version 1

Abstract

Figures

1 Introduction

2 Related works

3 Methodology

4 Discussion

4.1 Information synthesis

4.2 Information analysis of SMART metrics

4.3 Properties of good metrics

5 Conclusion

Declarations

References

Additional Declarations

Status:

Version 1