Testing and assessing the security posture of IT systems is a major part of the activities of our Red Team. Applications, services and infrastructure components are analyzed from a security perspective and probed for weaknesses and depending on the nature of the assignment, identified weaknesses may be exploited subsequently. Usually, this results in a comprehensive report that describes all findings in detail and suggests suitable countermeasures.

An essential part of the report is the assessment of the detected vulnerabilities from a security perspective. But what is the meaning of these ratings and how is the recipient of the test results supposed to deal with these conclusions? This is a question that repeatedly leads to discussions and uncertainties, especially regarding the risk management process. These relationships and a pragmatic approach to handling the ratings of findings are discussed in this article.

Ratings of Weaknesses

The findings of security tests are typically rated using a scale with several levels as illustrated in the following example.

These severities are a measure for the quality and the effectiveness of the implemented security controls for each test case. They range from controls cannot be circumvented to controls can be circumvented directly and confidentiality, integrity and availability cannot be guaranteed. Thus, the rating of a weakness reflects its severity or in other words, is a measure of the adequacy of the existing security controls. These ratings refer to the conducted test cases only, they usually do not include additional contextual factors.

Risks

Many definitions of the term risk exist. Two selected ones that are suitable in the context of cyber security are quoted below:

ISO/IEC 27005:

The potential that a given threat will exploit vulnerabilities of an asset or group of assets and thereby cause harm to the organization. It is measured in terms of a combination of the probability of occurrence of an event and its consequence.

NIST Glossary:

A measure of the extent to which an entity is threatened by a potential circumstance or event, and typically a function of: (i) the adverse impacts that would arise if the circumstance or event occurs; and (ii) the likelihood of occurrence.

So, a risk is more than just a rating of a weakness. Risks consider the context of a weakness and the possible adverse effects. In addition to the severity of a weakness and the extent of possible negative consequences, the rating of risks also includes relevant threats, existing risk-reducing measures and the likelihood or probability that the vulnerability will actually be exploited. For the qualitative rating of risks (e.g. low, medium, high) based on the impact and the probability of occurrence, a corresponding risk matrix is usually used, as shown in the following example.

Severity versus Risk

The severity of a weakness only changes if the test object itself is changed, for example by remediating the weakness by means of a bug fix or a configuration change. However, the context of a weakness, i.e. its environment is practically never constant over time. Therefore, a risk rating is a snapshot and can change over time, even if the severity of the underlying weakness doesn’t change.

An example: let’s assume that a penetration test identified a weakness in a network service and its exploitation easily allows for the unauthorized exfiltration of sensitive data from the affected system. Thus, the severity of this weakness would probably be rated das high or possibly even as emergency. However, the system is located in an internal, isolated network segment with strict access control and a very small number of users. Because of this, the probability that the weakness is actually exploited is much smaller than if the system were directly accessible from the Internet without additional protection. Therefore, the risk associated to this weakness would most likely not be rated as high but rather as medium or even as low because of the specific context. However, in case this system would be moved into a different, less protected network segment, the risk rating would increase correspondingly. This clearly shows that rating and managing risks is not a single, one-time action but must be carried out as part of a continuous risk management process.

The following table summarizes the most important differences between the terms severity and risk.

From Severities to Risks

As part of their risk management, organizations often also take cyber risks into account or are even obliged to do so by the law or by industry regulations. This often raises the question of how to deal with weaknesses and their ratings from a risk management perspective. In practice, the following pragmatic approach has worked well to compile a qualitative risk rating based on the severity of an underlying weakness:

• Based on the severity of the weakness, the magnitude of the impact that would result from a successful exploitation of the weakness is estimated. Factors such as the confidentiality of the affected data and the business relevance of the involved systems must be considered but also aspects such as reputational damage or the impairment of business relationships.
• It is then considered how likely it is that the vulnerability will actually be exploited, taking into account all relevant surrounding factors. In this context, topics such as existing measures that may reduce risks are relevant, but also the attractiveness for an attacker to exploit the vulnerability and to access information or data must be factored in as well as the systems exposure.

The resulting risk is then looked up in the risk matrix. It is often worth to base the reflections about impact and probability on risk scenarios and to discuss these scenarios in the team. Risk scenarios are established by imagining the entire sequence of events of a certain attack and describing it with enough details to be able to explain the attack to others in an understandable and comprehensible way.

In practice, this process is often not as easy and straightforward to execute and lively discussions about probabilities and risk thresholds may arise. While impact ratings typically relate closely to severities, humans are inherently bad at estimating probabilities and our own intuition easily misleads us. In real life, we are dealing with frequencies, not probabilities, and it is well known that our perception tends to be unconsciously influenced (cognitive bias, see Logical Fallacies when Assessing Risks). Therefore, when managing cyber risks, it is all the more important to focus on the facts, to analyze the situation carefully and to be aware of the context under which the risks were rated. External experts can provide advice and explain the issues they identified in their tests. However, the actual risk assessment must be performed by the organization itself because in most cases, only the organization has sufficient context information.

The apparent discrepancy between the rating of a vulnerability and the rating of the corresponding risk that sometimes occurs can lead to discussions. Often, similar terms and colors are used to indicate severities and risks, for example red or high. It is possible that the severity of weakness is rated as high (red) while the associated risk is rated medium (yellow). This can trigger requests for explanations because supposedly the assessment of the security tester was altered during the risk management process while actually only the difference between severity and risk was overlooked. Again, this illustrates the need for concentrating on the facts and for ensuring transparency by using suitable methods for data presentation.

Conclusion

An external security test identifies weaknesses and rates them in terms of severity. The distinction between severities and risks is important and requires a careful examination of the relevant facts and the test cases. Rating risks based on identified weaknesses and their severities can only be conducted by considering the context of the weaknesses, a process that usually must be performed by the respective organization.

Finally, it must not be forgotten that both the results of security tests and the rating of the corresponding risks reflect the security state at the time of observation. Given today’s dynamic nature of cybersecurity, regular security checks and continuous risk management practice are therefore mandatory.

About the Author

Tomaso Vasella has a Master in Organic Chemistry at ETH Zürich. He is working in the cybersecurity field since 1999 and worked as a consultant, engineer, auditor and business developer. (ORCID 0000-0002-0216-1268)

Links

• https://csrc.nist.gov/glossary/term/risk
• https://www.iso.org/obp/ui/#iso:std:iso-iec:27005:ed-3:v1:en