Patrick Thomas, a security partner at Netflix, wrote in a Twitter post in November 2016 about the overloaded definition of the term “penetration test” and showed in a graphic why the starting point of a security test should be a discussion between customer and tester about the objective. A few years later, the topic is still relevant, but the term has changed to “red teaming”.

The concept of red and blue teaming originated in the 1960s, and one of the earliest examples was the think tank RAND Corporation, which conducted simulations for the US military during the Cold War. In information security, red teaming refers to the process of testing an organisation’s security by simulating an attempt to infiltrate its IT infrastructure.

The initial discussion is used to clarify the customer’s needs, the maturity of their defences and whether they really need all the phases and specifics of a Red Team Assessment. This requires an understanding of the basic terms and a definition of the services and procedures that can be provided in such an assessment.

Definition of Security Tests

There is no standard definition for security tests, and the terms and services vary from service provider to service provider. From a technical point of view, the name of such a test is of minor importance. What is important is what such a test involves, how it is performed, and what requirements must be met.

We differentiate between three types of security tests:

• Review
• Penetration Test
• Assessment

A review is an assessment of a target based on documentation, interviews and read-only access to the target. The evaluation of existing reports, such as a vulnerability scan or baseline audit, can also be included in the review. Reviews may include an audit of the solution architecture, component configuration and source code. A review is usually conducted together with subject matter experts and all information is made available to the testers.

A penetration test checks a specific target using a defined scope. Testing Guides or checklists can be used for this purpose. In a penetration test, we document what we have assessed and also list in the report the checks where no vulnerabilities were found (severity Passed). A penetration test can be used to test web or mobile applications, involve examining a laptop, or performing a phishing exercise.

We use the term assessment when testing multiple components or an infrastructure with an open scope. Individual vulnerabilities can be combined into attack paths to demonstrate the exploitation of the vulnerabilities and their impact. The report includes documentation of the attack path and vulnerabilities rather than a checklist. Assessments include baseline security, attack simulations, and red and purple team assessments.

The types of tests can be combined. For example, if a web application is to be tested against the OWASP Application Security Verification Standard (ASVS) Level 2 (L2) or Level 3 (L3), this cannot simply be covered by a penetration test. L2 and L3 require access to documentation, source code, configuration and interviews with the development team.

Assessments

When a customer requests “red teaming”, it is usually to test their organisation’s IT infrastructure or to run a specific scenario within that infrastructure. In the article Security Testing – From Hypothesis to Experiment, Tomaso Vasella has already highlighted the differences between a penetration test and a red team assessment.

The pre-engagement phase involves talking to the client to find out which form of assessment is best suited to their needs.

We differentiate between four types of assessments:

• Baseline Security Assessment
• Attack Simulation Assessment
• Red Team Assessment
• Purple Team Assessment

Distinctions

The assessments distinguish between the attackers’ approach, the interaction with the customer, and the maturity of the defenders. The table below summarises the key differences:

In general, we do not consider a Red Team Assessment to be useful for customers without a Security Operation Centre (SOC) and without attack detection capabilities, as an essential part of the assessment is to remain undetected and to perform all operations cautiously. If there is no detection capability, the focus should not be on remaining undetected, but rather on finding vulnerabilities and attack paths.

If a customer explicitly wants to test different components of their infrastructure and get results on the reliability of the client, the VPN solution and the IT ticket portal, we recommend running an Attack Simulation on these components. This includes checking for vulnerabilities that can be combined to form an attack path that could lead to the compromise of a laptop and subsequent lateral movement in the network. In a Red Team Assessment, we act as attackers and look for the most effective attack path to achieve our objectives. This path may not include any of the components that the customer explicitly wants to test. The objectives are set, but the attackers decide the path to achieve them.

The budget for a security test also influences the choice of assessment. It is unrealistic to expect to complete a full Red Team Assessment within 10 days. Depending on the maturity of the defences, a Baseline Security Assessment or an Attack Simulation Assessment with a budget of 10 days is recommended. A Baseline Security Assessment focuses on finding vulnerabilities, for example, by automatically checking for accessible shares or examining the most common misconfigurations in Active Directory, Active Directory Certificate Services (ADCS), or the Microsoft Cloud. As long as there are many such easily exploitable vulnerabilities in the infrastructure, a stealth approach or the development of custom attack tools will not add value. However, if the infrastructure has already undergone a number of security tests, a Red Team Assessment can add value by testing the organisation’s ability to detect and respond to security incidents.

The Attack Simulation and, in particular, the Baseline Security Assessment are carried out using automated processes and open-source tools to find and exploit vulnerabilities. A Red Team Assessment uses purpose-built tools and spends time bypassing security tools. To this end, the customer’s environment can be simulated in our test lab as much as practical and necessary.

During a Purple Team Assessment, the detection capability is explicitly tested. Use cases are developed in advance with the customer and processed in collaboration with the SOC. Each use case runs through a workflow that measures whether an attack technique was detected, only logged, or not detected at all. If an attack technique cannot be executed due to security measures, this is also documented in the use case log. A use case is repeated until a detection rule has been created and successfully tested, or the desired state for the use case has been achieved.

Conclusion

There are several approaches to testing the security of an organisation’s infrastructure. Depending on the objective of the assessment and the maturity of the organisation in terms of resilience and detection capability, an Attack Simulation Assessment, a Red Team Assessment or a Purple Team Assessment will be more suitable. When conducting a “red teaming”, the pre-engagement phase, and the determination of the type of assessment are critical to defining the appropriate training for the organisation. This is key to gaining meaningful insights and improving the resilience and detection capabilities of the organisation.

About the Author

Michael Schneider has been in IT since 2000. Since 2010 he is focused on information security. He is an expert at penetration testing, hardening and the detection of vulnerabilities in operating systems. He is well-known for a variety of tools written in PowerShell to find, exploit, and mitigate weaknesses. (ORCID 0000-0003-0772-9761)

Links

• http://www.pentest-standard.org/index.php/Pre-engagement
• https://infosec.exchange/@coffeetocode
• https://owasp.org/www-project-application-security-verification-standard/
• https://owasp.org/www-project-web-security-testing-guide/
• https://twitter.com/coffeetocode/status/794593057282859008
• https://www.rand.org/
• https://www.scip.ch/?labs.20221013