Security Testing - From Hypothesis to Experiment

Security Testing

From Hypothesis to Experiment

Tomaso Vasella
by Tomaso Vasella
on April 18, 2024
time to read: 15 minutes


Security Testing in a nutshell

  • Security testing is the targeted, methodical search for vulnerabilities
  • This identifies the real security level of the tested object
  • A security test is an experiment that answers questions and thus confirms or disproves hypotheses
  • It must be clarified in advance which questions a security test shall answer

Penetration testing, or security testing in general, is the targeted search for vulnerabilities in order to assess the security level of an IT environment and make informed decisions accordingly. What sounds so simple and obvious is often much more complex in practice and frequently not as clearly defined as it may seem.

Penetration testing is an established and well-known practice in the field of information security. Penetration tests are an integral part of many security programs and various industry standards and supervisory authorities stipulate the regular performance of penetration tests. This is positive because it demonstrates that this very effective means of increasing the security level is widely adopted. At the same time there is a danger that because of the habituation effect penetration tests may not be viewed with the necessary care or may be seen purely as a compliance task.

This article looks at a few simple but essential points that contribute significantly to the success and benefits of security tests. In the field of offensive information security, many terms such as penetration testing, ethical hacking, red teaming and others are used. For the purpose of this article, the term security testing is used generically and refers to the entire spectrum of offensive security testing.

Why Security Testing?

Simply put: To determine the actual security level of an IT system, an IT environment or even an entire organization. The aim is to find out as realistically as possible what can happen in the event of real attacks, how far an attacker could get and what the resulting impact could be. In other words, the existing security measures are subjected to a reality check, which identifies weaknesses and potential gaps that could be exploited by real attackers.

With the insights obtained through testing, risks can be realistically assessed and decisions about necessary countermeasures are supported, enabling an organization to optimize its security level. Regular security testing can make a significant contribution to reducing the risk of security incidents and data leaks.

One might ask why security testing is still needed today, given the huge number of security products and services available. The answer is simple: The effectiveness of security measures cannot be adequately assessed without testing them. In addition, IT is particularly subject to constant change, therefore security measures need to be constantly adapted and tested. The idiom “Trust is good, control is better!” is particularly relevant in the field of information security.

Hypothesis and Experiment

The classic approach of the scientific method with the definitions of hypothesis and experiment can be applied very well to security testing. This is valuable because it allows the prerequisites and success factors to emerge more clearly.

Scientific Method

A hypothesis is an unproven statement or an assumption of facts or laws. A hypothesis is usually derived from a theory, for example in our context: Secure development processes mean secure software, best practices work, security products provide sufficient protection, compliance means security and so on. Corresponding hypotheses can be: Sufficient security measures have been implemented, therefore the environment is secure, the procured product is secure, the organization is secure because it is compliant, the system is isolated in such a way that it is inaccessible to attackers.

An experiment is a methodically designed examination to obtain information empirically. This is precisely the purpose of a penetration test. The value of an experiment is measured by the correctness, usability, and usefulness of the obtained information. This information answers questions, that is, it confirms or invalidates the hypothesis made. Therefore it becomes immediately clear that everything starts with the right questions.

Suitable questions could be:

When planning penetration tests, this frequently receives too little consideration. It is often not sufficiently clear which questions a security test should answer and what the answers are to be used for. The preparation and planning phase is therefore particularly important in security testing and must not be neglected. It can be challenging for a customer to know what options are available, what restrictions exist and what points need to be considered. If these aspects are paid too little attention to or are agreed on too vaguely, there is a risk that money will be spent on a test that is not useful enough or that resources cannot be used effectively, and the test will have to be repeated. An experienced service provider provides neutral and objective support and guidance during this phase and ensures that the questions to be answered by the test and the means to be used are clarified before the test assignment is placed.

After all, a penetration test is only a means to an end, namely, to gain information that can be used to make decisions. Usually, the findings are interpreted by the service provider as far as possible and are assessed with severity levels and suitable measures are recommended to limit the risks associated with the findings. It is then up to the customer to decide what the findings mean for them and how to deal with them.

The importance of being clear about the questions to be answered is emphasized once again. In this context, a few of the most common approaches in security testing are compared below to show which questions they can answer.

Comparison of three common security testing approaches

We’re explaining different security testing approaches down below in more detail, including penetration testing and red team testing, as well as the main differences between penetration testing and a bug bounty.

Penetration Testing and Red Team Testing

Penetration tests are usually executed out to examine the security of a test object with a defined scope from the perspective of a real attacker. It is tested whether the networks, platforms, applications, systems, etc. in the scope have vulnerabilities that can be exploited by an attacker. Care is taken to carry out as complete an examination of the test object as possible within the agreed scope. For example, for testing web applications the OWASP Testing Guide is usually used, which includes all topics relevant to the security of web applications. The aim is to achieve comprehensive test coverage to avoid partial statements with limited benefit. Penetration tests are used for the comprehensive investigation of a given scope and are not normally designed to evade detection or to test the detection capabilities of a SOC.

Red team tests attempt to achieve a defined goal from an agreed starting point, such as gaining access to sensitive data or systems. In contrast to a penetration test, however, the focus here is on simulating an attacker who uses all relevant means and methods to achieve the defined goal while remaining as undetected as possible. The aim is therefore not to identify as many weak points as possible in a given scope, but to find weak points that will help to achieve the defined objective. Accordingly, the SOC or the Blue Team is generally not pre-informed during Red Team tests, which also allows statements to be made about the detection and response capabilities regarding advanced attacks. Red team tests are useful if the examined organization already has a certain level of maturity in information security.

Aspect Penetration Testing Red Team Testing
GoalIdentify and possibly exploit vulnerabilities in a defined target system, network or application.Simulation of real attacks to assess an organization’s security posture and resilience.
ScopeFocus on specific test objects that are agreed in the scope during the assignment. A target is comprehensively examined.The scope includes all useful targets (within the agreed rules of engagement) from the attacker’s perspective to achieve the defined goal. Potentially many attack targets are touched and examined for their usability to achieve the objective, but are not necessarily comprehensively checked.
ApproachTypically based on a defined test catalog, for example a checklist of all relevant test points.Use of attack methods of real attackers. Creative, explorative approach without pre-defined methods and tools.
TimescaleSelective, a defined scope is examined as part of a test at a specific point in time. Provides a comprehensive snapshot.Red team tests are often extended over a longer period of time, for example to avoid detection. Depending on the start and target definition, considerably more effort is usually required than for a penetration test.
FrequencyUsually due to a specific trigger, such as a newly developed system or compliance requirements. Can be indicated frequently.Due to the comparatively high effort and the amount of information to be processed by the client, red team tests are usually carried out less frequently.
SuitabilityComprehensive identification of the security level of a test object. Results help to make a specific test object more secure. Broad coverage of a comparatively small scope.Holistic approach to simulate a real, advanced threat to an organization. Results help to increase the overall security level of an organization. In-depth coverage of a comparatively large scope.

The methods and tools used in penetration tests and red team tests are in many cases similar or even the same. Red team tests focus on the resilience of an organization to real, advanced attacks, while penetration tests generally relate to individual systems or areas.

Penetration Testing and Bug Bounty

Bug bounty programs have become increasingly popular in recent years. If set up correctly, important insights can be obtained with a defined maximum expenditure and in a comparatively short time. The most important differences between penetration tests and bug bounties are compared in the following table.

Aspect Penetration Testing Bug Bounty
ScopePrincipally applicable to all types of test objects. Applications and services that are not accessible on the Internet and aspects such as physical security can also be tested.Often, applications that can be accessed directly or indirectly on the Internet are tested. Factors such as particularly isolated applications can make standard bug bounty procedures difficult or impossible.
InteractionDuring testing, direct contact between the testers and the client is common. A presentation of the results and answering questions about vulnerabilities and recommended measures is common. Even after the penetration test has been completed, the testers are usually available to answer questions.Usually little to no direct interaction between testers and clients is possible, only via the bug bounty platform.
Test CasesComplete test coverage of an area, for example OWASP in the web application area.Focus on those issues and vulnerabilities that are known and familiar to the bounty hunters and that promise quick bounties (low hanging fruits). There may be a risk of incomplete test coverage.
TimescaleSelective, a defined scope is examined as part of a test at a specific point in time. Provides a comprehensive snapshot.
Depending on the characteristics of the program, continuous testing is carried out. This can take the form of an ongoing security test. Identified vulnerabilities are reported automatically, which can be an advantage.
ReportingComprehensive, target group-oriented report with management summary, rating of severities and all technical details as well as specific technical recommendations for suitable measures. The tests carried out that did not reveal any weaknesses are also documented.
Reporting of vulnerabilities and recommendations. No documentation of tests that did not lead to vulnerabilities.
Financial ApectsDefined costs according to the scope.Balancing of the attractiveness for bounty hunters against the costs. Expenditure usually depends on the number and type of vulnerabilities that are not known in advance. The amount of the paid bounties has a direct influence on the motivation of the testers.
SuitabilityAn individual penetration test tailored to the target is recommended for an initial test, for example before a go-live and for new, yet untested or less mature test objects. This achieves complete test coverage. In a penetration test, there is a high probability of discovering those vulnerabilities that could cause high costs in a bug bounty program.After a comprehensive initial test, security can be continuously improved through a bug bounty program. This reduces the initial uncertainty about the number of vulnerabilities and thus the unpredictable costs. Bug bounty programs are suitable for the continuous testing of test objects that already have a certain level of security and a relatively good maturity.

Bug bounty programs and penetration tests both aim to identify vulnerabilities, but differ in their engagement models, incentives, and areas of focus. Bug bounty programs leverage the expertise of a large number of testers, offer financial rewards for reporting vulnerabilities, and can provide quick, but perhaps not always comprehensive, results. Penetration testing takes a holistic approach, offers easier interaction with testers, and is conducted by a small number of dedicated experts.


A security test is an experiment that provides information and thereby answers questions. It is therefore very important for the customer and for the service provider to be clear about which questions are to be answered by the test and what the results are to be used for before starting a security test. This is a general statement which applies equally to the various types of security tests. An experienced service provider provides neutral and objective support and advice during this phase and ensures that it is clear which questions the test is intended to answer, and which methods are to be used.

About the Author

Tomaso Vasella

Tomaso Vasella has a Master in Organic Chemistry at ETH Z├╝rich. He is working in the cybersecurity field since 1999 and worked as a consultant, engineer, auditor and business developer. (ORCID 0000-0002-0216-1268)


Is your data also traded on the dark net?

We are going to monitor the digital underground for you!

The new NIST Cybersecurity Framework

The new NIST Cybersecurity Framework

Tomaso Vasella

Flipper Zero WiFi Devboard

Flipper Zero WiFi Devboard

Tomaso Vasella

Denial of Service Attacks

Denial of Service Attacks

Tomaso Vasella

System Log Monitoring

System Log Monitoring

Tomaso Vasella

You want more?

Further articles available here

You need support in such a project?

Our experts will get in contact with you!

You want more?

Further articles available here