Writing documentation or reports is not the favourite task of most technicians. IT security is no exception. In addition to technical knowledge, penetration testers should also have the ability to write reports and communicate clearly with people who do not have in-depth technical knowledge. The report shows the customer what has been tested and what has been found. Most customers do not see what happens during a test, so what is not in the report did not happen.

No less important is the documentation of techniques, tools, lessons learned, and knowledge. Newly learned techniques and experiences should be documented in such a way that they are useful in the future, can be shared with others, or can be found even six months later. Sometimes you remember that you wrote something down before, but you can no longer find a record of it.

Report

Once a security test has been completed, the customer receives a report. The purpose of the report is to identify vulnerabilities and provide recommendations on how to address them. It specifies the scope of the assessment, the areas tested and the results of the test. Vulnerabilities should be documented in such a way that the customer can reproduce them.

Our reports are structured in this way:

• Summary: Management summary, results graphs, questions and answers
• Administration: Revision history, distribution list, project team
• Introduction: Project description and objectives, scope definition
• Results: Modalities, listing of results with countermeasures
• Appendix: List of tools used, definition of severities, indexing lists

The contents of the management summary and results are discussed further in the text. The introduction section of the report defines the scope and objectives. This describes the scope of the security test and of the report, and whether certain areas were not covered. The modalities contain technical information on which user accounts with which permissions, which systems and IP addresses were used, and from when to when the test took place. It also lists the documents received for the audit. It is documented whether security controls such as a Web Application Firewall (WAF) were disabled and whether changes such as the installation of updates were made during the assessment.

Management Summary

The management summary is written with the target audience in mind. It should be clear and easy to understand. As a general rule, technical details do not belong in a management summary unless it is explicitly intended for a technical readership. This can be difficult, as one must resist the urge to describe in detail how to exploit a complex vulnerability. However, this exploit can be described in great detail in the results chapter.

Experience in report writing shows that not everyone is going to like the management summary. In particular, findings with critical weaknesses can lead to discussion and controversy. The summary should therefore be based on facts, politics or even polemics should be avoided. The results need to be documented in an understandable and reproducible way, and the ratings must be measurable, as they form the basis of the statements in the management summary.

Every rating in a management summary or across the document needs to include a metric. It must be clear how a rating is calculated or how a statement is quantified. For example, the statement “The result is sufficient”, without defining what is sufficient, leaves room for interpretation and makes it impossible for the reader to qualify the result. The question of how many of the findings need to be addressed for the result to be considered good cannot be answered in an adequate way. In the article on the HardeningKitty Score, we discussed the sense and nonsense of such a score for a complex construct. We therefore refrain from a simple score or rating and instead list the critical and most important vulnerabilities and the statistics of vulnerabilities found in a paragraph in the management summary, followed by the test object’s strengths and weaknesses in further sections.

Results

Veit Hailperin has already addressed the ideal way to document a security test in the article Checklists or Scenarios in 2016. We still use checklists for concept, configuration, and system reviews, as well as for penetration tests of web or mobile applications. For assessments, such as simulating specific attack techniques or a red team assessment, we use a mixture of scenario and checklist-based documentation.

We document all tests in checklists to make it transparent what has been tested and to ensure completeness. The customer knows not only where the vulnerabilities are, but also which areas have already been secured. If a vulnerability is found, we document it in such a way that the customer can reproduce it themselves if necessary. Technical details can be presented in detail, and it can be demonstrated how a vulnerability could be exploited. The inner urge to demonstrate the proof of concept (PoC) in every detail can and should be fulfilled here. In addition to the technical description of a vulnerability, this also includes providing evidence, for example by adding requests and responses in the case of a web vulnerability. A severity level is defined for each vulnerability. We use the classifications Passed, Low, Medium, High, and Emergency, which are based on the industry standard CVSS v3.1. In addition, a countermeasure is suggested for each vulnerability.

In addition to the PDF report, we provide the results in a machine-readable format, either in our own format or in a format defined by the customer. This is to simplify further processing of the results so that the results can be imported into a risk management or ticketing system without the need to copy and paste from a PDF.

Knowledge Documentation

In addition to writing a report, documenting knowledge is another important task that requires writing skills. Even if the knowledge is only recorded for oneself. What seems obvious at the time of use may require additional explanation later, after you have been away from the subject for some time. This is why notes are so important, and why small details can make the difference between success and failure. For example, if a tool requires a certain parameter for the attack to be successful, but this parameter is only mentioned briefly in the documentation without any indication that it needs to be set correctly, this can easily be missed. Especially if your own notes are shared with someone else who is not aware of their significance.

The projects The Hacker Recipes by Charlie Bromberg, PayloadsAllTheThings and Internal All The Things by Swissky are exemplary and a great source of knowledge.

The flow of information in IT security is enormous and changes occur frequently. New tools and attack techniques are published, existing techniques are discovered and need to be adapted, or new vulnerabilities are discovered that offer previously unimagined possibilities. Knowledge is acquired not only through new information, but also through our own research, training, attending courses and conferences, and dialogue with peers. Keeping track of all this knowledge requires writing skills, discipline, and the right tools.

Tools

Over the years I have tried various tools. It started with a disorganised collection of bookmarks, text files, and PDFs, followed by Notepad++ with plugins and tools like Microsoft’s OneNote or later the open-source alternative CherryTree. While OneNote uses a proprietary format for storing notebooks, CherryTree supports SQLite or XML files. I wanted to get away from tools that put everything into one file and tried Markdown with Sublime and ReStructuredText with Sphinx and still use them depending on the project. I am currently using Obsidian and am in the process of converting my existing data collections.

Obsidian uses the Markdown format, allows the creation of structures, supports the linking of notes, displays links in graphs and provides the option to draw simple diagrams or workflows. In addition, templates can be used to automate and predefine many things. Sam Link of TrustedSec shows how this can be used in his article Obsidian, Taming a Collective Consciousness.

Using the Tools

A fool with a tool is still a fool. My approach is to keep a list of unfinished tasks and add new information, such as a tool, to the to-do list. Once I have worked through an item on the list and processed the information, I document it accordingly. This is usually done in my personal Obsidian vault. The documentation is initially kept short, for example a list of commands on how to apply or implement something. With Obsidian, I can link this new information to existing knowledge, so if I document a new attack technique, I can embed it in the existing structure and link to similar techniques or the basic documentation. Next, I use variables instead of fixed values to make the commands easier to use and perhaps automate at a later stage. I also describe the purpose of the command and mention why a particular parameter needs to be set that way or what else needs to be considered. Depending on the topic, I share the information by adding it to an existing repository or publishing it elsewhere.

Naturally, the urge to try out the new knowledge and keep experimenting is sometimes greater than the motivation to write down the knowledge accurately, or the documentation remains half-finished for the first time. Based on my experience that the unfinished documentation will haunt me sooner or later, I then set myself a task with a reminder to finish the documentation later. This works well for me, although people have different strategies and routines for doing this. The important thing is that you manage to fully document what you have learned.

Conclusion and Outlook

Writing reports and documentation is not everyone’s cup of tea. However, it is an essential part of our work and should be practised as such. Practice, experience, and discussions with customers and peers will improve your ability to writing a report. And when you have to perform a retest with a report that you have not written yourself, you will appreciate a detailed, clear description with technical details and reproducible examples.

Documenting knowledge may not seem like a glamorous task, but without documentation it is more difficult to share knowledge. It is also difficult to reproduce a complex attack technique if there are no instructions on how to set it up and use it. Also, if something is written down in detail, it is more likely to be remembered. Having the right tool helps. We are grateful for all the work that goes into documenting protocols, attack techniques, tools, and more. It makes it easier for us to learn new things. That is why we do our part by creating documentation ourselves and sharing it with others.

There is an elephant in the room and that is the use of Large Language Models (LLMs) to create reports. A list of findings is passed to an LLM and the result is a report including a management summary. This is a dream that will not be a reality in a few years time. LLM tools can be used to support the generation of the description of individual findings or countermeasures, to translate texts or to improve the writing style. At the moment (January 2024), I do not believe that an LLM will be able to write an accurate management summary or even a full report. I will be happy to be proved wrong.

About the Author

Michael Schneider has been in IT since 2000. Since 2010 he is focused on information security. He is an expert at penetration testing, hardening and the detection of vulnerabilities in operating systems. He is well-known for a variety of tools written in PowerShell to find, exploit, and mitigate weaknesses. (ORCID 0000-0003-0772-9761)

Links

• https://github.com/swisskyrepo/PayloadsAllTheThings/tree/master
• https://notepad-plus-plus.org/
• https://obsidian.md/
• https://onenote.com/
• https://swisskyrepo.github.io/InternalAllTheThings/
• https://swisskyrepo.github.io/about/
• https://twitter.com/_nwodtuhs
• https://www.first.org/cvss/calculator/3.1
• https://www.giuspen.net/cherrytree/
• https://www.hailperin.ch/
• https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
• https://www.sublimetext.com/
• https://www.thehacker.recipes/
• https://www.trustedsec.com/
• https://www.trustedsec.com/blog/obsidian-taming-a-collective-consciousness