Security Log, Part 2: Requirements, Costs and Tools

Security Log, Part 2

Requirements, Costs and Tools

Rocco Gagliardi
by Rocco Gagliardi
time to read: 12 minutes

This article is the second in a series of two. In the first part we discussed how to approach log management and our experience with it. In this part, we’ll look more specifically at the requirements and costs of the various stages as well as provide an overview of some open-source tools.

Foreword

Drawing a log management infrastructure is a complex task and requires the collection and analysis of requests from different company departments. After the preparation phase and the definition of the requirements, the design of the infrastructure can begin.

Since 2005, the SANS Industry Analyst Team conducts annual surveys on installed Log management solutions; looking at the results by year, we can extrapolate the main reasons, or expectations, behind a log management solution deployment.

SANS Log Management Trends

The graph illustrates how the various request priorities have changed over the years. Note how it has gone from Important to Critical, while the overall trend is upward for the first three and falling for the last two.

In a first phase, the systems were used for the optimization and monitoring IT infrastructure. with time, it became more important to be compliant with regulations and to track complex suspicious activity (increase of Critical versus detriment of Important). Keep this trend in mind when planning your solution.

Without pretending to exhaust the subject in these few lines, below we list, without extensive discussions, some basic criteria that we consider necessary for each component of the solution.

Infrastructure

Collect and Store

A good solution should be able to collect log data from across an enterprise regardless of their source. Syslog is the well known protocol for collecting systems log records in the UNIX world. Thanks to its relative simplicity, it has also been used in the Windows world. It is also simple to build interfaces from the different applications to send logs trough syslog infrastructure.

The system administrators should be able to remove or compress reports of events known to be uninteresting in favor of alert monitoring for events known to be interesting. But be conservative in discarding useless events, remember that the log is a friend so keep all possible messages.

Based on the requirements, tune your applications to generate the required information:

About 3/4 of organizations implementing log management solutions, collect logs from:

Therefore, it is very important to not forget to sync time across devices and take care of the timezone.

Key points to consider while designing/choosing a log solution:

Report

A robust infrastructure will perform both logging and auditing and store data using configurable automated methods to summarize them. At its simplest, reporting is a selection and formatting of the data stored. Its primary goal is to make a huge amount of data readable, rendering them in graphical format or extract just a few lines having very important content.

Building a report is not trivial, it involves a deep understanding of the data and how it is stored. Good tools provide standard reports out of the box. It is preferable to pick up 10/15 of those reports, tune and use them, not start playing with Jasper Report to get a customized 3 columns table and a pie graph with company logo. Reports should:

In very special cases it is preferable to create customised reports and the tool should be flexible enough to permit the creation of that report without much effort.

Key points to consider while designing/choosing a log solution:

Interact

The first place to start for an investigation or analysis of any security incidents will be the log tool, its performance and general UI design should be intuitive enough so as to not slow down the operator. The tool must provide the ability to create customized dashboards. Ideally, the dashboards should operate in real-time and provide drill-down capabilities.

Key points to consider while designing/choosing a log solution:

Correlate

A systems administrator was told his servers will soon be outsourced. Three days later, the administrator launches a SSH session to a MySQL server, runs a mysqldump and logs the output on the desktop. Then sends 10 emails to different accounts, each containing 30 lines of what looks like garbage.

Do you want an alert? To start coding this case a lot of information must be collected from different devices, stored long enough and interpreted. The system should maintain a tree of possibilities and react if a branch ends in a case.

The correlation engine depends mainly on rules the product can use. A critical point is to have an easy rule creation process. The search function available for events is also very important, as you will need the ability to search across multiple devices, logs and timeframes.

Key points to consider while designing/choosing a log solution:

Additional points

Additional key points to consider while designing/choosing a log solution:

Costs

Calculating how much a solution will cost is hard. Talks with our customers about SIEM experience have pointed out a minimal model:

Initial costs

Ongoing costs

Occasional costs

Summary

Being tasked with planning a log management solution for your organization can be a bit overwhelming. Approaching the problem with sed/grep/awk, sorting and coloring a .csv file or directly installing a SIEM are all valid used solutions – But starting with the first one can help developing a mature concept and clarify the requests that a SIEM must satisfy.

With time, you will learn to understand your network, how different devices must work togheter to make an application run well, how they are interconnected, from which you need what information and so on. This takes time, but in the meantime you will look deep inside your infrastructure, understand how things work and prevent or resolve problems faster than before.

Once you know what you’re looking for, it will be easier to configure a SIEM, set alerts, correlate events and receive useful alerts.

In fact, before you can perform real-time analysis on all of the logs to detect threats as they occur, you need to capture all of the event data from the plethora of heterogenous event sources and store the logs in a centralized location, so start collecting today, then look inside the log, note how something should work and at the end configure a computer to check if the expectations are met.

Open-source tools

A list of useful open-source tools you can download and install on a Linux machine with relatively low effort. With these tools, you should be able to realise a pretty honest log management solution.

Some older tools you can also consider:

References

{$t:Security log - part 2,$a:rcc,$v:1.0}

About the Author

Rocco Gagliardi

Rocco Gagliardi has been working in IT since the 1980s and specialized in IT security in the 1990s. His main focus lies in security frameworks, network routing, firewalling and log management.

Links

Tags

Basel Dashboard Detect Firewall Framework Graylog Investigation Java Linux Logging NIST PHP Ping Policy Report Request SECO SSH Standard Talk Tool Unix VPN Windows

You want to test the security of your firewall?

Our experts will get in contact with you!

×
Transition to OpenSearch

Transition to OpenSearch

Rocco Gagliardi

Graylog v5

Graylog v5

Rocco Gagliardi

auditd

auditd

Rocco Gagliardi

Security Frameworks

Security Frameworks

Rocco Gagliardi

You want more?

Further articles available here

You need support in such a project?

Our experts will get in contact with you!

You want more?

Further articles available here