Finally, we have got the customer awareness to collect all of a system’s logs as a starting point to get more visibility inside his infrastructure.
But to get there, it’s a long journey indeed. First we’ll set up a log management infrastructure and at the end we might end up with a security monitoring solution that works. But even before that, we need to get all these events coming into our log management solution from several devices such as switches, routers, appliances and generic purposes systems. Of course it would make sense to collect everything with the de-facto standard syslog format that is supported natively by almost all systems. Almost all systems. The minor exception: Windows systems.
Well, in this case (and in many other cases) we have to deal with a large amount of Windows server systems generating lots of events that need to be collected, stored, normalized and analyzed.
So, how do we get all those Windows logs? Well, if you have only Microsoft systems in your network then you may use Windows Remote Management (WinRM) or Windows Management Instrumentation (WMI) solutions. However, we have to work with a heterogeneous environment and therefore we best proceed with syslog.
In log management you have to take care of several phases. They are:
|1||Event Generation||The system or application has to be configured/programmed to generate a log entry based on a specific event|
|2||Event Forwarding/Storing||The system generating the event forwards and/or stores it in a defined destination (locally or remotely) using a predefined protocol format|
|3||Event Collecting||A distributed mesh of collecting systems are getting all log events in a central place for further processing (at least two nodes)|
|4||Event Storing/Archiving||All events are stored and/or archived for further processing|
|5||Event Normalizing||All log events are parsed by filters to get a common format through all received events|
|6||Event Analyzing||Events are analyzed to get structured information such as meaningful reports|
|7||Event Correlation||Events are processed through predefined logical rules to get specific notification and alerts|
In this article, we are only taking a look at the second phase: Log Forwarding/Storing.
Let me just state one thing before we get into the thick of it: syslog is not the solution to all problems when it comes to log management, but as a transport protocol, it’s doing the right thing to get many systems events together in a common parsing format.
There are quite a number of syslog converters for Windows, but when it comes to stability and features, NXlog has the best chances to fulfill all the requirements.
We will focus on the Community Edition of NXlog. There is also the Enterprise Edition that comes with all the support requirements for enterprises. However, all the features of NXlog are present in the Community Edition, which sets it apart from other solutions, and I really like that ;)
So let’s take a look at these features:
|NXlog is a complete framework||It can act as client and/or as server for almost all systems: RedHat/CentOS-, Debian-, Ubuntu-Linux; Windows and Android|
|Supports TCP and UDP Transport Protocol||Default Syslog uses UDP/514 but the fire and forget principle of UDP may not satisfy reliability requirements|
|Transport Encryption trough SSL||Confidentiality requirements may imply an encryption over the line|
|Easy Deployment||low footprint installation, runs as service/daemon|
|Well documented||The manual is very well made and plenty of additional information is available online|
|Open Source||Honestly, do you miss this feature somewhere? ;)|
|Supports syslog format (RFC3164 and RFC5424)||Although still not the best event format out there, syslog always offers compatibility for further processing|
|Supports structured events format (meta-data structure awareness)||NXlog is capable of processing the Windows event log format natively. It reads CSV, JSON, XML, GELF as well as Windows EventLog|
|Clean and easy configuration||You can create a very complex and feature rich configuration. But the basic forwarding configuration is done a few minutes after the installation|
|Built-in scheduling and log rotation||NXlog has a built-in scheduler similar to cron, but with more advanced capabilities to specify timing|
|No Message Loss||NXlog will not drop log messages; it will throttle the input side wherever possible. However, it can be explicitly instructed to drop log messages to avoid possible resource exhaustion|
|Modular Architecture||Dynamically loadable modules (plugins) are available to provide different features and add functionality|
Windows EventLog allows multi-line messages, so this text is a lot more readable and nicely formatted by spaces, tabs and line-breaks as can be seen in Event Viewer. Here’s the XML view of a Windows Event Log:
Because syslog only reads/writes single-line messages, this formatting must be stripped of the EventLog message. In doing so, we lose the meta-data.
NXlog is capable of reading these fields, recognize the structure and forward these remotely (or act on them for alerting purposes), thus sparing you time and resources. So, if you use the NXlog framework (client/server) there will be no need to spend time writing patterns to extract usernames, IP addresses and similar meta-data.
If you ask me, NXlog has killer features.
Now let’s take a look at a working NXlog client configuration on a Windows Server 2008/2012 machine.
These are the requirements:
Let’s explain the configuration syntax (more information here):
The first part defines where the NXlog binaries and modules are to be found and where to write. NXlog has to write its own logs and other relevant information to execute itself
Here, NXlog declares what module to load, leveraging the power of modularity to be very efficient.
In this case we need the module to support the syslog export format and the module to get the EventLog format used in Microsoft Windows Vista or later.
There are different modules available; for example, you could use a module to read files (
im_file) where events are written. This applies for applications that do not use the Microsoft EventLog API)
Here you define how the incoming events should be processed and transformed.
This includes the definition to convert the events in syslog RFC/5424 compliant format
Here we define the transport layer to send the transformed events to the central collector on address x.y.z.1 using TCP on port 10514.
Before going over the network we strip all multiple line and tabs from the message part.
In this section we route the processing through our defined path.
Basically we say that the input
eventlog needs to be processed by
p_anco_01 processor and sent to the
Think of it as an abstraction layer, where we say what has to be done with the event information before going to the target collector.
To recap, NXlog does the following:
im_stands for input module)
pm_stands for processing module)
om_stands for output module)
Now, this is what a forwarded event message looks like on the receiving syslog server:
I’m not going to complain about the time format, as this would require an article on its own, but the format we get is usable by any syslog compatible processing tool.
As you can see, we’ve lost the meta-data format, but this was a requirement and the configuration results are as expected. In this case the receiving process is
rsyslogd running on an Ubuntu Linux server that requires a plain RFC3164 for further message processing.
You could of course use an NXlog server process to receive the event messages and keep the meta format, leveraging the embedded NXlog server filtering and analysis capabilities. However, this is not part of this article now as it refers to log-management phases three through seven.
Because log messages transferred over plain TCP can be eavesdropped or even altered with a man-in-the-middle attack, using the
im_ssl module provides a secure log message transport.
Make sure you have following prerequisites:
Add the following to your output section in the NXlog configuration
%CERTDIR%as directory where your certificates are stored at the beginning of the
nxlog.conf(the definition section).
om_tcpmodule with the
om_sslmodule. See commented line in the code above.
Needless to say, we only scratched the surface of the possibility of the NXlog framework, as already said you’ll find more information here.
I like to stress that this is only a piece of a puzzle that leads to a complete log management solution that sets the foundation of security monitoring.
Still, the bigger effort here will be the framework fine-tuning such as:
Tools like NXlog that work reliably and fulfill the requirements are a prerequisite, but to achieve the final goal of security monitoring there is – in addition to running NXlog – a lot to be done on process and operational level.
However: Enjoy your journey ;)
Our experts will get in contact with you!
Our experts will get in contact with you!
Further articles available here