Collecting Windows Logs with NXlog

Collecting Windows Logs with NXlog

Andrea Covello
by Andrea Covello
time to read: 14 minutes

Finally, we have got the customer awareness to collect all of a system’s logs as a starting point to get more visibility inside his infrastructure.

But to get there, it’s a long journey indeed. First we’ll set up a log management infrastructure and at the end we might end up with a security monitoring solution that works. But even before that, we need to get all these events coming into our log management solution from several devices such as switches, routers, appliances and generic purposes systems. Of course it would make sense to collect everything with the de-facto standard syslog format that is supported natively by almost all systems. Almost all systems. The minor exception: Windows systems.

Well, in this case (and in many other cases) we have to deal with a large amount of Windows server systems generating lots of events that need to be collected, stored, normalized and analyzed.

So, how do we get all those Windows logs? Well, if you have only Microsoft systems in your network then you may use Windows Remote Management (WinRM) or Windows Management Instrumentation (WMI) solutions. However, we have to work with a heterogeneous environment and therefore we best proceed with syslog.

The Logging Phases

In log management you have to take care of several phases. They are:

Phase Description Details
1 Event Generation The system or application has to be configured/programmed to generate a log entry based on a specific event
2 Event Forwarding/Storing The system generating the event forwards and/or stores it in a defined destination (locally or remotely) using a predefined protocol format
3 Event Collecting A distributed mesh of collecting systems are getting all log events in a central place for further processing (at least two nodes)
4 Event Storing/Archiving All events are stored and/or archived for further processing
5 Event Normalizing All log events are parsed by filters to get a common format through all received events
6 Event Analyzing Events are analyzed to get structured information such as meaningful reports
7 Event Correlation Events are processed through predefined logical rules to get specific notification and alerts

In this article, we are only taking a look at the second phase: Log Forwarding/Storing.

Let me just state one thing before we get into the thick of it: syslog is not the solution to all problems when it comes to log management, but as a transport protocol, it’s doing the right thing to get many systems events together in a common parsing format.

Windows Event Forwarding with NXlog

There are quite a number of syslog converters for Windows, but when it comes to stability and features, NXlog has the best chances to fulfill all the requirements.

We will focus on the Community Edition of NXlog. There is also the Enterprise Edition that comes with all the support requirements for enterprises. However, all the features of NXlog are present in the Community Edition, which sets it apart from other solutions, and I really like that ;)

So let’s take a look at these features:

Feature Note
NXlog is a complete framework It can act as client and/or as server for almost all systems: RedHat/CentOS-, Debian-, Ubuntu-Linux; Windows and Android
Supports TCP and UDP Transport Protocol Default Syslog uses UDP/514 but the fire and forget principle of UDP may not satisfy reliability requirements
Transport Encryption trough SSL Confidentiality requirements may imply an encryption over the line
Easy Deployment low footprint installation, runs as service/daemon
Well documented The manual is very well made and plenty of additional information is available online
Open Source Honestly, do you miss this feature somewhere? ;)
Supports syslog format (RFC3164 and RFC5424) Although still not the best event format out there, syslog always offers compatibility for further processing
Supports structured events format (meta-data structure awareness) NXlog is capable of processing the Windows event log format natively. It reads CSV, JSON, XML, GELF as well as Windows EventLog
Clean and easy configuration You can create a very complex and feature rich configuration. But the basic forwarding configuration is done a few minutes after the installation
Built-in scheduling and log rotation NXlog has a built-in scheduler similar to cron, but with more advanced capabilities to specify timing
No Message Loss NXlog will not drop log messages; it will throttle the input side wherever possible. However, it can be explicitly instructed to drop log messages to avoid possible resource exhaustion
Modular Architecture Dynamically loadable modules (plugins) are available to provide different features and add functionality

Windows Event Format

Windows EventLog allows multi-line messages, so this text is a lot more readable and nicely formatted by spaces, tabs and line-breaks as can be seen in Event Viewer. Here’s the XML view of a Windows Event Log:

XML view of a Windows Event Log

Because syslog only reads/writes single-line messages, this formatting must be stripped of the EventLog message. In doing so, we lose the meta-data.

NXlog is capable of reading these fields, recognize the structure and forward these remotely (or act on them for alerting purposes), thus sparing you time and resources. So, if you use the NXlog framework (client/server) there will be no need to spend time writing patterns to extract usernames, IP addresses and similar meta-data.

If you ask me, NXlog has killer features.

Configuring NXlog Client on a Windows Server

Now let’s take a look at a working NXlog client configuration on a Windows Server 2008/2012 machine.

These are the requirements:

A working NXlog config

NXlog Client Configuration for Windows Server 2008/2012 (nxlog.conf)

Let’s explain the configuration syntax (more information here):

Folder definition in NXlog

The first part defines where the NXlog binaries and modules are to be found and where to write. NXlog has to write its own logs and other relevant information to execute itself

Loading modules in NXlog

Here, NXlog declares what module to load, leveraging the power of modularity to be very efficient.

In this case we need the module to support the syslog export format and the module to get the EventLog format used in Microsoft Windows Vista or later.

There are different modules available; for example, you could use a module to read files (im_file) where events are written. This applies for applications that do not use the Microsoft EventLog API)

Handling of incoming events in NXlog

Here you define how the incoming events should be processed and transformed.

This includes the definition to convert the events in syslog RFC/5424 compliant format

Definition of the transport layer in NXlog

Here we define the transport layer to send the transformed events to the central collector on address x.y.z.1 using TCP on port 10514.

Before going over the network we strip all multiple line and tabs from the message part.

Definition of the working paths in NXlog

In this section we route the processing through our defined path. Basically we say that the input eventlog needs to be processed by p_anco_01 processor and sent to the heron output.

Think of it as an abstraction layer, where we say what has to be done with the event information before going to the target collector.

To recap, NXlog does the following:

A simplified NXlog sequence

Now, this is what a forwarded event message looks like on the receiving syslog server:

Eventmessage in syslog

I’m not going to complain about the time format, as this would require an article on its own, but the format we get is usable by any syslog compatible processing tool.

As you can see, we’ve lost the meta-data format, but this was a requirement and the configuration results are as expected. In this case the receiving process is rsyslogd running on an Ubuntu Linux server that requires a plain RFC3164 for further message processing.

You could of course use an NXlog server process to receive the event messages and keep the meta format, leveraging the embedded NXlog server filtering and analysis capabilities. However, this is not part of this article now as it refers to log-management phases three through seven.

Adding TLS/SSL Encryption

Because log messages transferred over plain TCP can be eavesdropped or even altered with a man-in-the-middle attack, using the im_ssl module provides a secure log message transport.

Make sure you have following prerequisites:

Add the following to your output section in the NXlog configuration

SSL encryption in NXlog

Conclusion

Needless to say, we only scratched the surface of the possibility of the NXlog framework, as already said you’ll find more information here.

I like to stress that this is only a piece of a puzzle that leads to a complete log management solution that sets the foundation of security monitoring.

Still, the bigger effort here will be the framework fine-tuning such as:

Tools like NXlog that work reliably and fulfill the requirements are a prerequisite, but to achieve the final goal of security monitoring there is – in addition to running NXlog – a lot to be done on process and operational level.

However: Enjoy your journey ;)

About the Author

Andrea Covello

Andrea Covello has been working in information security since the 1990s. His strengths are in engineering, specializing in Windows security, firewalling and advanced virtualization.

Links

You want to bring your logging and monitoring to the next level?

Our experts will get in contact with you!

×
Lattice-Based Cryptography

Lattice-Based Cryptography

Noah Berner

OAuth 2.0 Flows

OAuth 2.0 Flows

Ralph Meier

Executable Categorization

Executable Categorization

Rocco Gagliardi

Web Proxy with Kerberos

Web Proxy with Kerberos

Michael Schneider

You want more?

Further articles available here

You need support in such a project?

Our experts will get in contact with you!

You want more?

Further articles available here