Log messages are generated on many system components and applications on said systems. To collect them centrally for management and analysis purposes, transmission over the net is necessary.

It is vital to expose the message and involved components as little as possible in order to assure Confidentiality/Integrity/Availability.

Problems

Log messages are normally transmitted using syslog protocol. By default, syslog uses UDP packets to fire and forget messages from client to one or multiple servers. UDP packets can easily be spoofed, delivery is not guaranteed and information is transmitted in clear text.

Solutions

For many years, syslog daemons offer the possibilities to use TCP as transport protocol. TCP has many advantages over UDP: it is harder to spoof and delivery is guaranteed but information is still transmitted in clear text.

To solve the clear text problem, some syslog daemons offer the possibility to encrypt the message using TLS: just deploy certificates and encrypted transmissions between syslog daemons supporting TCP/TLS will start.

Easy.

Too easy!

Experiences

The solution above requires that all involved syslog daemons:

• support TCP
• support TCP/TLS
• support Generic CA certificates (not proprietary or only self-signed)

The requirements can be satisfied by syslog daemons like rsyslog, syslog-ng, Snare and others. But what’s about a syslog daemon in a switch? Does it support TLS?

If we cannot use TLS, it is reasonable to use at least TCP. A large number of devices offer the option to send syslog using TCP packets. Experience shows that even if a syslog daemon offers TCP support, the server polling feature is not always offered! That means: if the server goes down or isn’t – for any reason – reachable, the client may stop sending messages and may require a restart to start sending messages again!

This behaviour is still hard to monitor and a correction takes time. If you have 100 of these clients you must restart each client.

Practical solution

After many implementations, we ended up with a design balancing the security and the manageability of the systems:

• We prefer to use syslog over UDP for all network devices
• We try to keep the physical path between the UDP client and the syslog proxy as short as possible
• We implement additional mitigations to keep this path as secure as possible (dedicated network/switches/etc.)
• For redundancy, we prefer to use two syslog servers instead of a cluster
• For communication over untrusted paths, we use two rsyslogd with TCP/TLS
• rsyslog can be configured to spool locally in case of forwarding problems
• All communication is ended in a central log collector where each messages is filtered and forwarded to its final destination (local storage, indexing engine, reporting tool, SIEM, …)

The following picture shows the setup:

Keep in mind that many syslog daemons use also the best effort principle, so the generation of messages is not guaranteed.

Summary

syslog is a great protocol but each implementation may be different, supports different features and may create unexpected behaviours. To avoid surprise, use the KISS principle at least for network devices, carefully test the client and use the advanced features only with well know daemons.

About the Author

Rocco Gagliardi has been working in IT since the 1980s and specialized in IT security in the 1990s. His main focus lies in security frameworks, network routing, firewalling and log management.