Security Log Standard - Still an Open Question

Security Log Standard

Still an Open Question

Rocco Gagliardi
by Rocco Gagliardi
time to read: 11 minutes

Keypoints

  • A log standard is still missing
  • IoT urges a common format to exchange security events
  • Log will probably evolve towards a more structured and cryptic machine-oriented formats

Even if you write a small script, you are facing the question: “What and how should I log what is going on?” The problem of event representation, communication, and interpretation consists in identify and transmit useful information, so that the counterpart understands context and content; in the specific case of security information, it is important to determine if a security event can be correlated by different sensors like network, host, or other. This problem is critical if we consider the number and the type of components that need to exchange such information.

Brief history of log formats

If multiple systems observe the same occurrence, it should be expected that their description of that event is identical. When combined with relevant event details (time, source, destination), a computer should be able to immediately determine whether two or more logs, data logs, audit logs, alerts, alarms, or audit trails refer to the same event.

In order to make this happen, we need:

As NIST 800-92, Guide to Computer Security Log Management states “there is no consensus in the security community as to the standard terms to be used to describe the composition of log entries and files.”

Many attempt to address this problem have been started, but we still miss a recognized standard. The following list is not exhaustive; I just picked up some formats I loved or hated during my career, but there are around thousand (~1000) different (syslog) message formats.

Format Type Proposed by Year Status Comment Used by
XDAS Open OpenGroup 1997 Dead XDAS provides several key features like a common audit record format and a standardized audit event taxonomy that allows audit records to be classified in a well-known manner. Unknown
CIDF Open DARPA 1999 Dead The Common Intrusion Detection Framework defined the Common Intrusion Specification Language (CISL), was the base for US Navy log standards. Merged in IDMEF. US Navy
IDMEF Open IETF 2002 Dead The Intrusion Detection Message Exchange Format (RFC4765) is an IETF effort that followed CIDF. IDMEF was designed to enable the communication of intrusion events observed by IDS devices. Snort
CIEL Open MITRE 2002 Dead The “CVE for intrusion detection” indented to provide a naming scheme for all network or host related events. None
SDEE/CIDEE Proprietary ICSA Labs 2003 Dead The Security Device Event Exchange has an XML syntax and SOAP transport. Supported only by Cisco. Cisco
CBE Proprietary IBM, Cisco 2003 Dead The Common Base Event model, is part of the “Autonomic Computing” IBM’s initiative. CBE is described as a “common language to detect, log and resolve system problems”. Tivoli
CIM Open DMTF 2005 Alive The DMTF’s Common Information Model (CIM) provides a common definition of management information for systems, networks, applications and services, and allows for vendor extensions. Microsoft WMI
CEF Proprietary ArcSight 2006 Alive A CEF message is composed of delimited plaintext strings with optional sets of key-value pairs. It is relatively simple to generate and parse, and is transport independent. Many
CEE Open MITRE 2007 Killed Address different areas of the log management: syntax (CLS), transport (CLT), vocabulary (CEET), recommendations (CELR). None
OLF Proprietary eIQNetworks 2007 Dead OLF was designed for logging network events such as those often generated by firewalls, but it can also be used for events not related to the network. Unknown
WELF Proprietary WebTrends 2008 Alive WELF log is composed of records, in chronological order, of 4 mandatory and 20 optional fields, with focus on firewalls and network devices. Many
LEEF Proprietary Q1 Labs 2013 Alive The Log Event Extended Format (LEEF) is a customized event format for IBM Security QRadar. Like CEF, is easy to read and parse. Qradar
CADF Open DMTF 2015 Alive The Cloud Auditing Data Federation (CADF) standard defines a full event model anyone can use to fill in the essential data needed to certify, self-manage and self-audit application security in cloud environments. OPENShift

The MITRE promise and other failures

The MITRE corporation in the early 2000 started a series of projects to address the information exchange problem, focusing on different areas of the log management. They decided to kill CIEL (Common Intrusion Event List) and created the CEE (Common Event Expression) Framework, that covers transport (Common Log Transport), syntax (Common Log Syntax), taxonomy (Common Event Expression Taxonomy), and a set of recommendations on when and what log (Common Event Log Recommendations), as part of the plan to build a national cyber information sharing ecosystem composed by CVE, CWE, CAPEC, ATT&CK, and CAR. But in 2014 someone decided that the CEE was no longer a priority.

As result, we have dictionaries, the words to identify the different parts of the cyber-security puzzle, but we miss the glue: we still use syslog to transmit the analysis produced by cyber security systems like IDS/IPS/EPS/++, and we still must ad-hoc grok the message field.

Why is so hard to define a log format? Why have so many initiatives failed?

It is not hard to define a standard! Sure, some proposal was overkill (IDMF), some other too complex to implement, but basically all standards proposed a solution for a problem not recognized by developers. Remember the "xennet: skb rides the rocket: 19 slots" message flooding /var/log/syslog for “some reason”! For a coder who creates such meaningful message, all the efforts to standardize content are just waste of time.

The future

Still in 2018 we don’t have a standard for security log messages! We have syslog somehow recognized as “universal” transport, that’s all.

The message field remain a land of conquest!

CEF, LEEF, CIM/CADF are the most used and supported formats:

Format Transport Encoding Structure Num of fields Extensibility Remarks
CEF syslog k→v Loose, no dictionary 91 Some additional fields Easy to read and parse, lot of information
LEEF syslog k→v Loose, no dictionary 51 Not specified Easy to read and parse, focus on network
CIM agnostic agnostic XML schema 59 New instances Easy to parse. Structures describing security events should be enhanced.

At the moment, the winner is one with a loose format, transmitted over syslog, readable by human, and parsable by code. But the IoT will produce most of the messages for a code, not for a user! So, we will probably see more structured and cryptic machine-oriented formats, compressed, and composed more by IDs and less by words.

My fear is that, without a central authority, we will have a lot of git-wanna-be-master-clones.

Summary

Especially with the IoT knocking at the door, it is urgent to have a log standard to facilitate the correlation and identification of abnormal behaviors.

Many organization tried to define them, but without success. In the last years, support for CEE and LEEF transported over syslog established a de facto standard, but they are still archaic, big, user oriented logs, in a world demanding more compact, agile, machine centric logs.

About the Author

Rocco Gagliardi

Rocco Gagliardi has been working in IT since the 1980s and specialized in IT security in the 1990s. His main focus lies in security frameworks, network routing, firewalling and log management.

Links

Tags

AD Anonym API ATT&CK Cisco Cloud CVE Detect Exchange Firewall Framework HTML IBM IETF IoT Live Logging Master Microsoft MITRE NIST Standard Tool Word XML

You want to test the security of your firewall?

Our experts will get in contact with you!

×
Transition to OpenSearch

Transition to OpenSearch

Rocco Gagliardi

Graylog v5

Graylog v5

Rocco Gagliardi

auditd

auditd

Rocco Gagliardi

Security Frameworks

Security Frameworks

Rocco Gagliardi

You want more?

Further articles available here

You need support in such a project?

Our experts will get in contact with you!

You want more?

Further articles available here