Fine tuning log routing (osc25)

Chaos Computer Club - recent events feed (high quality)

Do you want to detect threats real-time and easily search logs to find relevant information? By default, incoming log messages are treated in log management apps as one long string. This is good enough if you just want to store logs. However, if you also want to act on log messages, you need more structured information. Syslog-ng and other applications also let you parse log messages. While parsing structured logs, like JSON, is easy, unstructured logs need a lot of work. This is where sequence can help you: if you have enough logs to analyze, it can automatically generate parser rules for syslog-ng and LogStash. Name-value pairs (sometimes also called “macros”) are probably one of the most useful parts of syslog-ng. Macros have been a core feature of syslog-ng since the beginning: incoming log messages were parsed by syslog-ng and various fields (priority, facility, date, program, PID and message) were stored into name-value pairs. You could use these to reformat log messages or use these values in file names. Without additional parsing, syslog-ng treats the message part of a log message as one long string. While this might be valid in some use cases, many log messages look like an almost complete English sentence with some variable parts in it. Just think about SSH login messages: they include the username, the source IP and port, and the login method embedded in a sentence. You might want to create an alert in syslog-ng for such messages (for example, if a root user logs in). However, if the whole message is a single string, you cannot do that. PatternDB can find important information in unstructured log messages, like the above-mentioned SSH login message, and create name-value pairs from the information it finds. PatternDB rules are easier to write than regular expressions and also need less resources to run. There are various parsers for structured log messages as well, like the CSV, JSON and XML parsers. You can also combine these and build really complex parsers. Sometimes, the syslog header might be missing from log messages, so you need to create a parser for the whole message. Message parsing and name-value pairs give you a lot more flexibility when it comes to filtering or templating log messages. For example, you can send an alert to Slack when someone logged in as a root user through SSH, or you can forward two important fields from an extremely long log message to save network bandwidth and disk space. You can use it to hide sensitive information and send only what is required to various destinations. Sequence-RTG allows you to automatically generate parser rules for syslog-ng and LogStash. Creating parser rules from scratch is a painful process. However, if you have enough log messages to analyze, sequence can do the heavy lifting for you. Of course, you might need to rename some of the name-value pairs, but that is a lot easier. Licensed to the public under https://creativecommons.org/licenses/by-sa/4.0/ about this event: https://c3voc.de