Tag Archives: rsyslog

rsyslog v7.6.3

This version bump was long overdue sorry and it has happened only thanks to the great work of Thomas D. aka @Whissi, thanks again mate.

Please read carefully because this version introduces major ebuild changes, you’ll probably have to adapt your current configuration !

ebuild changes

"/var/log/syslog" log file is now deprecated

   Beginning with rsyslog-7.6, the "/var/log/syslog" log file will no
   longer being written per default. We are considering this file as
   deprecated/obsolet for the typical user/system.
   The content from this log file is still availble through other
   (dedicated) log files, see

     - /var/log/cron.log
     - /var/log/daemon.log
     - /var/log/mail.log
     - /var/log/messages

   If you really need the old "/var/log/syslog" log file, all you have to
   do is uncommenting the corresponding configuration directive in
   "/etc/rsyslog.d/50-default.conf".

   If you do so, don't forget to re-enable log rotation in
   "/etc/logrotate.d/rsyslog", too.
  • An additional input socket in /var/empty/dev/log (default chroot
    location) will be created per default
  • brand new and modern init script

rsyslog-7.6.3

Coming from the rsyslog release announcement page, this is what happened with the 7.6 branch release :

With 7.6 being the successor of the 7.5 development branch, everything that has been added there has now found its way into the stable version.

The major additions consist of :
- imrelp/omrelp now support TLS & (zip) compression
- impstats is now emitting resource usage counters, can directly emit delta values and can now be bound to a ruleset
- mmpstrucdata is a new module to parse RFC5424 structured data into JSON message properties
- mmutf8fix is a new module to fix invalid UTF-8 sequences
- mmsequence is a new module that helps with action load balancing
- new defaults for main/ruleset queues to be more enterprise-like

Also the new stable version has undergone a lot of bug fixes, performance improvements and optimizations that make rsyslog 7.6 a lot more reliable and performing than before.

rsyslog v7.4.7

I’ve been slacking on this package for some months but slowly got the chance to catch-up with the multiple bugs open. So today I’m glad to say to make this blog post not only for a version bump release but also for some nice added features and ebuild improvements fixing 6 bugs in a row.

ebuild

  • added support for the mongoDB output template module thanks to Vadim Kuznetsov
  • added support for sub-slot operators for json-c and libgcrypt dependencies using EAPI5 thanks to Thomas D.
  • libgcrypt is now a required dependency of rsyslog as I didn’t want to add another USE flag for such a widely spread dependency
  • I’ve also bumped net-libs/czmq with approval of @jlec which fixes dependencies when trying to build rsyslog with zeromq support, thanks to Allen Parker for his report and valuable debugging

rsyslog v7.4.7

This is a bugfix release, see the full changelog here.

rsyslog : v7.4.4 released

Contribution. This must be what I love the most and one of the more rewarding thing in the Open Source community.

I always found it more natural with Gentoo as you’re really close to the source code so we (devs and users) can see and propose fixes natively to upstream. I known some devs of cool upstreams (mongoDB) do use Gentoo on their developing box because of this close-to-source philosophy.

I don’t know if Rainer does, but I’m glad our contribution (thx to @hwoarang) has been merged to the newer rsyslog. I’m glad to say that we now need no patch at all to build rsyslog on Gentoo !

Long time warning had been issued, I took the opportunity of this bump to clean and drop older and unsupported versions of rsyslog. The only stable branch remaining is v7 from now on. I also made a step towards systemd integration thanks to @slyfox.

highlights

  • make rsyslog use the new json-c pkgconfig file if available. Thanks to the Gentoo team for the patches.
  • bugfix: imfile parameter “persistStateInterval” was unusable due to a case typo in imfile; work-around was to use legacy config
  • bugfix: slightly malformed SMTP handling in ommail
  • bugfix: segfault in omprog if no template was provided (now dflt is used)
  • bugfix: segfault in ompipe if no template was provided (now dflt is used)
  • bugfix: segfault in omsnmp if no template was provided (now dflt is used)
  • bugfix: some omsnmp optional config params were flagged as mandatory

rsyslog : v7.4.3 released

It’s been more than 3 months since the last version bump of rsyslog, I’m sorry about that (special kudos to @Opportunist for his patience). Since June 6th, we have a new stable 7.4 branch of rsyslog containing all the bugfixes and improvements made in the 7.3 dev branch.

So here comes the catch-up to current v7.4.3. Please note that as upstream do not support older branches, I will soon remove them from portage and close related bugs.

highlights

  • tons of bugfixes I won’t bother to list
  • imjournal: add ratelimiting capability
  • max number of templates for plugin use has been increased to five
  • added support for encrypting log files
  • omhiredis: added support for redis pipeline support

rsyslog : new v7 branch released

It’s been a long time since I took care of the rsyslog package so bear with me for these quite huge version releases. The main thing to note is that I finally packaged the new v7 branch which is stamped “stable for production” by upstream. The v5 branch is not supported anymore unless you have a professional contract with Adiscon so I encourage you to read Rainer’s blog post.

 

v7.2.2

Optimizations, code refactoring for way improved performances and a lot of new features are there. The v7 series bring a lot of changes and neat stuff to rsyslog as you can see on this v5 vs v7 link. Here are some hints :

  • Improved configuration language
  • Improved execution engine
  • Full support for structured logging and JSON-based log messages (I discussed this matter on a previous post)
  • Ability to normalize legacy text log messages to JSON

The changelog is very long, so you can browse this to find out more.

v5.10.1 and v6.6.0

Those releases contain backports bugfixes (v5) and enhancements (v6) fixed on the v7 branch so if you’re not ready to jump straight to the new version, you might consider updating your branch for a last time 🙂

rsyslog : new v6 branch in portage

The first ebuild of the v6.2 stable branch of rsyslog is finally available in portage. This branch provides functional and performance enhancements of rsyslog.

Quick highlights :

  • Hadoop (HDFS) support has been considerably speeded up by supporting batched insert mode.
  • TCP transmission overhead for TLS has been dramatically improved.
  • TCP supports input worker thread pools.
  • Support of log normalization via liblognorm rule bases. This permits very high performance normalization of semantically equal messages from different devices (and thus in different syntaxes).

Interesting upcoming features such as mongoDB support and the enhanced config language are on the way with v6.4. Stay tuned !

State of the event log architecture enhancements

Interesting stuff is happening on the event log (syslog) community and more precisely on the topic of syslog format extension and structuring syslog data.

As of today there’s no real standard on how to format and structure data on a syslog message. Every project has its own log message structure and syntax (qmail and postfix don’t log a mail delivery failure the same way for example), so we rely on parsers to extract any given data from a log message because the syslog software has no way to do it for us. I for one have coded a postfix log parser and believe me it’s not a pleasant thing to do and maintain !

The main idea about structuring syslog messages is to represent them using JSON along with the current free form strings to prevent backward compatibility breakage. To achieve this, we need to normalize and extend this format so that syslog softwares such as rsyslog and syslog-ng can directly understand them. That’s where CEE-enhanced messages and Lumberjack kick in.

CEE-enhanced messages

The CEE project aims at defining a syntax which extends the current log message format while being compatible with all the currently and widely used log frameworks or the well known glibc’s syslog() call. To achieve this the main idea is to use what is called a cookie before the JSON representation of the data we want to pass to the syslog software.

To make it simple, let’s pretend we see this postfix log meaning that a queued mail has been removed from the queue (I removed the date etc to only focus on the message part) :

CAA3B607DA: removed

The equivalent CEE-enhanced message could (this would be up to postfix) be represented as :

@cee: {"id":"CAA3B607DA", "removed":"true"}
  • @cee: is what is called the cookie which tells the syslog software that this message is using the CEE-enhanced syntax

I guess you already see how handy this would be and how we could then rely on the syslog software to automagically use our favorite storage backend to store this structured data (think mongoDB).

More information on the handy and quick video presentation by Rainer Gerhards and his article about it.

The Lumberjack project

So now how do we format the JSON part ? Could we have other types such as booleans and integers directly interpreted by the syslog software ? Well this needs definitions and standardization proposals, that’s what project Lumberjack is for.

Have a nice read on Lumberjack origins on Rainer Gerhards’s blog.