Monthly Archives: May 2012

State of the event log architecture enhancements

Interesting stuff is happening on the event log (syslog) community and more precisely on the topic of syslog format extension and structuring syslog data.

As of today there’s no real standard on how to format and structure data on a syslog message. Every project has its own log message structure and syntax (qmail and postfix don’t log a mail delivery failure the same way for example), so we rely on parsers to extract any given data from a log message because the syslog software has no way to do it for us. I for one have coded a postfix log parser and believe me it’s not a pleasant thing to do and maintain !

The main idea about structuring syslog messages is to represent them using JSON along with the current free form strings to prevent backward compatibility breakage. To achieve this, we need to normalize and extend this format so that syslog softwares such as rsyslog and syslog-ng can directly understand them. That’s where CEE-enhanced messages and Lumberjack kick in.

CEE-enhanced messages

The CEE project aims at defining a syntax which extends the current log message format while being compatible with all the currently and widely used log frameworks or the well known glibc’s syslog() call. To achieve this the main idea is to use what is called a cookie before the JSON representation of the data we want to pass to the syslog software.

To make it simple, let’s pretend we see this postfix log meaning that a queued mail has been removed from the queue (I removed the date etc to only focus on the message part) :

CAA3B607DA: removed

The equivalent CEE-enhanced message could (this would be up to postfix) be represented as :

@cee: {"id":"CAA3B607DA", "removed":"true"}
  • @cee: is what is called the cookie which tells the syslog software that this message is using the CEE-enhanced syntax

I guess you already see how handy this would be and how we could then rely on the syslog software to automagically use our favorite storage backend to store this structured data (think mongoDB).

More information on the handy and quick video presentation by Rainer Gerhards and his article about it.

The Lumberjack project

So now how do we format the JSON part ? Could we have other types such as booleans and integers directly interpreted by the syslog software ? Well this needs definitions and standardization proposals, that’s what project Lumberjack is for.

Have a nice read on Lumberjack origins on Rainer Gerhards’s blog.

Clustering : corosync v1.4.3 & pacemaker v1.1.7 released

I’ve finally taken the time to take care of the corosync and pacemaker ebuilds. The new versions are now available in portage.

Corosync 1.4.3 (10/04/2012)

This is one of the last supported old stable release of the Corosync Cluster Engine. FYI, I’ve also bumped the new corosync-2.0.0 version but it needs more testing before I hard-unmask it.

Pacemaker 1.1.7 (28/03/12)

This is a bug fix release of Pacemaker. See the changelog for details.

Special thanks to my fellow Gentoo Linux developer Kacper Kowalik (xarthisius) for his help on these bumps.

uWSGI : new ebuild in portage

I started to rework the uwsgi ebuild on March 7th because I was not satisfied with the one available in portage. The current version was out of date and the package itself was not really suited for production deployment.

Luckily my fellow Gentoo Linux developer Tiziano Müller (dev-zero) was also in the same kind of process for his own needs so we teamed up to achieve this goal. Our main focuses were :

  • Bring the emperor mode support
  • Ease and clarify the overall configuration
  • Code a more versatile init script and conf.d file
  • Add a better support of the available plugins and python versions
  • Support PHP

I’m glad to announce that our reworked ebuild is now available in portage for all users, we hope that it will come handy to everyone who needs it.

Thanks again Tiziano, it’s always a pleasure to work with you !

mongoDB : v2.0.5 released

This is a bug fix release of mongoDB, it is now live in portage as well.

+*mongodb-2.0.5 (11 May 2012)
+
+  11 May 2012; Ultrabug <ultrabug@gentoo.org> -mongodb-2.0.3.ebuild,
+  -files/mongodb-2.0.3-fix-scons.patch, +mongodb-2.0.5.ebuild:
+  Version bump, generic mms-agent URL, drop old.
+

Bug fix highlight :

  • Inconsistent query results on large data and result sets
  • Race during static destruction of CommitJob object

See the complete changelog.