Monthly Archives: January 2014

keepalived v1.2.11 & glusterfs v3.4.2

Quick post for two quick bumps related to clustering.

glusterfs-3.4.2

  • quite a lot of bug fixes and improvements
  • contains a backport for libgfapi support for integrating with NFS Ganesha
  • nfs/mount3: fix crash in subdir resolution

keepalived-1.2.11

  • autoconf: better libnl3 detection
  • Fix memory allocation for MD5 digest
  • Quite some nice memory leak fixes on different components
  • vrrp: dont try to load ip_vs module when not needed
  • Pim van den Berg work on libipvs-2.6 to sync with libipvs from ipvsadm 1.27
  • vrrp: extend ip parser to support default and default6
  • vrrp: fix/extend gratuitous ARP handling (multiple people reported issues where MASTER didnt recover properly after outage due to no gratuitous ARP sent)
  • Multiple fixes to genhash
  • vrrp: fix vrrp socket sync while leaving FAULT state (old old bug here)
  • Full changelog here

uWSGI v2.0

Yesterday was a big day for the famous application container uWSGI. We released the brand new version 2.0 LTS along with quite a huge bump of the ebuild, closing 6 bugs at once. I thought I’d give some input about the ebuild changes and some quick notes about uWSGI. Many thanks again to @dev-zero !

New plugins selection : UWSGI_PLUGINS

We introduced a new USE_EXPAND named UWSGI_PLUGINS so that you can now select which plugins to build individually. This is a great step as it makes the compilation more clear and lets you fine tune your uWSGI installation.

Along this work, we had to describe each plugin which was also quite a challenge. To my knownledge, this has not been done anywhere else so here it is. Please ping me if you have something to add or if we failed to describe a plugin correctly.

Migration note : You will need to change your package.use configuration to switch to using UWSGI_PLUGINS. As an example, where you had the USE flag spooler enabled you’ll now need to use uwsgi_plugins_spooler.

uWSGI v2.0 highlights

These are my biased favorites, go check for more, it’s huge !

py3status v1.2

I’m glad to announce a new release of py3status with an exciting main new feature giving the ability to modify any of i3status’ module output from any of your modules !

feature

changelog

  • new module dpms.py allowing activation and deactivation of DPMS thx to André Doser
  • order i3status output updates to prevent it from overwritting any modification made on i3status json list by a user module, this avoids a possible user filter flapping on i3status modules
  • fix delay on first execution of each module which could be equal to py3status interval time before being executed : your modules get executed and displayed immediately no matter py3status’ interval
  • the real i3status thread output json list is passed to all modules as the i3status_output_json parameter, this allows any user module to change any of the i3status output by simply altering the given json on the list, inspired thx to @drestebon on issue #23
  • add validation for the position parameter
  • add cpu usage info to sysdata script, by Patrick Shan

contributors

Many thanks to all contributors for their work and inspiration.

  • Patrick Shan, @patrickshan
  • @drestebon
  • André Doser, @tasse

Tuning pacemaker for large clusters

We’ve been running quite a lot of production clusters using pacemaker/corosync for a while. Some of them are large, handling more than 200 resources across multiple nodes and we’ve exceeded some limits on pacemaker’s CIB size.

I thought I’d share how to tune your cluster to handle such a bunch of resources since there are some default limits on the IPC buffer size which can lead to problems when your resources (and thus CIB) grows too much.

Hitting the IPC limit

When running a large cluster you may hit the following problem :

error: crm_ipc_prepare: Could not compress the message into less than the configured ipc limit (51200 bytes).Set PCMK_ipc_buffer to a higher value (2071644 bytes suggested)

Evaluating the buffer size

Have a look at the size of your current CIB :

# cibadmin -Ql > cib.xml
# ls -l cib.xml
# bzip2 cib.xml
# ls -l cib.xml.bz2

The CIB is compressed on the wire using bzip2 so you have to compare the compressed cib.xml.bz2 with the IPC default buffer size of 51200 and you’ll find the sufficient PCMK_ipc_buffer value for you (take more just to be safe).

Setting the environment variables

On Gentoo Linux, you’ll have to create the /etc/env.d/90pacemaker file containing :

PCMK_ipc_type=shared-mem
PCMK_ipc_buffer=2071644
  • PCMK_ipc_buffer : you may need to increase this depending on your cluster size and needs
  • PCMK_ipc_type : the shared-mem one is the default now, other values are socket|posix|sysv

You will also need to set these env. vars in your .bashrc so that the crm CLI doesn’t break :

export PCMK_ipc_type=shared-mem
export PCMK_ipc_buffer=2071644

Future

Finally, I wanted to let you know that the upcoming Pacemaker v1.1.11 should come with a feature which will allow the IPC layer to adjust the PCMK_ipc_buffer automagically !

Hopefully you shouldn’t need this blog post anymore pretty soon 🙂

EDIT, Jan 16 2014

Following this blog post, I had a very interesting comment from @beekhof (lead dev of pacemaker)

beekhof> Ultrabug: regarding large clusters, the cib in 1.1.12 will be O(2) faster than 1.1.11.
Ultrabug> beekhof: that's great news mate ! when is it scheduled to be released ?
beekhof> 30th of Feb

Aircraft Boneyard

After Arcosanti, we kept on going south into Arizona where we headed to Phoenix and stayed in Tuscon for a few days. It was hot and sunny of course and we loved this part of Arizona.

We went to see the US Army Aircraft Boneyard which is located a bit South-East of Tuscon, in the middle of the desert. Imagine hundreds of planes parked in the sand, waiting to be dismantled !

_0006_0005 _0004