mongoDB 2.6.8, 2.4.13 & the upcoming 3.0.0

I’m a bit slacking on those follow-up posts but with the upcoming mongoDB 3.x series and the recent new releases I guess it was about time I talked a bit about what was going on.

 mongodb-3.0.0_rcX

Thanks to the help of Tomas Mozes, we might get a release candidate version of the 3.0.0 version of mongoDB pretty soon in tree shall you want to test it on Gentoo. Feel free to contribute or give feedback in the bug, I’ll do my best to keep up.

What Tomas proposes matches what I had in mind so for now the plan is to :

  • split the mongo tools (mongodump/export etc) to a new package : dev-db/mongo-tools or app-admin/mongo-tools ?
  • split the MMS monitoring agent to its own package : app-admin/mms-monitoring-agent
  • have a look at the MMS backup agent and maybe propose its own package if someone is interested in this ?
  • after the first release, have a look at the MMS deployment automation to see how it could integrate with Gentoo

mongodb-2.6.8 & 2.4.13

Released 2 days ago, they are already on portage ! The 2.4.13 is mostly a security (SSL v3) and tiny backport release whereas the 2.6.8 fixes quite a bunch of bugs.

Please note that I will drop the 2.4.x releases when 3.0.0 hits the tree ! I will keep the latest 2.4.13 in my overlay if someone asks for it.

py3status v2.0

I’m very pleased to announce the release of py3status v2.0 which I’d like to dedicate to the person who’s behind all the nice improvements this release features : @tablet-mode !

His idea on issue #44 was to make py3status modules configurable. After some thoughts and merges of my own plans of development, we ended up with what I believe are the most ambitious features py3status provides so far.

Features

The logic behind this release is that py3status now wraps and extends your i3status.conf which allows all the following crazy features :

For all your i3bar modules i3status and py3status alike thanks to the new on_click parameter which you can use like any other i3status.conf parameter on all modules. It has never been so easy to handle click events !

This is a quick and small example of what it looks like :

# run thunar when I left click on the / disk info module
disk / {
    format = "/ %free"
    on_click 1 = "exec thunar /"
}
  • All py3status contributed modules are now shipped and usable directly without the need to copy them to your local folder. They also get to be configurable directly from your i3status config (see below)

No need to copy and edit the contributed py3status modules you like and wish to use, you can now load and configure them directly from your i3status.conf.

All py3status modules (contributed ones and user loaded ones) are now loaded and ordered using the usual syntax order += in your i3status.conf !

  • All modules have been improved, cleaned up and some of them got some love from contributors.
  • Every click event now triggers a refresh of the clicked module, even for i3status modules. This makes your i3bar more responsive than ever !

Contributors

  • @AdamBSteele
  • @obb
  • @scotte
  • @tablet-mode

Thank you

  • Jakub Jedelsky : py3status is now packaged on Fedora Linux.
  • All of you users : py3status has broken the 100 stars on github, I’m still amazed by this. @Lujeni’s prophecy has come true :)
  • I still have some nice ideas in stock for even more functionalities, stay tuned !

Gentoo Linux PXE builder

Due to a bad hardware failure a few weeks ago at work, I had to rebuild a good part of our PXE stack and I ended up once again looking for the steps to build a PXE-ready Gentoo initramfs.

Then I realized that, while I was at it, I wanted this PXE initramfs to feature more than a Live CD like boot because I use PXE to actually install my servers automatically using ansible. So why not embed all my needs straight into the PXE initramfs and automate the whole boring creation process of it ?

That what the gentoo-pxe-builder project is about and I thought I’d open source it in case it could help and spare some time to anyone else.

The main idea is to provide a simple bash script which bases itself on the latest Gentoo liveCD kernel/initramfs to prepare a PXE suitable version which you can easily hack into without having to handle all the squashfs/cpio hassle to rebuild it.

Quick steps it does for you :

  • download the latest live CD
  • extract the kernel / initramfs from it
  • patch the embedded squashfs to make it PXE ready
  • setup SSH and a default root password so you can connect to your PXE booted machine directly
  • add a hackable local.d start script which will be executed at the end of the PXE boot

The provided local.d start script provides IP address display so you can actually see the IP address being setup on your PXE host and it will also display the real name of the network interfaces detected on the host based on udev deterministic naming.

You can read everything in more details on the project’s README.

Of course it’s mainly oriented to my use case and I’m sure the process / patching could be even more elegant so anyone feel free to contribute or ask/propose some features, I’ll happily follow them up !

RIP ns2

Today we did shutdown our now oldest running Gentoo Linux production server : ns2.

Obviously this machine was happily spreading our DNS records around the world but what’s remarkable about it is that it has been doing so for 2717 straight days !

$ uptime
 13:00:45 up 2717 days,  2:20,  1 user,  load average: 0.13, 0.04, 0.01

As I mentioned when we did shutdown stabber, our beloved firewall, our company has been running Gentoo Linux servers in production for a long time now and we’re always a bit sad when we have to power off one of them.

As usual, I want to take this chance to thank everyone contributing to Gentoo Linux ! Without our collective work, none of this would have been possible.

One month in Turkey

Our latest roadtrip was as amazing as it was challenging because we decided that we’d spend an entire month in Turkey and use our own motorbike to get there from Paris.

Transportation

Our main idea was to spare ourselves from the long hours of road riding to Turkey so we decided from the start to use ferries to get there. Turns out that it’s pretty easy as you have to go through Italy and Greece before you set foot in Bodrum, Turkey.

  • Paris -> Nice : train
  • Nice -> Parma (IT) -> Ancona : road, (~7h drive)
  • Ancona -> Patras (GR) : ferry (21h)
  • Patras -> Piraeus (Athens) : road (~4h drive, constructions)
  • Piraeus -> Kos : ferry (~11h by night)
  • Kos -> Bodrum (TR) : ferry (1h)

Turkish customs are very friendly and polite, it’s really easy to get in with your own vehicle.

Tribute to the Nightster

This roadtrip added 6000 kms to our brave and astonishing Harley-Davidson Nightster. We encountered no problem at all with the bike even though we clearly didn’t go easy on her. We rode on gravels, dirt and mud without her complaining, not to mention the weight of our luggages and the passengers ;)

That’s why this post will be dedicated to our bike and I’ll share some of the photos I took of it during the trip. The real photos will come in some other posts.

A quick photo tour

I can’t describe well enough the pleasure and freedom feeling you get when travelling in motorbike so I hope those first photos will give you an idea.

I have to admit that it’s really impressive to leave your bike alone between the numerous trucks parking, loading/unloading their stuff a few centimeters from it.

IMG_20140905_130004

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We arrived in Piraeus easily, time to buy tickets for the next boat to Kos.

IMG_20140906_164101  IMG_20140906_191845

 

 

 

 

 

 

Kos is quite a big island that you can discover best by … riding around !

IMG_20140907_121148

After Bodrum, where we only spent the night, you quickly discover the true nature of Turkish roads and scenery. Animals are everywhere and sometimes on the road such as those donkeys below.

IMG_20140909_180844

 

 

 

 

 

 

 

This is a view from the Bozburun bay. Two photos for two bike layouts : beach version and fully loaded version ;)

IMG_20140909_191337 IMG_20140910_112858

 

 

 

 

 

 

 

On the way to Cappadocia, near Karapinar :

IMG_20140918_142943

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The amazing landscapes of Cappadocia, after two weeks by the sea it felt cold up there.

IMG_20140920_140433 IMG_20140920_174936 IMG_20140921_130308

 

 

 

 

 

 

 

Our last picture from the bike next to the trail leading to our favorite and lonely “private” beach on the Datça peninsula.

IMG_20140925_182326

 

 

 

 

 

 

 

 

 

 

 

 

py3status v1.6

Back from holidays, this new version of py3status was due for a long time now as it features a lot of great contributions !

This version is dedicated to the amazing @ShadowPrince who contributed 6 new modules :)

Changelog

  • core : rename the ‘examples’ folder to ‘modules’
  • core : Fix include_paths default wrt issue #38, by Frank Haun
  • new vnstat module, by Vasiliy Horbachenko
  • new net_rate module, alternative module for tracking network rate, by Vasiliy Horbachenko
  • new scratchpad-counter module and window-title module for displaying current windows title, by Vasiliy Horbachenko
  • new keyboard-layout module, by Vasiliy Horbachenko
  • new mpd_status module, by Vasiliy Horbachenko
  • new clementine module displaying the current “artist – title” playing in Clementine, by François LASSERRE
  • module clementine.py: Make python3 compatible, by Frank Haun
  • add optional CPU temperature to the sysdata module, by Rayeshman

Contributors

Huge thanks to this release’s contributors :

  • @ChoiZ
  • @fhaun
  • @rayeshman
  • @ShadowPrince

What’s next ?

The next 1.7 release of py3status will bring a neat and cool feature which I’m sure you’ll love, stay tuned !

Using uWSGI and Consul to design a distributed application

Foreword

Let’s say we have to design an application that should span across multiple datacenters while being able to scale as easily as firing up a new vm/container without the need to update any kind of configuration.

Facing this kind of challenge is exciting and requires us to address a few key scaffolding points before actually starting to code something :

  • having a robust and yet versatile application container to run our application
  • having a datacenter aware, fault detecting and service discovery service

Seeing the title of this article, the two components I’ll demonstrate are obviously uWSGI and Consul which can now work together thanks to the uwsgi-consul plugin.

While this article example is written in python, you can benefit from the same features in all the languages supported by uWSGI which includes go, ruby, perl ad php !

Our first service discovering application

The application will demonstrate how simple it is for a client to discover all the available servers running a specific service on a given port. The best part is that the services will be registered and deregistered automatically by uWSGI as they’re loaded and unloaded.

The demo application logic is as follows :

  1. uWSGI will load two server applications which are each responsible for providing the specified service on the given port
  2. uWSGI will automatically register the configured service into Consul
  3. uWSGI will also automatically register a health check for the configured service into Consul so that Consul will also be able to detect any failure of the service
  4. Consul will then respond to any client requesting the list of the available servers (nodes) providing the specified service
  5. The client will query Consul for the service and get either an empty response (no server available / loaded) or the list of the available servers

Et voilà, the client can dynamically detect new/obsolete servers and start working !

Setting up uWSGI and its Consul plugin

On Gentoo Linux, you’ll just have to run the following commands to get started (other users refer to the uWSGI documentation or your distro’s package manager). The plugin will be built by hand as I’m still not sure how I’ll package the uWSGI external plugins…

$ sudo ACCEPT_KEYWORDS="~amd64" emerge uwsgi
$ cd /usr/lib/uwsgi/
$ sudo uwsgi --build-plugin https://github.com/unbit/uwsgi-consul
$ cd -

 

You’ll have installed the uwsgi-consul plugin which you should see here :

$ ls /usr/lib/uwsgi/consul_plugin.so
/usr/lib/uwsgi/consul_plugin.so

 

That’s all we need to have uWSGI working with Consul.

Setting up a Consul server

Gentoo users will need to add the ultrabug overlay (use layman) and then install consul (other users refer to the Consul documentation or your distro’s package manager).

$ sudo layman -a ultrabug
$ sudo ACCEPT_KEYWORDS="~amd64" USE="web" emerge consul

 

Running the server and its UI is also quite straightforward. For this example, we will run it directly from a dedicated terminal so you can also enjoy the logs and see what’s going on (Gentoo users have an init script and conf.d ready for them shall they wish to go further).

Open a new terminal and run :

$ consul agent -data-dir=/tmp/consul-agent -server -bootstrap -ui-dir=/var/lib/consul/ui -client=0.0.0.0

 

You’ll see consul running and waiting for work. You can already enjoy the web UI by pointing your browser to http://127.0.0.1:8500/ui/.

Running the application

To get this example running, we’ll use the uwsgi-consul-demo code that I prepared.

First of all we’ll need the consulate python library (available on pypi via pip). Gentoo users can just install it (also from the ultrabug overlay added before) :

$ sudo ACCEPT_KEYWORDS="~amd64" emerge consulate

 

Now let’s clone the demo repository and get into the project’s directory.

$ git clone git@github.com:ultrabug/uwsgi-consul-demo.git
$ cd uwsgi-consul-demo

 

First, we’ll run the client which should report that no server is available yet. We will keep this terminal open to see the client detecting in real time the appearance and disappearance of the servers as we start and stop uwsgi :

$ python client.py 
no consul-demo-server available
[...]
no consul-demo-server available

 

Open a new terminal and get inside the project’s directory. Let’s have uWSGI load the two servers and register them in Consul :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2
[...]
* server #1 is up on port 2001


* server #2 is up on port 2002

[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully
[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully

 

Now let’s check back our client terminal, hooray it has discovered the two servers on the host named drakar (that’s my local box) !

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001

Expanding our application

Ok it works great on our local machine but we want to see how to add more servers to the fun and scale dynamically.

Let’s add another machine (named cheetah here) to the fun and have servers running there also while our client is still running on our local machine.

On cheetah :

  • install uWSGI as described earlier
  • install Consul as described earlier

Run a Consul agent (no need of a server) and tell him to work with your already running consul server on your box (drakar in my case) :

$ /usr/bin/consul agent -data-dir=/tmp/consul-agent -join drakar -ui-dir=/var/lib/consul/ui -client=0.0.0.0

The -join <your host or IP> is the important part.

 

Now run uWSGI so it starts and registers two new servers on cheetah :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2

 

And check the miracle on your client terminal still running on your local box, the new servers have appeared and will disappear if you stop uwsgi on the cheetah node :

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2001
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2002

Go mad

Check the source code, it’s so simple and efficient you’ll cry ;)

I hope this example has given you some insights and ideas for your current or future application designs !

HD Daymaker LED Headlamp

2014-08-12-064921_571x472_scrot

Short post to share my experience with the Harley-Davidson Daymaker LED Headlamp.

I came to buy it because I was not satisfied with the standard lamp fitted on my sportster and I guess whoever has to drive by night would feel that unpleasant feeling to not actually be able to properly see what’s going on in front of you.

The LED Headlamp is worth the few hundred bucks it costs if at least for the sake of your own life but furthermore for the incredible improvement from the standard lamp. Don’t hesitate a second just go for it and it’s dead simple to mount yourself !

See the difference (passing lights) :

IMG_20140804_215931

IMG_20140804_224657

Now I feel way safer to drive on unlitten roads.

Europython 2014

I had the chance to participate to europython 2014 as my company was sponsoring the event.

IMG_20140725_161445-1024x576

This was a great week where I got to meet some very interesting people and hear about some neat python use cases, libraries and new technologies so I thought I’d write a quick summary of my biased point of view.

ZeroMQ

I had the chance to meet Pieter Hintjens and participate in a 3 hours workshop on ZeroMQ. This was very interesting and refreshing as to go in more depth into this technology which I’ve been using in production for several years now.

Pieter is also quite a philosophical person and I strongly encourage you to listen to his keynote. I ended up pinging him in real life for an issue I’ve been waiting for bug correction on the libzmq and it got answered nicely.

uWSGI

Another big thing in our python stack is the uWSGI application container which I love and follow closely even if my lack of knowledge in C++ prevents me for going too deep in the source code… I got the chance to speak with Roberto De Ioris about the next 2.1 release and propose him two new features.

Thanks a lot for your consideration Roberto !

Trends

  • Not tested = broken !
  • Python is strong and very lively in the Big Data world
  • Asynchronous and distributed architectures get more and more traction and interest

Videos

All the talks videos are already online, you should check them out !

Iran : Yazd

We took our first bus trip to reach Yazd from Shiraz using the Hamsafar company. Booking a bus trip is as easy as it is cheap in Iran so this is by far the best way to get around even tho it’s a bit slow mostly due to the police controls along the road.

Yazd was a shock as it’s a small and beautiful desert town with an unique athmosphere. This city still haunts me and remains my favorite of the trip. The baazar, the covered streets and the mud walls gives you a feeling which is difficult to describe.

We stayed at the very pleasant Orient Hotel and spent one night in a caravanserai where I tried my luck and succeeded to rent a motorcycle for one day ! That was a fun and incredible experience and we’ll always remember the look on the amuzed face of the Iranian people when they realized some tourists where riding a motorcycle among them.

05130001

05090008

05130004

05110008

05130003

05110012

05130008

05090012

05110006

Yazd is of course not only about desert and features some beautiful and peaceful gardens.

05110001

05110002