Using uWSGI and Consul to design a distributed application

Foreword

Let’s say we have to design an application that should span across multiple datacenters while being able to scale as easily as firing up a new vm/container without the need to update any kind of configuration.

Facing this kind of challenge is exciting and requires us to address a few key scaffolding points before actually starting to code something :

  • having a robust and yet versatile application container to run our application
  • having a datacenter aware, fault detecting and service discovery service

Seeing the title of this article, the two components I’ll demonstrate are obviously uWSGI and Consul which can now work together thanks to the uwsgi-consul plugin.

While this article example is written in python, you can benefit from the same features in all the languages supported by uWSGI which includes go, ruby, perl ad php !

Our first service discovering application

The application will demonstrate how simple it is for a client to discover all the available servers running a specific service on a given port. The best part is that the services will be registered and deregistered automatically by uWSGI as they’re loaded and unloaded.

The demo application logic is as follows :

  1. uWSGI will load two server applications which are each responsible for providing the specified service on the given port
  2. uWSGI will automatically register the configured service into Consul
  3. uWSGI will also automatically register a health check for the configured service into Consul so that Consul will also be able to detect any failure of the service
  4. Consul will then respond to any client requesting the list of the available servers (nodes) providing the specified service
  5. The client will query Consul for the service and get either an empty response (no server available / loaded) or the list of the available servers

Et voilà, the client can dynamically detect new/obsolete servers and start working !

Setting up uWSGI and its Consul plugin

On Gentoo Linux, you’ll just have to run the following commands to get started (other users refer to the uWSGI documentation or your distro’s package manager). The plugin will be built by hand as I’m still not sure how I’ll package the uWSGI external plugins…

$ sudo ACCEPT_KEYWORDS="~amd64" emerge uwsgi
$ cd /usr/lib/uwsgi/
$ sudo uwsgi --build-plugin https://github.com/unbit/uwsgi-consul
$ cd -

 

You’ll have installed the uwsgi-consul plugin which you should see here :

$ ls /usr/lib/uwsgi/consul_plugin.so
/usr/lib/uwsgi/consul_plugin.so

 

That’s all we need to have uWSGI working with Consul.

Setting up a Consul server

Gentoo users will need to add the ultrabug overlay (use layman) and then install consul (other users refer to the Consul documentation or your distro’s package manager).

$ sudo layman -a ultrabug
$ sudo ACCEPT_KEYWORDS="~amd64" USE="web" emerge consul

 

Running the server and its UI is also quite straightforward. For this example, we will run it directly from a dedicated terminal so you can also enjoy the logs and see what’s going on (Gentoo users have an init script and conf.d ready for them shall they wish to go further).

Open a new terminal and run :

$ consul agent -data-dir=/tmp/consul-agent -server -bootstrap -ui-dir=/var/lib/consul/ui -client=0.0.0.0

 

You’ll see consul running and waiting for work. You can already enjoy the web UI by pointing your browser to http://127.0.0.1:8500/ui/.

Running the application

To get this example running, we’ll use the uwsgi-consul-demo code that I prepared.

First of all we’ll need the consulate python library (available on pypi via pip). Gentoo users can just install it (also from the ultrabug overlay added before) :

$ sudo ACCEPT_KEYWORDS="~amd64" emerge consulate

 

Now let’s clone the demo repository and get into the project’s directory.

$ git clone git@github.com:ultrabug/uwsgi-consul-demo.git
$ cd uwsgi-consul-demo

 

First, we’ll run the client which should report that no server is available yet. We will keep this terminal open to see the client detecting in real time the appearance and disappearance of the servers as we start and stop uwsgi :

$ python client.py 
no consul-demo-server available
[...]
no consul-demo-server available

 

Open a new terminal and get inside the project’s directory. Let’s have uWSGI load the two servers and register them in Consul :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2
[...]
* server #1 is up on port 2001


* server #2 is up on port 2002

[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully
[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully

 

Now let’s check back our client terminal, hooray it has discovered the two servers on the host named drakar (that’s my local box) !

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001

Expanding our application

Ok it works great on our local machine but we want to see how to add more servers to the fun and scale dynamically.

Let’s add another machine (named cheetah here) to the fun and have servers running there also while our client is still running on our local machine.

On cheetah :

  • install uWSGI as described earlier
  • install Consul as described earlier

Run a Consul agent (no need of a server) and tell him to work with your already running consul server on your box (drakar in my case) :

$ /usr/bin/consul agent -data-dir=/tmp/consul-agent -join drakar -ui-dir=/var/lib/consul/ui -client=0.0.0.0

The -join <your host or IP> is the important part.

 

Now run uWSGI so it starts and registers two new servers on cheetah :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2

 

And check the miracle on your client terminal still running on your local box, the new servers have appeared and will disappear if you stop uwsgi on the cheetah node :

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2001
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2002

Go mad

Check the source code, it’s so simple and efficient you’ll cry ;)

I hope this example has given you some insights and ideas for your current or future application designs !

HD Daymaker LED Headlamp

2014-08-12-064921_571x472_scrot

Short post to share my experience with the Harley-Davidson Daymaker LED Headlamp.

I came to buy it because I was not satisfied with the standard lamp fitted on my sportster and I guess whoever has to drive by night would feel that unpleasant feeling to not actually be able to properly see what’s going on in front of you.

The LED Headlamp is worth the few hundred bucks it costs if at least for the sake of your own life but furthermore for the incredible improvement from the standard lamp. Don’t hesitate a second just go for it and it’s dead simple to mount yourself !

See the difference (passing lights) :

IMG_20140804_215931

IMG_20140804_224657

Now I feel way safer to drive on unlitten roads.

Europython 2014

I had the chance to participate to europython 2014 as my company was sponsoring the event.

IMG_20140725_161445-1024x576

This was a great week where I got to meet some very interesting people and hear about some neat python use cases, libraries and new technologies so I thought I’d write a quick summary of my biased point of view.

ZeroMQ

I had the chance to meet Pieter Hintjens and participate in a 3 hours workshop on ZeroMQ. This was very interesting and refreshing as to go in more depth into this technology which I’ve been using in production for several years now.

Pieter is also quite a philosophical person and I strongly encourage you to listen to his keynote. I ended up pinging him in real life for an issue I’ve been waiting for bug correction on the libzmq and it got answered nicely.

uWSGI

Another big thing in our python stack is the uWSGI application container which I love and follow closely even if my lack of knowledge in C++ prevents me for going too deep in the source code… I got the chance to speak with Roberto De Ioris about the next 2.1 release and propose him two new features.

Thanks a lot for your consideration Roberto !

Trends

  • Not tested = broken !
  • Python is strong and very lively in the Big Data world
  • Asynchronous and distributed architectures get more and more traction and interest

Videos

All the talks videos are already online, you should check them out !

Iran : Yazd

We took our first bus trip to reach Yazd from Shiraz using the Hamsafar company. Booking a bus trip is as easy as it is cheap in Iran so this is by far the best way to get around even tho it’s a bit slow mostly due to the police controls along the road.

Yazd was a shock as it’s a small and beautiful desert town with an unique athmosphere. This city still haunts me and remains my favorite of the trip. The baazar, the covered streets and the mud walls gives you a feeling which is difficult to describe.

We stayed at the very pleasant Orient Hotel and spent one night in a caravanserai where I tried my luck and succeeded to rent a motorcycle for one day ! That was a fun and incredible experience and we’ll always remember the look on the amuzed face of the Iranian people when they realized some tourists where riding a motorcycle among them.

05130001

05090008

05130004

05110008

05130003

05110012

05130008

05090012

05110006

Yazd is of course not only about desert and features some beautiful and peaceful gardens.

05110001

05110002

Consul on Gentoo Linux

As a clustering and distributed architecture enthusiast, I’m naturally interested in software providing neat ways to coordinate any kind of state/configuration/you-name-it over a large number of machines.

My quest, as many of you I guess, were so far limited to tools like zookeeper (packaged on my overlay but with almost no echo) and doozerd (last commit nearly 6 months ago) which both cover some of the goals listed above with more or less flavors and elegance (sorry guys, JAVA is NOT elegant to me).

I recently heard about consul, a new attempt to solve some of those problems in an interesting way while providing some rich fuctionnalities so I went on giving it a try and naturally started packaging it so others can too.

WTF is consul ?

consul logo

Consul is a few months’ old project (and already available on Gentoo !) from the guys making Vagrant. I especially like its datacenter centric architecture, intuitive deployment and its DNS + HTTP API query mecanisms. This sounds promising so far !

This is a descripion taken from the Hashicorp’s blog :

Consul is a solution for service discovery and configuration. Consul is completely distributed, highly available, and scales to thousands of nodes and services across multiple datacenters.

Some concrete problems Consul solves: finding the services applications need (database, queue, mail server, etc.), configuring services with key/value information such as enabling maintenance mode for a web application, and health checking services so that unhealthy services aren’t used. These are just a handful of important problems Consul addresses.

Consul solves the problem of service discovery and configuration. Built on top of a foundation of rigorous academic research, Consul keeps your data safe and works with the largest of infrastructures. Consul embraces modern practices and is friendly to existing DevOps tooling.

app-admin/consul ?

This is a RFC and interest call about the packaging and availability of consul for Gentoo Linux.

The latest version and live ebuilds are present in my overlay so if you are interested, please tell me (here, IRC, email, whatever) and I’ll consider adding it to the portage tree.

I want to test it !

Now that would be helpful to get some feedback about the usability of the current packaging. So far the ebuild features what I think should cover a lot of use cases :

  • full build from sources
  • customizable consul agent init script with reload, telemetry and graceful stop support
  • web UI built from sources and installation for easy deployment
# layman -a ultrabug
# emerge -av consul

Hope this interests some of you folks !

uWSGI v2.0.5.1

This release is important to me (and my company) as it officially introduces a few features we developed for our needs and then contributed to uWSGI.

Special congratulations to my co-worker @btall for his first contribution and for those nice features to the metrics subsystem with many thanks as usual to @unbit for reviewing and merging them so quickly.

new features

  • graceful reload of mule processes (Credits: Paul Egan) : SIGHUP is now sent to mules instead of directly killing them, by default you have 60 seconds to react before a SIGKILL
  • –metrics-no-cores, –stats-no-cores, –stats-no-metrics : don’t calculate and process all those core related metrics (gevent anyone ?)
  • reset_after_push for metrics (Credits: Babacar Tall) : this metric attribute ensures that the metric value is reset to 0 or its hardcoded initial_value every time the metric is pushed to some external system (like carbon, or statsd)
  • new metric_set_max and metric_set_min helpers (Credits: Babacar Tall) : can be used to avoid having to call “metric_get“ when you need a metric to be set at a maximal or minimal value. Another simple use case is to use the “avg“ collector to calculate an average between some *max* and *min* set metrics. Available in C and python.

See the full changelog here, especially some interesting bugfixes.

Iran : Shiraz

Our Iranian friend booked us a night train from Tehran to our next stop in the South of Iran : Shiraz.

The Tehran train station is well organized and as foreigners we had to show our passports to the station’s police where we were greeted with the big smile by the policemen. The train itself was good, clean, comfortable and on schedule even if it was quite slow : departure at 20h and we arrived at 11h30 a few kilometers up from Shiraz.

Travelling by train is always a great experience. As we woke up early in the morning, this is what was waiting for us behind the curtains :

05080012

Compared to Tehran, the streets are more typical and fitting what you’d expect from a desert city. That’s the exact kind of town you want to get lost in, just walking around and going into its nice and large bazaar.

05120002

05040005

05120005

05120006

Shiraz is known for its many palaces and beautiful gardens.

05120001

05120012

05090004

05090002

This city is also the usual starting point to visit the famous Persepolis. While the place is great, I must admit that we loved the abandonned park right next to it which is used as a playground and food camping by the schools and other Iranian tourists.

05120009

05120011

rsyslog v7.6.3

This version bump was long overdue sorry and it has happened only thanks to the great work of Thomas D. aka @Whissi, thanks again mate.

Please read carefully because this version introduces major ebuild changes, you’ll probably have to adapt your current configuration !

ebuild changes

"/var/log/syslog" log file is now deprecated

   Beginning with rsyslog-7.6, the "/var/log/syslog" log file will no
   longer being written per default. We are considering this file as
   deprecated/obsolet for the typical user/system.
   The content from this log file is still availble through other
   (dedicated) log files, see

     - /var/log/cron.log
     - /var/log/daemon.log
     - /var/log/mail.log
     - /var/log/messages

   If you really need the old "/var/log/syslog" log file, all you have to
   do is uncommenting the corresponding configuration directive in
   "/etc/rsyslog.d/50-default.conf".

   If you do so, don't forget to re-enable log rotation in
   "/etc/logrotate.d/rsyslog", too.
  • An additional input socket in /var/empty/dev/log (default chroot
    location) will be created per default
  • brand new and modern init script

rsyslog-7.6.3

Coming from the rsyslog release announcement page, this is what happened with the 7.6 branch release :

With 7.6 being the successor of the 7.5 development branch, everything that has been added there has now found its way into the stable version.

The major additions consist of :
- imrelp/omrelp now support TLS & (zip) compression
- impstats is now emitting resource usage counters, can directly emit delta values and can now be bound to a ruleset
- mmpstrucdata is a new module to parse RFC5424 structured data into JSON message properties
- mmutf8fix is a new module to fix invalid UTF-8 sequences
- mmsequence is a new module that helps with action load balancing
- new defaults for main/ruleset queues to be more enterprise-like

Also the new stable version has undergone a lot of bug fixes, performance improvements and optimizations that make rsyslog 7.6 a lot more reliable and performing than before.

Iran : Tehran

For our latest vacations, we spent two weeks in Iran and I must say that this country is astonishingly beautiful ! So forget what you think to know about this country as it’s either biased or doesn’t apply to its people who are very friendly and welcoming.

We started with two days in Tehran, a gigantic and lovely city where an Iranian friend we met on the internet invited us to a treck in the surroundings montains.

The first thing to have to get acquainted with is how to cross the streets. Seriously the traffic is madness and uncontrolled (no traffic lights) so you have to watch out everywhere. Then you discover their nice and cooled subway which is clean and shiny and so cheap you try and count four times as you’re sure you’ve heard the man wrong : 10 000 rials a ticket, yeah that’s like 0,2€…

06340006

06340008

Treck startpoint was Darban and then up to close to 3000m, thanks again Shayan. When we were awed by the beautiful mountains they had at the doorstep or their city, our Iranian friends told us that these mountains are also the cause of the massive pollution of Tehran in sping and summer because it blocks the winds from clearing the air…

05080008

05080011

05080010

Sunset from the Jamshidieh park, the tall tower on the background is the Milad Tower.

05060012

mongoDB v2.6.1

This is a great pleasure to announce the version bump of mongoDB to the brand new v2.6 stable branch !

This bump is not trivial and comes with a lot of changes, please read carefully as you will have to modify your mongodb configuration files !

ebuild changes

As a long time request and to be more in line with upstream’s recommendations (and systemd support) I moved the configuration of the mongoDB daemons to /etc so make sure to adapt to the new YAML format.

  • the mongodb configuration moved from /etc/conf.d/mongodb to the new YAML formatted /etc/mongodb.conf
  • the mongos configuration moved from /etc/conf.d/mongos to the new YAML formatted /etc/mongos.conf
  • the MMS agent configuration file has moved to /etc/mms-agent.conf

The init scripts also have been taken care of :

  • new and modern mongodb, mongos and mms-agent init scripts
  • their /etc/conf.d/ configuration files are only used to modify the init script’s behavior

highlights

The changelog is long and the goal of this post is not to give you an already well covered topic on the release notes but here are my favorite features :

  • MongoDB preserves the order of the document fields following write operations.
  • A new write protocol integrates write operations with write concerns. The protocol also provides improved support for bulk operations.
  • MongoDB can now use index intersection to fulfill queries supported by more than one index.
  • Index Filters to limit which indexes can become the winning plan for a query.
  • Background index build allowed on secondaries.
  • New cleanupOrphaned command to remove orphaned documents from a shard.
  • usePowerOf2Sizes is now the default allocation strategy for all new collections.
  • Removed upward limit of 20 000 connections for the maxIncomingConnections for mongod and mongos.
  • New cursor.maxTimeMS() and corresponding maxTimeMS option for commands to specify a time limit.

Make sure you follow the official upgrade plan to upgrade from a previous version, this release is not a simple drop-in replacement.

thanks

Special thanks go to Johan Bergström for his continuous efforts and responsiveness as well as Mike Limansky and Jason A. Donenfeld.