Monthly Archives: November 2012

uWSGI : v1.4.2 released

Quick post for a bugfix release with some added flavor features for PHP users.

highlights

  • improved perl mules support
  • fixed pending datas management in https router
  • fixed thread-offloading of really big static files
  • added –php-app-qs as micro-optimization (in-place of rewrite rules) for php apps
  • added –php-var to inject custom (fixed) vars in the request

Full changelog here, as usual.

mongoDB v2.2.2 and pymongo v2.4 released

mongodb-2.2.2

This is a bugfix release of mongoDB, there is nothing major to note about it so just have a look at the changelog.

pymongo-2.4

Ok here’s a bigger cake we’ll have to focus on as you will have to adapt your code to use it properly. Don’t be scared, upgrading will not instantly break your current apps but…

Connection / ReplicaSetConnection deprecation

Those classes are still available and provide the old safe=False behavior meaning that by default, operations are not acknowledged. They are being replaced by MongoClient and MongoReplicaSetClient classes which on the contrary do acknowledge operations by default. So yes, now your operations will run with a safe=True by default !

Write concern

In the mongoDB talks we never hear about safe writes but about write concerns. A new API now handles these operations’ behavior such as fsync / journal committing / write acknowledgment which is in line with the internals of mongoDB. I think it’s more clear and straightforward to handle this that way so it’s a good job done by upstream even if it means we have to adapt our code for it. They come in the number of four options which are applied on a database or collection level :

  • w = integer : A value of 0 means we don’t care so it’s the fire-and-forget behavior we knew as safe=False. A value > 0 is the equivalent of the safe=True but with a more fine tuning on how many servers should confirm the operation.
  • wtimeout = integer : Adds a timeout on the w parameter.
  • j = bool : Wait until the operation has been committed to the journal.
  • fsync = bool : Wait until the database to fsync all files to disk.

 Highlights

  • Cursor can be copied with functions from the copy module
  • The set_profiling_level() method now supports a slow_ms option

See the rest in the full changelog.

 

rsyslog : new v7 branch released

It’s been a long time since I took care of the rsyslog package so bear with me for these quite huge version releases. The main thing to note is that I finally packaged the new v7 branch which is stamped “stable for production” by upstream. The v5 branch is not supported anymore unless you have a professional contract with Adiscon so I encourage you to read Rainer’s blog post.

 

v7.2.2

Optimizations, code refactoring for way improved performances and a lot of new features are there. The v7 series bring a lot of changes and neat stuff to rsyslog as you can see on this v5 vs v7 link. Here are some hints :

  • Improved configuration language
  • Improved execution engine
  • Full support for structured logging and JSON-based log messages (I discussed this matter on a previous post)
  • Ability to normalize legacy text log messages to JSON

The changelog is very long, so you can browse this to find out more.

v5.10.1 and v6.6.0

Those releases contain backports bugfixes (v5) and enhancements (v6) fixed on the v7 branch so if you’re not ready to jump straight to the new version, you might consider updating your branch for a last time 🙂

uWSGI : v1.4.1 LTS released

Ok that’s a neat uwsgi release here and stamped for Long Time Support by unbit. I’ve tested it before bumping it to portage and it’s behaving nicely with my gevent apps as there have been some nice improvements on this part as well. Quite a bunch of code have been refactored and optimized for enhanced performances. Have a look at the highlights, they’re to be read with care.

highlights :

  • Support for the Go language
  • Enhanced gevent reloading and handling (truly non-blocking wsgi.input, truly non-blocking writes…)
  • Improved http/https router and fastrouter (better event-based engineering, reduced syscall usage)
  • Improved systemd support
  • Log filtering and routing
  • Smart attach daemon to start a daemon along with your app
  • A big work is being done about the documentation

Yep that’s some heavy stuff, there is more so you should definitely read the different changelogs on the mailing list !

Clustering : corosync v2.1.0 & pacemaker v1.1.8

I recently bumped quite a bunch of the clustering suite packages such as :

  • sys-cluster/cluster-glue-1.0.11
  • sys-cluster/libqb-0.14.3
  • sys-cluster/corosync-1.4.4
  • sys-cluster/corosync-2.1.0
  • sys-cluster/crmsh-1.2.1 (new package)

corosync-2.1.0

You should be aware that the corosync-2.x packages are still hard masked because the 2.0.x ones didn’t compile properly and we didn’t have a suitable pacemaker version for it to work with.

There is another thing for us to consider and handle in the ebuilds when we’ll be willing to release corosync-2 : it is not backward compatible ! So yes, you will either have to start with a fresh cluster or break your existing ones to migrate to corosync-2. The reason is that upstream decided to drop the plugins support from their software. So a non-plugin cluster cannot work with a plugin-enabled one.

pacemaker-1.1.8

As for pacemaker-1.1.8, the bump request took quite some time. The reason is that it was released without backward compatibility support as well so you couldn’t join your existing pacemaker-1.1.x cluster even when using corosync-1.x ! I found it unacceptable because this meant I would have to force for corosync-2 usage starting from pacemaker-1.1.8 for no real reason. Pacemaker upstream, namely Andrew Beekhof, is very responsive and kind so he offered a solution which I’m happy to provide to Gentoo users.

crmsh-1.2.1

The crm command is not included in the pacemaker sources anymore. It is now an independent project lead by Dejan Muhamedagic to allow more versatility in its development and a better iteration of releases. I packaged and released it along with pacemaker-1.1.8 as well.

Todo

I’m still slacking on the sys-cluster/resource-agents package unfortunately. I already discussed with upstream about it as it’s not a straightforward bump because they merged two different resource-agents developments. I think to have a pretty good idea of what needs to be done and will do my best to fix this gap as soon as I can.

mongoDB : ebuilds cleanup and v2.2.1 released

Another bug killing spree has happened 🙂 I’m glad to have closed quite a bunch of bugs related to mongoDB today.

No more spidermonkey-1.7 dependency

Thanks to the help from Ian Stakenvicius, we managed to drop the spidermonkey-1.7 dependency and use the embedded version shipped in the sources. I extended this fix to all 2.0.x and 2.2.x ebuilds, the only package remaining is the one for v1.8.5 but it will be dropped soon.

Newer boost compatibility

Boost-1.50 introduced a new filesystem v3 version while breaking compatibility with older v2 filesystems. This broke mongoDB compilation for the guys running unstable version of boost. As I didn’t want to force stable users to keyword their boost versions, I kept a version of each 2.x series compatible with v2 <boost-1.50 filesystem.

  • <dev-libs/boost-1.50 users should use the ebuilds revisions 1 (-r1)
  • >=dev-libs/boost-1.50 users should use the ebuilds revisions 2 (-r2)

v2.2.1 version bump

Last but not least, the new v2.2.1 is also available but only for >=boost-1.50 users. This is a nice bugfix release which you should consider to apply since it’s the first of the 2.2 series.

See the full changelog.

My views on Python

We’re having a pretty hot debate in my company about the core development language we would like to embrace in order to enhance our work flow and unlock both our innovation and development iteration.

I thought of doing a blog post instead of writing an email summarizing my point of view about why we should choose python in case it could help other people to make their own mind or at least understand mine. I don’t want to be dragged into the typical “language x VS language y” type of post as there are a lot of those already but instead focus on specific use cases. Anyway, to be totally transparent with you, dear reader, I have to tell you that the main “opponent” we’re considering is Javascript / node.js.

Python is easy to learn and write

I work for an online marketing/advertising company and we have four different IT/development teams with specific work to do.  That means that we all have our own constraints to take into account and we use different technologies / languages to achieve them. Opening a debate on a development language rationalization thus means that people will have to be able to learn the one we will choose, the quicker the better.

Well let’s be honest, python is not widely taught on IT schools so our guys definitely WILL have to learn it. But python is one of the most simple language to learn as you can quickly test and make progress. This mainly comes from its coding syntax which makes the code cleaner and easy to read, also the language’s keywords are simple to remember, no boring brackets or semicolons to drive you crazy. Python is about coding clean and instinctively.

A simple and straightforward syntax is very important to me because :

  • it is easier to read
  • it is easier to maintain
  • your code is lighter
  • you spend less time coding so you spend more time thinking

Python everywhere

When choosing your core language, you must make sure you can rely on it today and tomorrow for a wide range of use cases. I find it very interesting to have a look at a list of its application domains as it does really show that with this language at our core, we would be able to achieve anything.

The versatility of python is my main argument in favor of its adoption. This is a mature and proven language with a lot of libraries and frameworks to suit all our present and future needs. It is widely supported on an extensive set of platforms and I can’t think of an open-source project not supporting python right from the start.

At this point, I will quote the folks at AppNexus from their conference in the recent PyData NYC 2012 :

"Python's versatility allows us to use it both for offline analytical tasks as well as production system development. Doing so allows us to bridge the gap between prototypes and production by relying on the same code libraries and frameworks for both, thereby tightening our innovation loop". 

They say python helped them grow very rapidly and efficiently by permitting them to focus on innovation, needless to say that I share their point of view and would love to see this happening in my company.

Python use cases in my company

Now let’s talk about our present and projected use cases. This is not an exhaustive list as I want to keep it simple and demonstrate the versatility I just talked about.

Scripting and automation

I am a sysadmin and I am lazy. Nothing new here okay, but that’s how I met python a bunch of years ago. At that time my company’s infrastructure became more complex everyday as was growing very rapidly. New servers were arriving and provided new functionalities and new technologies which lead more and more to heterogeneity in the things we had to monitor, automate and configure.

Python is a sysadmin’s heaven with all its libraries capable of handling complex tasks easily, even in our cluster environments where you have to deal with parallel and high availability computing. This is a big relief to know that whatever the task you’re asked to carry you can safely say : “python can do it”. The keyword here is efficiency.

Complete and complex applications

A lot of modern and cross-platform applications are written in python or based on it. I for one wrote an email parsing software a few years ago and it’s still kicking in production, its maintenance is easy and it has evolved smoothly with our growth and needs.

Another thing I like about this language is that it’s fast and can benefit from a semi-compiled “byte-code” which speeds up your application. No, python is not C++ and speed is not it’s biggest advantage of course but it’s really fast enough to compete with others easily.

Let’s sample some famous software written in python :

  • BitTorrent, original client, along with several derivatives
  • Dropbox, a web-based file hosting service
  • OpenStack, a cloud computing IaaS platform
  • Portage, the heart of Gentoo Linux
  • Ubuntu Software Center, a graphical package manager

Web applications

Python also have some solid and very powerful librairies able to manage asynchronous, real-time and scalable web applications and services. We already do have some of those robust web apps running in production and python demonstrates everyday all of the strenghts I already talked about here. We use librairies such as gevent along with web frameworks like flask and message queuing with zeromq. Someday I may write a post about our python web stack, it may be interesting to share about it.

I have been able to recode a web app written in .NET very quickly while enhancing it in every way possible. It is way faster, reliable, fault tolerant and maintainable that it was before. Thanks to python we have a short development iteration which proves itself everyday as the application grows and is capable to meet and achieve any new challenge we’re asked to take care of. I’m convinced that no other language could have been so powerful and versatile than python to do so.

We’re not the only ones thinking and experiencing this of course, still in the list we can see :

  • Google uses Python for many tasks including the backends of web apps such as Google Groups, Gmail, and Google Maps, as well as for some of its search-engine internals
  • AppNexus uses Python for some of their web apps backend
  • YouTube uses Python “to produce maintainable features in record times, with a minimum of developers”
  • Yahoo! Groups uses Python “to maintain its discussion groups”

That’s some big players indeed and it’s interesting to see they use python for their web app backends.

Conclusion

I am not a software developer as I never took strong development courses at school. I am a sysadmin of complex, clustered and heterogeneous environments so this affects my development standards and point of view in a way that my expectations will surely be different from a pure developer.

My main concerns can be defined with words like proven, easy, clean, versatile, maintainable, fast (to code and execute), scalable, fault-tolerant and cross-platform. All of my choices have been based on those standards and concerns and I think they apply well in our debate because I chose Python to meet them all and I’ve never been disappointed or limited by it.

I hope this post reflects my thoughts and helped you understand them. I will tell you about the result and the decision my company made on this debate.