Tag Archives: pymongo

uWSGI, gevent and pymongo 3 threads mayhem

This is a quick heads-up post about a behaviour change when running a gevent based application using the new pymongo 3 driver under uWSGI and its gevent loop.

I was naturally curious about testing this brand new and major update of the python driver for mongoDB so I just played it dumb : update and give a try on our existing code base.

The first thing I noticed instantly is that a vast majority of our applications were suddenly unable to reload gracefully and were force killed by uWSGI after some time !

worker 1 (pid: 9839) is taking too much time to die...NO MERCY !!!

uWSGI’s gevent-wait-for-hub

All our applications must be able to be gracefully reloaded at any time. Some of them are spawning quite a few greenlets on their own so as an added measure of making sure we never loose any running greenlet we use the gevent-wait-for-hub option, which is described as follow :

wait for gevent hub's death instead of the control greenlet

… which does not mean a lot but is explained in a previous uWSGI changelog :

During shutdown only the greenlets spawned by uWSGI are taken in account,
and after all of them are destroyed the process will exit.

This is different from the old approach where the process wait for
ALL the currently available greenlets (and monkeypatched threads).

If you prefer the old behaviour just specify the option gevent-wait-for-hub

pymongo 3

Compared to its previous 2.x versions, one of the overall key aspect of the new pymongo 3 driver is its intensive usage of threads to handle server discovery and connection pools.

Now we can relate this very fact to the gevent-wait-for-hub behaviour explained above :

the process wait for ALL the currently available greenlets
(and monkeypatched threads)

This explained why our applications were hanging until the reload-mercy (force kill) timeout option of uWSGI hit the fan !

conclusion

When using pymongo 3 with the gevent-wait-for-hub option, you have to keep in mind that all of pymongo’s threads (so monkey patched threads) are considered as active greenlets and will thus be waited for termination before uWSGI recycles the worker !

Two options come in mind to handle this properly :

  1. stop using the gevent-wait-for-hub option and change your code to use a gevent pool group to make sure that all of your important greenlets are taken care of when a graceful reload happens (this is how we do it today, the gevent-wait-for-hub option usage was just over protective for us).
  2. modify your code to properly close all your pymongo connections on graceful reloads.

Hope this will save some people the trouble of debugging this ūüėČ

mongoDB 2.4.10 & pymongo 2.7

I’m pleased to announce those latest mongoDB related bumps. The next version bump will be for the brand new mongoDB 2.6 for which I’ll add some improvements to the Gentoo ebuild so stay tuned ūüėČ

mongodb-2.4.10

  • ¬†fixes some memory leaks
  • start elections if more than one primary is detected
  • fixes issues about indexes building and replication on secondaries
  • chunk size is decreased to 255 KB (from 256 KB) to avoid overhead with usePowerOf2Sizes option

All mongodb-2.4.10 changelog here.

pymongo-2.7

  • of course, the main feature is the mongoDB 2.6 support
  • new bulk write API (I love it)
  • much improved concurrency control for MongoClient
  • support for GridFS queries

All pymongo-2.7 changelog here.

mongoDB v2.4.7 & pymongo v2.6.3

First of all, I’d like to point out a quite big change in the Gentoo mongodb package. The Chromium team responsible for the v8 package decided to stop its maintenance as it was too much trouble to be used efficiently as a shared library (mainly due to upstream’s breakage behavior). Even tho I don’t like bundled libraries on sources, I understand my fellow developers point of view.

I’ve thus been asked and did switch the mongodb ebuild to use the bundled v8 library. This means that mongodb has no more v8 packaging dependency now. The mongodb v2.2.x¬†users are advised that since upstream does not bundle the v8 lib in their source, I dropped the v8 USE flag and support altogether on this version (it’s not officially supported anyway) !

This being said, I’ll drop the old ebuilds from tree on the next releases iterations.

mongodb-2.4.7

Yet another bugfix release on this unfamous 2.4.x series :

  • Fixed over-aggressive caching of V8 Isolates
  • Removed extraneous initial count during mapReduce
  • Cache results of dbhash command
  • Fixed memory leak in aggregation

pymongo-2.6.3

BSON parser hardening and fixes in the connection pool mechanism. More info here.

 

mongoDB v2.4.6 & pymongo v2.6

mongodb-2.4.6

The folks at 10gen discovered a severe bug in the mongoDB chunk migration process on sharded environments.

Basically, depending of the size of your documents, there was a chance that some get lost during data migration ! Relax tho, this case affected only the chunks containing documents which size are in the range of 16,776,185 and 16,777,216 bytes (inclusive) so this means that if you don’t have quite big documents in your cluster, you have not been affected by this bug.

Still, as a maintainer and production user of mongoDB, this is not the kind of news I like to hear especially when thinking of you Gentoo users. On top of this, I had the bad surprise to experience again the stale replication bug that was supposed to be fixed on 2.4.5 on my production cluster.

So I decided this was time for a major cleanup of the mongoDB ebuilds in portage to make sure we’re not shipping known broken versions of mongoDB to you guys. I thus :

  • dropped all obsolete <mongodb-2.2 ebuilds (I warned about this quite some time ago now)
  • dropped the known bugged versions of mongodb (<2.2.6 and <2.4.6)
  • cleaned up all obsolete files in $FILESDIR
  • added, on the v2.4.6 ebuild, the embedded-v8¬†USE flag as a user convenience¬†so you can now have packages requiring v8-3.19 and mongodb installed on your machine. I added an explicit warning about this as this is not the way to go on normal usage as this is against Gentoo policy.
*mongodb-2.4.6 (21 Aug 2013)
*mongodb-2.2.6 (21 Aug 2013)

  21 Aug 2013; Ultrabug <ultrabug@gentoo.org> -mongodb-2.0.7-r1.ebuild,
  -mongodb-2.0.7-r2.ebuild, -mongodb-2.0.8-r1.ebuild, -mongodb-2.0.8-r2.ebuild,
  -mongodb-2.2.0-r1.ebuild, -mongodb-2.2.0-r2.ebuild, -mongodb-2.2.4.ebuild,
  +mongodb-2.2.6.ebuild, -mongodb-2.4.5.ebuild, -mongodb-2.4.6_rc1.ebuild,
  +mongodb-2.4.6.ebuild, -files/mongodb-1.8.5-fix-smokepy.patch,
  -files/mongodb-1.8-fix-scons.patch, -files/mongodb-2.2-fix-scons.patch,
  -files/mongodb-2.2-fix-sconscript.patch,
  -files/mongodb-2.4.4-fix-sharedclient.patch, -files/mongodb.initd,
  -files/mongodb-linux3.patch, -files/mongos.initd, metadata.xml:
  version bump, add embedded-v8 USE, drop critically bugged versions, drop
  obsolete versions, filesdir cleanup

I understand some people may still need some of those ebuilds so if that’s the case, just shout and I’ll gladly add them to my overlay so you can still use them easily.

other highlights

  • Improved replication robustness in presence of high network latency
  • Resolved replica set initial sync issue on certain virtualized platforms
  • Resolved sharding migration issue that produced excessive small chunks
  • Resolved C++ client shutdown issues

pymongo-2.6

This one is quite interesting as it brings both new and improved features as well as some bug fixes. Also note that they fixed some gevent compatibility stuff.

highlights / explanations

  • The max_pool_size¬†option actually means what it says now. Pymongo will open at most this number of sockets to your servers. Do remember that if you share a connection between threads, then your (max_pool_size+1) thread will wait for a socket to be freed before being able to process your command.
  • waitQueueMultiple and¬†waitQueueTimeoutMS options will help you define how much and how long you want a process to wait for a socket to be available before raising an exception.
  • Pymongo automatically splits batch inserts into 48MB chunks so you don’t have to worry about pushing a huge list of documents for insertion.
  • Support for aggregation cursors (for use with dev version 2.5.1, not used on production now)
  • Support for exhaust cursors.¬†When you queried a large amount of data, the client had to ask the server for each batch of results. An exhaust cursor will instead stream batches to the client as quick as possible. This make pulling large sets of data faster and more reliable than before !

You can see the full extend of this bump on the pymongo Jira.

mongoDB : latest releases

mongodb-2.4.4

Just bumped it to portage and fixed an open bug along. This is yet another bugfix release which backports the switch to the Cyrus SASL2 library for sasl authentication (kerberos). Dependencies were adjusted so you no longer need libgsasl on your systems (remember to depclean).

highlights

  • config upgrade fails if collection missing “key” field
  • migrate to Cyrus SASL2 library for sasl authentication
  • rollback files missing after rollback

pymongo-2.5.2

This one is important to note and I strongly encourage you to upgrade asap as it fixes an important security bug (CVE-2013-2132). I’ve almost dropped all other versions from tree anyway…

highlights 2.5.x

  • support GSSAPI (kerberos) authentication
  • support for SSL certificate validation with hostname matching
  • support for delegated and role based authentication

mongodb-2.5.x dev

What’s cooking for the next 2.6 releases ? Let’s take a quick look as of today.

  • background indexing on secondaries (hell yes!)
  • new implementation of external sort
  • add support for building from source with particular C++11 compilers (will fix a gentoo bug reported quite a long time ago)
  • mongod automatically continues in progress index builds following restart

Python : new zeroMQ and mongoDB drivers

pyzmq-2.2.0.1

This one is very interesting to me because the code from the mighty gevent-zeromq library which brought gevent support to pyzmq has been merged into it ! I find it very humble and positive for the Open Source community to see such merges and want to express my gratitude to Travis Cline and the zeroMQ team for that.

Migrating is as easy as :

# gevent-zeromq previous way
from gevent_zeromq import zmq

# pyzmq new way
from pyzmq.green import zmq

I strongly encourage you to read the changelog.

pymongo-2.4.2

Bugfix release, the main point is that PyMongo will no longer select a replica set member for read operations that is not in primary or secondary state. Here is the changelog.

mongoDB v2.2.2 and pymongo v2.4 released

mongodb-2.2.2

This is a bugfix release of mongoDB, there is nothing major to note about it so just have a look at the changelog.

pymongo-2.4

Ok here’s a bigger cake we’ll have to focus on as you will have to adapt your code to use it properly. Don’t be scared, upgrading will not instantly break your current apps but…

Connection / ReplicaSetConnection deprecation

Those classes are still available and provide the old safe=False behavior meaning that by default, operations are not acknowledged. They are being replaced by MongoClient and MongoReplicaSetClient classes which on the contrary do acknowledge operations by default. So yes, now your operations will run with a safe=True by default !

Write concern

In the mongoDB talks we never hear about safe writes but about write concerns. A new API now handles these operations’ behavior such as fsync / journal committing / write acknowledgment which is in line with the internals of mongoDB. I think it’s more clear and straightforward to handle this that way so it’s a good job done by upstream even if it means we have to adapt our code for it. They come in the number of four options which are applied on a database or collection level :

  • w¬†= integer¬†: A value of 0 means we don’t care so it’s the fire-and-forget behavior we knew as safe=False. A value > 0 is the equivalent of the safe=True but with a more fine tuning on how many servers should confirm the operation.
  • wtimeout = integer : Adds a timeout on the w parameter.
  • j = bool : Wait until the operation has been committed to the journal.
  • fsync = bool : Wait until the database to fsync all files to disk.

 Highlights

  • Cursor can be copied with functions from the copy module
  • The set_profiling_level() method now supports a slow_ms option

See the rest in the full changelog.