Tag Archives: nosql

mongoDB 3.0.1

This is a quite awaited version bump coming to portage and I’m glad to announce it’s made its way to the tree today !

I’ll right away thank a lot Tomas Mozes and Darko Luketic for their amazing help, feedback and patience !

mongodb-3.0.1

I introduced quite some changes in this ebuild which I wanted to share with you and warn you about. MongoDB upstream have stripped quite a bunch of things out of the main mongo core repository which I have in turn split into ebuilds.

Major changes :

  • respect upstream’s optimization flags : unless in debug build, user’s optimization flags will be ignored to prevent crashes and weird behaviour.
  • shared libraries for C/C++ are not built by the core mongo respository anymore, so I removed the static-libs USE flag.
  • various dependencies optimization to trigger a rebuild of mongoDB when one of its linked dependency changes.

app-admin/mongo-tools

The new tools USE flag allows you to pull a new ebuild named app-admin/mongo-tools which installs the commands listed below. Obviously, you can now just install this package if you only need those tools on your machine.

  • mongodump / mongorestore
  • mongoexport / mongoimport
  • mongotop
  • mongofiles
  • mongooplog
  • mongostat
  • bsondump

app-admin/mms-agent

The MMS agent has now some real version numbers and I don’t have to host their source on Gentoo’s infra woodpecker. At the moment there is only the monitoring agent available, shall anyone request the backup one, I’ll be glad to add its support too.

dev-libs/mongo-c(xx)-driver

I took this opportunity to add the dev-libs/mongo-cxx-driver to the tree and bump the mongo-c-driver one. Thank you to Balint SZENTE for his insight on this.

mongoDB v2.6.1

This is a great pleasure to announce the version bump of mongoDB to the brand new v2.6 stable branch !

This bump is not trivial and comes with a lot of changes, please read carefully as you will have to modify your mongodb configuration files !

ebuild changes

As a long time request and to be more in line with upstream’s recommendations (and systemd support) I moved the configuration of the mongoDB daemons to /etc so make sure to adapt to the new YAML format.

  • the mongodb configuration moved from /etc/conf.d/mongodb to the new YAML formatted /etc/mongodb.conf
  • the mongos configuration moved from /etc/conf.d/mongos to the new YAML formatted /etc/mongos.conf
  • the MMS agent configuration file has moved to /etc/mms-agent.conf

The init scripts also have been taken care of :

  • new and modern mongodb, mongos and mms-agent init scripts
  • their /etc/conf.d/ configuration files are only used to modify the init script’s behavior

highlights

The changelog is long and the goal of this post is not to give you an already well covered topic on the release notes but here are my favorite features :

  • MongoDB preserves the order of the document fields following write operations.
  • A new write protocol integrates write operations with write concerns. The protocol also provides improved support for bulk operations.
  • MongoDB can now use index intersection to fulfill queries supported by more than one index.
  • Index Filters to limit which indexes can become the winning plan for a query.
  • Background index build allowed on secondaries.
  • New cleanupOrphaned command to remove orphaned documents from a shard.
  • usePowerOf2Sizes is now the default allocation strategy for all new collections.
  • Removed upward limit of 20 000 connections for the maxIncomingConnections for mongod and mongos.
  • New cursor.maxTimeMS() and corresponding maxTimeMS option for commands to specify a time limit.

Make sure you follow the official upgrade plan to upgrade from a previous version, this release is not a simple drop-in replacement.

thanks

Special thanks go to Johan Bergström for his continuous efforts and responsiveness as well as Mike Limansky and Jason A. Donenfeld.

mongoDB 2.4.10 & pymongo 2.7

I’m pleased to announce those latest mongoDB related bumps. The next version bump will be for the brand new mongoDB 2.6 for which I’ll add some improvements to the Gentoo ebuild so stay tuned 😉

mongodb-2.4.10

  •  fixes some memory leaks
  • start elections if more than one primary is detected
  • fixes issues about indexes building and replication on secondaries
  • chunk size is decreased to 255 KB (from 256 KB) to avoid overhead with usePowerOf2Sizes option

All mongodb-2.4.10 changelog here.

pymongo-2.7

  • of course, the main feature is the mongoDB 2.6 support
  • new bulk write API (I love it)
  • much improved concurrency control for MongoClient
  • support for GridFS queries

All pymongo-2.7 changelog here.

Couchbase on Gentoo Linux

Back in 2010 when I was comparing different NoSQL solutions I came across CouchDB. Even tho I went for mongoDB in the end, it was still a nice and promising technology even more since the merge with the Membase guys in late 2012 which lead to the actual Couchbase.

I won’t go into the details of Couchbase itself since it’s way covered all around the net but I wanted to let you guys know that I’ve packaged most of the couchbase ecosystem for Gentoo Linux :

  • dev-db/couchbase-server-community-2.2.0 : the community server edition (bin)
  • dev-libs/libcouchbase-2.2.0 : the C client library
  • dev-python/couchbase-1.2.0 : the python client library

Those packages are still only available on my overlay (ultrabug on layman) since I’m not sure about the interest of other users in the community and I still need to make sure it’s production ready enough.

If you’re interested in seeing this package in portage, please say so !

I dedicate this packaging to @atorgfr 🙂

mongoDB v2.4.9/v2.2.7, rabbitMQ v3.2.3

Quick post about some recent bumps.

mongodb-2.4.9 & mongodb-2.2.7

IMPORTANT : These versions fix a mongos bug which could lead it to report a write as successful when it was not. This affects all versions of MongoDB prior to and including v2.4.8.

Stay tuned on mongoDB, the next post will probably talk about the release of pymongo v2.7 which supports some neat futures from the upcoming mongoDB v2.6 series.

rabbitMQ-3.2.3

I skipped a bump post when releasing the v3.2.2 so you should check out the v3.2.3 changelog as well if you’re willing to know more about those bug fix releases.

mongoDB v2.4.8, rabbitMQ v3.2.1, rsyslog v7.4.6

mongodb-2.4.8

You should consider this important update if you have a cluster running v2.4.7. It contains a fix for the config servers which can have them possibly disagree on chunks hashes and thus prevent mongos to start or balancing to happen. See this bug for more info.

rabbitMQ-3.2.1

The famous message queuing server got a nice bunch of bug fixes on a lot of its modules along with some interesting additions such as :

  • support for federated queues
  • report client authentication errors during connection establishment explicitly using connection.close
  • inform clients when memory or disk alarms are set or cleared
  • allow policies to target queues or exchanges or both
  • offer greater control over threshold at which messages are paged to disk
  • allow missing exchanges & queues to be deleted and unbound without generating an AMQP error
  • implement consumer priorities

Full changelog here and here.

rsyslog-7.4.6

This is a bug fix release, nothing too big about it as reported by Thomas D (thanks again).

Please note that rsyslog-7.4.4 is being stabilized, mainly for security purposes.

mongoDB v2.4.7 & pymongo v2.6.3

First of all, I’d like to point out a quite big change in the Gentoo mongodb package. The Chromium team responsible for the v8 package decided to stop its maintenance as it was too much trouble to be used efficiently as a shared library (mainly due to upstream’s breakage behavior). Even tho I don’t like bundled libraries on sources, I understand my fellow developers point of view.

I’ve thus been asked and did switch the mongodb ebuild to use the bundled v8 library. This means that mongodb has no more v8 packaging dependency now. The mongodb v2.2.x users are advised that since upstream does not bundle the v8 lib in their source, I dropped the v8 USE flag and support altogether on this version (it’s not officially supported anyway) !

This being said, I’ll drop the old ebuilds from tree on the next releases iterations.

mongodb-2.4.7

Yet another bugfix release on this unfamous 2.4.x series :

  • Fixed over-aggressive caching of V8 Isolates
  • Removed extraneous initial count during mapReduce
  • Cache results of dbhash command
  • Fixed memory leak in aggregation

pymongo-2.6.3

BSON parser hardening and fixes in the connection pool mechanism. More info here.

 

mongoDB v2.4.6 & pymongo v2.6

mongodb-2.4.6

The folks at 10gen discovered a severe bug in the mongoDB chunk migration process on sharded environments.

Basically, depending of the size of your documents, there was a chance that some get lost during data migration ! Relax tho, this case affected only the chunks containing documents which size are in the range of 16,776,185 and 16,777,216 bytes (inclusive) so this means that if you don’t have quite big documents in your cluster, you have not been affected by this bug.

Still, as a maintainer and production user of mongoDB, this is not the kind of news I like to hear especially when thinking of you Gentoo users. On top of this, I had the bad surprise to experience again the stale replication bug that was supposed to be fixed on 2.4.5 on my production cluster.

So I decided this was time for a major cleanup of the mongoDB ebuilds in portage to make sure we’re not shipping known broken versions of mongoDB to you guys. I thus :

  • dropped all obsolete <mongodb-2.2 ebuilds (I warned about this quite some time ago now)
  • dropped the known bugged versions of mongodb (<2.2.6 and <2.4.6)
  • cleaned up all obsolete files in $FILESDIR
  • added, on the v2.4.6 ebuild, the embedded-v8 USE flag as a user convenience so you can now have packages requiring v8-3.19 and mongodb installed on your machine. I added an explicit warning about this as this is not the way to go on normal usage as this is against Gentoo policy.
*mongodb-2.4.6 (21 Aug 2013)
*mongodb-2.2.6 (21 Aug 2013)

  21 Aug 2013; Ultrabug <ultrabug@gentoo.org> -mongodb-2.0.7-r1.ebuild,
  -mongodb-2.0.7-r2.ebuild, -mongodb-2.0.8-r1.ebuild, -mongodb-2.0.8-r2.ebuild,
  -mongodb-2.2.0-r1.ebuild, -mongodb-2.2.0-r2.ebuild, -mongodb-2.2.4.ebuild,
  +mongodb-2.2.6.ebuild, -mongodb-2.4.5.ebuild, -mongodb-2.4.6_rc1.ebuild,
  +mongodb-2.4.6.ebuild, -files/mongodb-1.8.5-fix-smokepy.patch,
  -files/mongodb-1.8-fix-scons.patch, -files/mongodb-2.2-fix-scons.patch,
  -files/mongodb-2.2-fix-sconscript.patch,
  -files/mongodb-2.4.4-fix-sharedclient.patch, -files/mongodb.initd,
  -files/mongodb-linux3.patch, -files/mongos.initd, metadata.xml:
  version bump, add embedded-v8 USE, drop critically bugged versions, drop
  obsolete versions, filesdir cleanup

I understand some people may still need some of those ebuilds so if that’s the case, just shout and I’ll gladly add them to my overlay so you can still use them easily.

other highlights

  • Improved replication robustness in presence of high network latency
  • Resolved replica set initial sync issue on certain virtualized platforms
  • Resolved sharding migration issue that produced excessive small chunks
  • Resolved C++ client shutdown issues

pymongo-2.6

This one is quite interesting as it brings both new and improved features as well as some bug fixes. Also note that they fixed some gevent compatibility stuff.

highlights / explanations

  • The max_pool_size option actually means what it says now. Pymongo will open at most this number of sockets to your servers. Do remember that if you share a connection between threads, then your (max_pool_size+1) thread will wait for a socket to be freed before being able to process your command.
  • waitQueueMultiple and waitQueueTimeoutMS options will help you define how much and how long you want a process to wait for a socket to be available before raising an exception.
  • Pymongo automatically splits batch inserts into 48MB chunks so you don’t have to worry about pushing a huge list of documents for insertion.
  • Support for aggregation cursors (for use with dev version 2.5.1, not used on production now)
  • Support for exhaust cursors. When you queried a large amount of data, the client had to ask the server for each batch of results. An exhaust cursor will instead stream batches to the client as quick as possible. This make pulling large sets of data faster and more reliable than before !

You can see the full extend of this bump on the pymongo Jira.

mongoDB : latest releases

mongodb-2.4.4

Just bumped it to portage and fixed an open bug along. This is yet another bugfix release which backports the switch to the Cyrus SASL2 library for sasl authentication (kerberos). Dependencies were adjusted so you no longer need libgsasl on your systems (remember to depclean).

highlights

  • config upgrade fails if collection missing “key” field
  • migrate to Cyrus SASL2 library for sasl authentication
  • rollback files missing after rollback

pymongo-2.5.2

This one is important to note and I strongly encourage you to upgrade asap as it fixes an important security bug (CVE-2013-2132). I’ve almost dropped all other versions from tree anyway…

highlights 2.5.x

  • support GSSAPI (kerberos) authentication
  • support for SSL certificate validation with hostname matching
  • support for delegated and role based authentication

mongodb-2.5.x dev

What’s cooking for the next 2.6 releases ? Let’s take a quick look as of today.

  • background indexing on secondaries (hell yes!)
  • new implementation of external sort
  • add support for building from source with particular C++11 compilers (will fix a gentoo bug reported quite a long time ago)
  • mongod automatically continues in progress index builds following restart

mongoDB and Pacemaker recent bumps

mongoDB 2.4.3

Yet another bugfix release, this new stable branch is surely one of the most quickly iterated I’ve ever seen. I guess we’ll wait a bit longer at work before migrating to 2.4.x.

pacemaker 1.1.10_rc1

This is the release of pacemaker we’ve been waiting for, fixing among other things, the ACL problem which was introduced in 1.1.9. Andrew and others are working hard to get a proper 1.1.10 out soon, thanks guys.

Meanwhile, we (gentoo cluster herd) have been contacted by @Psi-Jack who has offered his help to follow and keep some of our precious clustering packages up to date, I wish our work together will benefit everyone !

All of this is live on portage, enjoy.