RIP stabber

Today we did shutdown the oldest Gentoo Linux server of our oldest production datacenter. It was running since April, 5th of year 2006 so that’s a total of 2793 days of production level service as a stateful firewall. Its name was stabber, in reference of a vessel in the Eve Online MMORPG which I played a lot at the time.

Our company has been running on Gentoo Linux since 2004 for its Linux platforms and I often hear and experience the astonishment of the other persons I speak to about this : “Gentoo Linux in production, really ?” or “Wow you guys are a bunch of crazy hardcore Gurus”

As if Gentoo Linux did not meet the production level requirements or the security level you expect from another major (usually not free) distribution and as if you had to master some major skills to have it done…

7 years later, stabber is in my opinion a proof that all those assumptions are wrong.

  • I was a junior sysadmin at the time I made this server, we didn’t want to pay for having a proper firewall so we decided to make our own (that’s what Gentoo is to me : simple things done right, no added sugar)
  • The rolling updates of Gentoo did not brake our system and it evolved along our infrastructure
  • The GLSA kept our server immune to security breaches over the years (thx to the Gentoo security team)
  • This server/firewall passed the security tests of both Paypal and Ebay, this looks production level enough to me

We did shutdown this server because it was a single point of failure on an old part of our architecture. Its role has been taken over by two fault tolerant servers/firewalls running… Gentoo Linux of course !

First emerge.log entry

Wed Apr  5 12:53:22 2006 >>> sys-kernel/hardened-sources-2.6.14-r5

Latest uname -a

Linux stabber 2.6.16-hardened-r11 #1 SMP PREEMPT Wed Aug 30 15:51:49 CEST 2006 i686 Intel(R) Xeon(TM) CPU 3.20GHz GenuineIntel GNU/Linux

Latest commands

stabber ~ # echo "je taime" >> last.letter
stabber ~ # shutdown now -h

Dear fellow Gentoo Linux developers, your work makes all this possible, thank you !

20 thoughts on “RIP stabber

  1. Joerg Neikes

    Old but running.

    Hm lets see our old firewall:

    emerge
    Started emerge on: Mar 01, 2006 16:01:21
    *** emerge –oneshot >=sys-apps/portage-2.0.51.22
    uname -a
    Linux hades 2.6.16-gentoo-r4 #1 SMP Sun Apr 30 15:07:33 CEST 2006 i686 Intel(R) Pentium(R) 4 CPU 2.80GHz GenuineIntel GNU/Linux

    I must mention this one was running a debian system 4 years long before we set up using it with gentoo too.

    Reply
  2. hshhhhh

    OMG. Bye-bye, gentoo-server :(

    I don’t understand why people don’t like use gentoo on servers. I think if you are using gentoo you don’t need to use chief or puppet.

    Reply
      1. hshhhhh

        Of course if you have >5 servers puppet rocks.
        In another case if you will copy your world file & /etc/portage/ from server to server it very easy to administrate. More easy then any debian server.

        Reply
    1. Alan McKinnon

      There are several very good reasons why Gentoo is often not appropriate for production servers. The way Gentoo works, it encourages bespoke customization – after all, this IS it’s strong point. It’s also its weakest point as it doesn’t scale, and it doesn’t scale even faster when the admin is no longer the guy that built it – emerge world takes a significant amount of brain power to parse and you cannot just omit the -a and walk away. And yet, in real life on production, that is what you must do.

      The team I washead admin in till recently has around 300 servers to maintain at an ISP and only about 50 have duplicate configs with something else (DNS cache and auth servers, the AAA service running Tacacs+ and Radius, the rancid hosts, and a few other thing). Everything else is different to everything else. The current sysadmins – let’s just say they are not in my pay grade – and wouldn’t know a USE flag on a silver plate. I could train them, but that will take ages, and this is prod. Much easier, cheaper and faster to push an invoice from RH to the manager.

      Having said that, Gentoo excels in dev environments. The developers are forever asking for different daemon and python versions for testing. Gentoo makes this a walk in the park; RH and Ubuntu et al make this impossible. And if you have a server farm with many identical machines running a highly bespoke config – nothing beats Gentoo there. Build once, deploy binpkg many times, and run “emerge -uND world” in complete safety everywhere.

      Sadly, I had to ban Gentoo in prod for all these reasons (I couldn’t meet business expectations considering all the above constraints). But I still have a few left, the oldest:

      Linux subversive 2.6.18-gentoo-r3 #1 SMP Wed Dec 6 12:40:52 SAST 2006 i686 Pentium III (Coppermine) GenuineIntel GNU/Linux
      $ uptime
      01:01:22 up 1330 days, 5:22, 1 user, load average: 0.01, 0.02, 0.00

      And all my personal machines run Gentoo (expect the ones I treat as appliances like the FreeNAS storage boxes and the OpenElec HTPC). My own laptops have run nothing else for 9 years!

      Reply
      1. ultrabug Post author

        I agree this is hard to keep on going with unique servers all over the place. Being in a web company, our infrastructure is made only of clusters (except dev servers sometimes) so we definitely benefit from Puppet there to build and manage our server farms.

        I’m also enforcing a strong policy on my Linux sysadmins and they have to learn then prove their understanding of Gentoo before they can ssh anywhere. This is time spent on integration but I think it proves worthy in the long run. Our devs also run Gentoo and they love it for its flexibility as you said indeed.

        Now I know a lot of them converted to Gentoo at home and everywhere, I sure did and still do like you on every machine I own :)

        Reply
        1. Alan McKinnon

          Your infrastructure sounds like the kind of thing where Gentoo does excel. I find in situations like that it is much easier for everyone to just learn Gentoo and understand your specific customizations (thsi is not too hard), rather than figure out how to get Centos to do this different thing that is critical to your needs (this is often very hard to do and to docu).

          Reply
      2. hshhhhh

        300 servers. OMG :(
        Hate you lol.

        Of course gentoo not the silver bullet and at my work we using ubuntu for production just because I’m a programmer and not sysadmin and I’m only one in my company who loves and uses gentoo.

        But with ubuntu we have troubles with update — more easy reinstall system from zero then update it to next LTS version. So I still think if you understand what gentoo is — it great choose.

        Reply
  3. FrostyX

    I think you totally should write an article about managing gentoo servers in production environment.
    Keep in mind – The more the merrier ;-)

    Reply
  4. Florian D.

    great, thanks for sharing this.
    …and yes, thank you to the gentoo security team and all the fellow developers. I switched from SUSE 5.1 (don’t know what year this was!) to gentoo and still happy.

    Reply
  5. rudregues

    Inspirational post, thanks for sharing your experience. Maybe I recommend Gentoo for my father (he is an IT professional).
    Good luck with the new 2 systems and RIP stabber!

    Reply
  6. Sander Knopper

    I’m using Gentoo wherever I can and really appreciate its flexibility. Though (fortunately) I don’t have to manage a very large amount of production servers, so I can cope with all the manual work that’s involved.

    I am however very interested in using puppet in combination with Gentoo. If there’s anything you can share with us on top of the “using Gentoo in a production environment” I’d be very grateful.

    Reply
    1. ultrabug Post author

      I’ll try my best to share about this soon as you don’t seem to be the only one interested (I didn’t expect that tbh)

      Reply
  7. Bertrand Jacquin

    Hi,

    What are theses tests that you mention here :

    « This server/firewall passed the security tests of both Paypal and Ebay, this looks production level enough to me ».

    This is really interesting me.

    Reply
    1. ultrabug Post author

      I’ll sum it up a bit as those companies tests were quite similar (tjo Paypal has a quite long security questionnaire compared to ebay)

      Our company created and operated some online marketing/advertising websites and emailings for ebay UK and later for Paypal. Such companies have very strict security policies and do not delegate work easily, even more when they don’t host it.

      We basically had to give them a schema of every piece of infrastructure which would host the given projects and were told we’d have to pass a security scan in a time frame of our mutual choosing. Their tests went on smoothly and we passed them.

      Their report is not very detailled as you imagine since those companies do not share the details of their security tests but they did scan and try several tricks on our firewall at that time, stabber ;) The rest was more about SSL ciphers and other web server vulnerability detection (quite standard stuff).

      Reply
  8. Mike

    Hi,

    impressive life-time.
    Do I understand this right? You run a kernel built in 2006?

    Bye
    Mike

    Reply
    1. ultrabug Post author

      Well no, it’s lifetime wasn’t exactly its uptime. We upgraded it a few times for obvious security related reasons, nothing biggie tho.

      EDIT: sorry your question was about the kernel, yes you understand it correctly.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>