Patrick's playground

# ping somemachine.local
PING somemachine.local.intern (67.215.65.132) 56(84) bytes of data.
64 bytes from hit-nxdomain.opendns.com (67.215.65.132): icmp_req=1 ttl=59 time=585 ms

That's WRONG WRONG WRONG

I wish that the idiots who thought that breaking standards is a good idea get all their mail misdelivered. Instead of properly returning a NXDOMAIN failure the OpenDNS idiocy "nameserver" returns a wrong domain, so now the ordering of my nameservers in /etc/resolv.conf is critical to reach the right machine.

The original problem is that the local network has a nameserver that resolves local names (*.company), and a VPN I use uses .local - and because the local network DNS comes first and the admin thought "OpenDNS must be Open and such" I now need to figure out if I want to have clean DNS, or if I want to access the local network by name, or if I want to access the VPN properly..

So much BRAINDAMAGE, AGAIN, just because people think that, liek, fix0ring DNS to return0rz a niec websiet is moar fun.
Please don't even think about thinking about doing that, it's wrong and makes people like me consider breaking some of your metacarpals just to see if you notice.

Some days you wake up and see something like this in the server logs:

Out of memory: kill process 16955 (postgres) score 1253029 or a child
Killed process 16958 (postgres) vsz:2220624kB, anon-rss:2612kB, file-rss:1107116kB
postgres: page allocation failure. order:0, mode:0x20058

And then you're quite unhappy because the kernel OOM killer triggered, and it decided to take out your database because it could.
Not Funny!

So first of all we want to see what scores the processes have:

  # for i in `pidof /usr/lib64/postgresql-9.0/bin/postgres`; do cat /proc/$i/oom_score; done

And that's returning a lot:

92560
139000
555569
555492
555857
39676
277779
138970
138991
8691
8619
277666
277486
277801

What does this mean? Basically that's the score the kernel gives, and the highest score is the first one to die. Which is ... uhm ... pretty bad, because no one else scores higher.
Now let's be a slightly bit evil:

for i in `pidof /usr/lib64/postgresql-9.0/bin/postgres`; do echo "-17" >  /proc/$i/oom_adj; done

What does this do? It blesses all children of the main postgres process with a value of "not killable by OOM-killer"

Now this doesn't actually fix the bug, but our database and all its children (including newly spawned ones, as far as I can tell) won't be randomly killed.

for i in `pidof /usr/lib64/postgresql-9.0/bin/postgres`; do cat /proc/$i/oom_adj; done

And what do we see? They are all blessed, and at least some other process dies now. This gives you enough time to figure out why it exploderated in the first place, but you may wish to apply this "protection" to all critical processes.

Like sshd (which seems to do that automatically) or your webserver, maybe?

$ diff -Naur sshd_config sshd_nohup 
--- sshd_config 2011-04-14 10:15:28.565148992 +0200
+++ sshd_nohup  2011-04-14 10:14:12.015036717 +0200
@@ -12,6 +12,7 @@
 HostKey /etc/ssh/ssh_host_dsa_key
 #Privilege Separation is turned on for security
 UsePrivilegeSeparation yes
+PermitTunnel yes
 
 # Lifetime and size of ephemeral version 1 server key
 KeyRegenerationInterval 3600
@@ -64,6 +65,10 @@
 PrintMotd no
 PrintLastLog yes
 TCPKeepAlive yes
+
+KeepAlive yes
+ClientAliveInterval 60
+ClientAliveCountMax 3
 #UseLogin no
 
 #MaxStartups 10:30:60

Tadaah!
Now you don't get disconnected so often. Maybe, optionally, one could set that as defaults, maybe?
Or does that make too much sense ...

Lately I've been reading about some companies that had troubles because their management was doing rather confusing and not really optimal things. There appears to be a new influx of graduates of the school of "damagement" where the goal of the upper levels of a company seems to be to dismantle the company in a short time. Great examples include hallucinatory visionaries like Carly Fiorina who almost managed to wreck HP, Stephen Elop who managed about 25% decrease of stock price within a reasonably short time, and recently Bob Parsons of GoDaddy fame who upset people with his rifle antics near elephants. But they all lack dedication and execution, so the results are quite average and unmotivated.

I feel that these companies deserve better, so here's my offer. It's a unique chance for all of you that need to properly run a company into the ground:

Hire me as CEO. I have no relevant experience (which puts me roughly in the middle of the bell curve), but the mindset of an engineer and an unwavering loyality that will keep me properly isolated from facts and other nasty parts of reality. My cultural background mixes german precision and swift execution of random ideas with french delusions of grandeur and a sense of invincibility. The mix of luxemburgian nationality and having lived in switzerland gives me a special talent for manipulating finances in ways that should not be possible.

If you need to have the wrong decisions made I will defend them until I never defended them. Proper flipfloping is a subtle art that most politicians seem to have mastered - I shall demonstrate you how this skill can demoralize your employees and reduce productivity at unprecedented rates. I guarantee maximum dissatisfaction, and for the right price I'll make sure that your company gets run into the ground properly. Only 25% loss of stock value in a week? Pishposh. If you wish to exit strategic markets and break the backbone of an enterprise you need proper planning and a PR strategy that allows vilification of your brand.

So please, if you need someone to take the blame and cause a proper mess feel free to contact me. My rates are very competitive, and I guarantee that you will not be disappointed. Even morally shady things can be discussed - it's only illegal if you get caught!