Patrick's playground

One of the things I've been wanting to work on was a Gentoo Newsletter.
We used to have a Weekly Newsletter, then people got upset that the editor was an editor or something, then it was on life support for a while until it became the Monthly Newsletter, and after not even a year that just ceased to be. So for roughly the last 3 years there hasn't been a (semi-)regular publication to keep us all happy.

What people regularly underestimate is the amount of time that goes into a newsletter - just little things like doing mailinglist summaries easily takes an hour for every newsletter. Then there's items like interviews that are open-ended ... of course you can finish one up in 30 minutes, but that will be a bit bland and boring. So you find new questions, ask for clarifications on answers and soon you're looking at a few hours of time to process it nicely. Then you get semi-automated tasks like bug statistics and GLSAs, and once you have all those fragments you need to glue them together sanely, check that the formatting makes sense and send it to the gentoo-core@ mailinglist. People will find dozens of issues you've overlooked, so you correct them all, send it again and wait for the next round of corrections. And once that is done you can think about publishing ...

Of course you want to keep it sustainable, it'd be silly if everyone involved quit after two weeks and you had no one writing it. So you want to be able to plan ahead a bit and commit to a long-term schedule. Usually life gets in the way anyway, but you can mitigate it by having backup people for every role.
As you can guess I've not found the stability and time to get things going again, but it's itching me. So I'm scheming and plotting and preparing ... soon, soon I say! We shall have a Newsletter again. And if it kills me ;)

I think that many others want it to happen too, so if everyone just gives one or two hours of work every other week we should have enough manpower behind it to keep it going indefinitely. And you get the awesome feeling that you made people happy by telling them what happened in Gentooland this week. How can you say no to that!

Some people might have noticed that I haven't been able to spend as much time as usual on all things Gentoo.

Well, I have a pretty decent excuse. Work has been absorbing quite a bit of time, and I haven't been much at home in the last weeks. Whatever home means ...
Through some luck and chance I was able to spend 3 weeks in China, in the insane place they call Shanghai. It's a really intense and exciting place. I really enjoyed my time there, and if I get another chance to go there ...
Of course there's a language barrier (but then most chinese appear to have trouble understanding each other), and lots of little things that are just different enough to confuse you badly.
But ... the food! Oh dear. The food alone is worth it, and that's why most foreigners gain 15kg in the first year. Then there's things like the omnipresent taxis - want to go somewhere? Raise your hand, there's a taxi. Like in movies, only a bit more random - sometimes normal motorists stop and offer you an impromptu taxi.

At night, when it's raining, the city center is exactly like a William Gibson book that became alive - neon lights making everything glow in false colours, people running around like ants, the rain like static noise hiding everything in the distance ... it's pretty awesome. Did I mention the food?
There's some not so nice things like the lack of regulations we take for granted in europe - a construction site will run 24/7 and no one cares that it might be noisy. The air quality is at times questionable, "blue sky" is a hilarious idea. The amount of people is pretty insane, everything is alive - even at 4 in the morning there's still people running around, and there's so many shops that are open 24/7. Well, most shops are open every day, why would you not be open? That's just time you're not earning money!
In Shanghai even at night I never felt threatened, crime is relatively low, even when there's such a huge disparity between rich and poor. At worst some beggars follow you for a while in the hope of getting a generous donation out of you. Police is almost omnipresent, but passive - they don't interfere with normal life if they can avoid it.
The traffic is insane (not just carribean-insane or belgium-insane, but omgwtf insane), sometimes you see someone driving backwards on the highway because he missed an exit ... cyclists never seem to care about traffic lights, so you need to be vigilant all the time. But once you accept that it's a freeform self-organizing traffic that is surprisingly efficient. And it reminds me of the old game "Frogger" - just in 3D on expert mode.

So anyway, if you are a PHP coder ... the nice people at The Net Circle are really decent folks with a great work environment. You'd have to move to Shanghai, but I guess that's just another job bonus. I had lots of fun there ...

And just when I had returned to Europe I was summoned by the happy people at Lieferheld, a german startup trying to conquer the market of online food delivery ordering. Compared to Shanghai the nice little town of Berlin appears so calm and peaceful, but it's still a great city.
Actually Berlin feels like multiple little towns glued together - every area has a distinct feel and architecture. There's insane stuff like the remains of the Wall that was built through the middle of the city - what on earth were people thinking back then!?
It's an international city in the sense that speaking german is optional. But this just makes it much more accessible and communicative. There's alwas something happening, so the concept of getting bored is pretty much a theoretical construct. Especially in the late spring and summer it's nice because there are also many little parks and lots of places where you can just chill, grill or get a nice sunburn.

Now if you are a python-programmer or maybe even someone with experience in django you might find an interesting challenge there.

And there's a good chance that after my stay at Lieferheld I'll be absorbed into the family of OTRS, one of germany's few real internet companies - when you work online you can be wherever you are, so they have managed to allow homeoffice for most of their programmers and IT-people.

So blame these people for absorbing way too much of my time - I really want to do more than 2 alibi-commits a month, but these modern slavedrivers just find good ways to absorb all of my time ;)

And during all my travels I've met lots of great people, made some good new friends and learned quite a bit about life, the universe and everything.

So - what's next? We'll see. And if you have some good ideas for me, please, tell me. I'm not opposed to random new experiences ...

I've been quite annoyed by the Ubuntu "LOL UR CAN HAZ UPGRADE" message during login.

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc
New release 'natty' available.
Run 'do-release-upgrade' to upgrade to it.

Blargh. I CANNOT upgrade, because we use a certain release because ... well ... oh shut up.
Either way I need to get rid of that message, so ... I just overwrite /etc/motd, right?

Strangely that change disappears. WTF. Aargh.

# grep -R /etc/motd /etc/*
/etc/apparmor/severity.db:/etc/motd     1 3 0
grep: /etc/blkid.tab: No such file or directory
grep: /etc/fonts/conf.d/30-defoma.conf: No such file or directory
/etc/update-motd.d/99-footer:# /etc/motd.
/etc/update-motd.d/99-footer:[ -f /etc/motd.tail ] && cat /etc/motd.tail || true

Ooooh! So someone overwrites stuff in /etc automatedly!
https://wiki.ubuntu.com/UpdateMotd

A Cronjob... rewriting ... mah configfielz ?!? whut the fraq!

Ubuntu, you haz disappoint. I no can wriet werdz to eggspress how sad I haz.
Where's the Vodka ...

On a rather short notice I've moved the gentooexperimental.org vserver. As a result not everything has been set up properly.

I'll fix user accounts and such things in the next days. Feel free to poke me if you have an urgent need to access some bits, I'll see what I can do then.

Sorry for the troubles, and of course you get a full refund ;)

The last few days I've tried to figure out a funky bug with nginx.
Reading the upstream documentation and trying on Gentoo worked as expected (relevant feature: multiple upstreams for a proxy_pass / fastcgi_pass statement).

But on Ubuntu it breaks. Badly. Nginx claims that there's a syntax error in the config. Bad bad bad.

I narrowed it down to a few potential causes until I compared the ./configure lines - "nginx -V" prints those. Some little variations like mail support being on here and off there... and then I struck gold.

--add-module=/build/buildd/nginx-1.0.0/debian/modules/nginx-echo 
--add-module=/build/buildd/nginx-1.0.0/debian/modules/nginx-upstream-fair

Oooooh, someone patched in random third party modules. Guess what.

The smegging stupid nonstandard upstream-fair module BREAKS the default syntax.
And since it's compiled in EVERY SINGLE user of debian and ubuntu and derivatives will have stupid idiotic failures and not be able to use the upstream docs for that.

I do wonder a lot why the maintainer(s) decided to break things for no apparent reason. And it's not just a package from a random overlay or unofficial source - if you look at The Debian Packages Website for NGINX you notice:

Third Party Modules:

  Upstream Fair Queue, Echo

SRSLY ... WTF.

And here, just to help anyone who is forced to work around this idiotic braindamage:
Works with unpatched, fails on debian/ubuntu:

upstream foo {
	server 127.0.0.1;
	server 127.0.0.2;
}

Works with debian/ubuntu, fails on unpatched official sources:

upstream foo {
        server 127.0.0.1;
        server 127.0.0.2;
	fair;
}

Spot the difference ... Yeah, AWESOME.
I can barely contain my happiness!

Every now and then I am really positively surprised that there are still so many people that go beyond the rules and regulations and do whatever they can to help.
Sometimes it's a little thing like the salesperson in a shop giving you two free samples because you might like them (or even a whole bar of chocolate, because "I know that you'll be back for more"). Sometimes it's the conductor in the train giving you a verbal warning instead of a fine because you forgot your wallet.

And even if we don't always show our appreciation immediately, it's still a pretty awesome way to make someone's day better. I'll try to help others wherever I can, and I hope that all of you out there who help out others continue to make this world a better place. And maybe we can even motivate random other people to be nicer too, in a chain reaction of goodness that makes everything awesome.

So thank you, good people out there!

Sometimes you need to work on legacy platforms where not everything is handled by the package manager. Since that seems to happen too often to people they tend to reinvent random almost-package-managers. And then when naive people try to actually use them ... Oh well. Fun.
The offender I'm fighting with it "pip", which seems so be shorthand for "python's impossible packagethingy". It has no concept of updating, and it fails randomly at the tasks it should be able to do.
Plus, to make life more interesting, it is pure yoda-coding - there is no try. Exceptions won't happen. Well, unless they do, then you're screwed. Like this:

$ pip freeze | wc -l
Warning: cannot find svn location for flup==1.0.3.dev-20110111
33

So there's 33 packages handled by this thingy, and it throws random warnings. I have no idea what a svn location is supposed to be, but it can't be serious.

$ pip uninstall `pip freeze`
Warning: cannot find svn location for flup==1.0.3.dev-20110111
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/pip-1.0-py2.6.egg/pip/basecommand.py", line 126, in main
    self.run(options, args)
  File "/usr/local/lib/python2.6/dist-packages/pip-1.0-py2.6.egg/pip/commands/uninstall.py", line 33, in run
    InstallRequirement.from_line(name))
  File "/usr/local/lib/python2.6/dist-packages/pip-1.0-py2.6.egg/pip/req.py", line 98, in from_line
    return cls(req, comes_from, url=url)
  File "/usr/local/lib/python2.6/dist-packages/pip-1.0-py2.6.egg/pip/req.py", line 38, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2563, in parse
    raise ValueError("No requirements found", s)
ValueError: ('No requirements found', '##')

Storing complete log in /root/.pip/pip.log

Now this is amusing. I can't uninstall.
I get a random traceback that lacks context, so I can't even figure out which of the 33 packages is the offender.
The logfile contains the same output, so no debug help
So apart from figuring out where it stores data and manually removing everything I see no clean method to fix this. And it could all have been avoided by using the distro package manager. But that's, like, too complicated, man ... so we waste lots of time now instead of just having a sane working thingy that maybe takes 5 minutes more during the initial setup.

And apparently there's no easy way to clone the config to another machine, so I'll just go through bash_history and glue all those commands together. That's SO MUCH EASIER than just USING SYSTEM TOOLS.
Is bad for my mood, this idiocy. Seriously. Don't support such schemes for confusion! Do things sanely, and then don't break them. Can't be that hard ...

In the last months I've been travelling a lot more than usual. This has made me very aware of the deficiencies in the always-on, just stream it, store it in teh cloud mindset.

First of all, just getting a connection. While on a train, for example, I am usually happy if I get enough throughput to browse random webpages. There are gaps where it just isn't possible to get even a signal, the mobile phone can't even do a phone call at that point. So streaming music is a hilarious idea that makes no sense!

Second, latency and bandwidth. At times you can get ~800kB/s effective throughput with ~40ms latency, which is like a first-generation DSL connection and pretty decent. But as soon as you start moving, or someone esleuses his connection for downloading, or you leave the large cities, you can be glad when you get ~4kB/s with a latency under 2000msec. (As I'm writing I went through a connection gap again, and I switched to a local editor because the lag was too high for a ssh connection. And that's on one of the more advanced train routes [ Berlin-Frankfurt ] and not even somewhere in the middle of nowhere.)
The bandwidth is barely enough to read emails, fetch RSS feeds and that's about it. Quite exquisitely underwhelming.

Bandwidth caps - even in glorious kommunist europe there are silly caps on mobile connections. Right now I have a 2GB/mo limit, above that the connection gets cut down to 64kB/s max. And hitting that limit is so easy that I've started queueing larger downloads for the times when I get to a wired connection. Feels like 1996 all over again ;)
If I were streaming music at a decent quality I'd be hitting the 2GB limit in about 3-4 days. If I were watching Videos in decent quality I'd hit that limit in about 3-4h of playtime. Thus I'm forced to still have all my media available in an offline storage just to not cripple my internet experience for the other 29 days of a month. That's surprisingly sucky.



And as a bonus most mobile ISPs do image recoding on HTTP requests, so now stuff looks like green or brown blobs and the quality settings must be the absolute minimum. So it saves me ~20kB for an image, but I don't even see what the image tries to show me, and I use an ssh tunnel to see the unmodified original.
So much work to work around a workaround ... instead of just giving me the content properly. BLARGH.

Extra bonus: Youtube doesn't work, there seems to be a transparent video recoder in the way that randomly exploderates. More than the first 10 seconds of a YT video are unlikely, and those are usually 3 images because the framerate is terminally confused. So, uhm, err, it's like a slideshow, only worse.

And if you care about integrity you'll get an aneurism when you figure out that there's a transparent proxy that isn't transparent, rewriting most image requests to come from the 1.1.1.1...1.1.1.5 IP range, which definitely absolutely does NOT belong to that ISP. Just for fun they even break some marginal parts of the internet because ... because they can. HAH.

So tell me, with all these restrictions and procedural failures in the way, why should I risk not having access to my data? Cloudy things might be nice, but for me it's a simple "No thanks" and I'll continue carrying all data locally.

Sometimes one gets really used to sanity, and then it gets very hard to tolerate the lack of sanity often found in the wild.

As an admin one sometimes has to work with legacy installations that have "always" been there and because things are kinda running right now we can't change them.

One of my personal favourites is the handling of init scripts, because for some reason even most linux distros seem to hate them with a passion. I have no idea why you'd want to make it more likely that stuff fails, but I guess it also keeps admins busy.

On OpenRC you can be totally lazy:

rc-update add varnishd default
rc

Now magically rc will start all services that are missing for this runlevel, and as you just added one it'll start that one plus all dependencies that weren't started yet. Tadaaaah. Done.
Of course you can restart it:

/etc/init.d/varnishd restart

Unsurprisingly this restarts ... that service. And all services that might be depending on it (although you might be able to disable that smartness as it sometimes gets in the way)

Now here's some of the little things that make Ubuntu unbearable for me:

root@webserver-test:/foo# /etc/init.d/lighttpd stop
 * Stopping web server lighttpd
[ OK ]
root@webserver-test:/foo# /etc/init.d/lighttpd stop
 * Stopping web server lighttpd
[ OK ]
root@webserver-test:/foo# /etc/init.d/lighttpd stop
 * Stopping web server lighttpd

So now I can't figure out if a service runs, and I can just run the "stop" action of the init script and it makes no difference. A "This service has already been stopped" would have, like, optionally, be nice. Y'know?
Now let's increase the difficulty: I want a service to autostart on boot.

# update-rc.d
usage: update-rc.d [-n] [-f]  remove
       update-rc.d [-n]  defaults [NN | SS KK]
       update-rc.d [-n]  start|stop NN runlvl [runlvl] [...] .
       update-rc.d [-n]  disable|enable [S|2|3|4|5]
                -n: not really
                -f: force

The disable|enable API is not stable and might change in the future.

ok, uhm, thanks. Let's try this ...

# update-rc.d supervisord enable
update-rc.d: warning: supervisord start runlevel arguments (none) do not match LSB Default-Start values (2 3 4 5)
update-rc.d: warning: supervisord stop runlevel arguments (none) do not match LSB Default-Stop values (S 0 1 6)
 System start/stop links for /etc/init.d/supervisord do not exist.

I guess that is an indirect way of saying "no" ? Why does it warn me, and why didn't it tell me properly that it couldn't do what I asked it to do?
In other words, WHY U NO SENCE? CAN HAZ MAEK WERK?!

# update-rc.d supervisord start 2 3 4 5
update-rc.d: warning: supervisord start runlevel arguments (3 4 5) do not match LSB Default-Start values (2 3 4 5)
update-rc.d: warning: supervisord stop runlevel arguments (none) do not match LSB Default-Stop values (S 0 1 6)
update-rc.d: error: start|stop arguments not terminated by "."
usage: update-rc.d [-n] [-f]  remove
       update-rc.d [-n]  defaults [NN | SS KK]
       update-rc.d [-n]  start|stop NN runlvl [runlvl] [...] .
       update-rc.d [-n]  disable|enable [S|2|3|4|5]
                -n: not really
                -f: force

The disable|enable API is not stable and might change in the future.

OMG U EGGSPACT ME TO KNOW TEH ORDRE? OMG OMG OMG I R GATLING DRUNKERATED! NAO!
also ... why ... dot ... at end? is no parse for argument? U LOOKING FOR ARGUMENT? I GIEV YOU NUCLEAR FIRST STRIKE ARGUMENT!

At this point my sanity is suffering too much, I see artefacts of Really Bad Shell Scripts, overcomplicated output and I still haven't managed to autostart it (and since I don't KNOW the dependencies I can't even be sure that it'd be starting properly! Omg, is this liek the 1980s ?!)

But here's a doubleplusgood bonus:

# apt-get install varnish
[...]
Setting up varnish (2.1.3-7ubuntu0.1) ...
 * Not starting HTTP accelerator varnishd
[ OK ]

What the smegging fnark! EVERY other init script gets autostarted, and now this one suicides away and I have to figure out why it is speciuhl.
So ... uhm ... tehehehe ... wheeee. Sorry, mind malfunctioning a bit. Let's look into the init script:

disabled_varnishd() {
    log_daemon_msg "Not starting $DESC" "$NAME"
    log_progress_msg "disabled in /etc/default/varnish"
    log_end_msg 0
}

Say what? Start does not start until you configure start to start the start start start
Sorry, got stuck again. I don't understand this insanity, but it cthuhlhu's my ft'hagn badly.
What's in the /etc/default/varnish ?

# Should we start varnishd at boot?  Set to "yes" to enable.
START=no

OMG SANE DEFAULTS
So unlike every other init script this one has a random protection so you don't accidentally purposefully start it. Which makes sense, because ... it is basically like all others. Just different!

As you can see this insanity turns a 2-minute operation into a confusing frustrating experience that is best served with a bottle of Vodka. And then if you run the "stop" function of the init script the daemon is still running, so you use killall just to notice ...

-bash: killall: command not found

Le Sigh. C'est trop magnifique!