Patrick's playground

Good news: The AMD64 tinderbox finished one run. AMD64 is still significantly faster than x86 because it has less packages. Also AMD64 has a higher average code quality because it is relatively young and thus has less legacy stuff / leftovers.
Some devs are fixing the bugs about as fast as I can file them, especially Diego seems to be a script ;-) As a sideeffect I've been testing newer versions of many packages to see if the new version compiles (not runs, mind you, that's not yet my target). Maybe I'm an AT in disguise, who knows ;-) (although I must say that the ATs must be scripts too, they test stuff at a rate that is really stunning!)

Now, my bugfiling rate has gone down. I blame the rising quality of ebuilds as things either get eaten by treecleaners or fixed. So I hope to be unable to file bugs soon because there's nothing to fix - but then I could do USEflag-permutations and other nice things that will find other errors. Mmmmhfun.
I've been asked to test packages by a few persons, the tinderbox approach has some distinct advantages there. So I've been thinking about a webinterface that allows to register overlays, sync the overlay and compile everyhting in it - would be fun, but I lack time and motivation :-( I still haven't finished documenting and cleaning up the code, but work is turning into a soul-sucking endurance contest. I hope people will keep poking me so that I have a bad conscience and do more stuff ... haha. ha. hrmmm.

So I hope my motivation doesn't try to reach absolute zero, so much to do, so little time ...

My two tinderboxen are still running, but as I restarted them they have only munched through "old" failed packages. It's nice to see about 200 new packages compile just because dependencies get fixed, but it's getting annoying to wait ~12h just for it to catch up to the old position.
I've punted games-* for now as it is too interactive and my patience is limited. Will be fun to thoroughly analyze that part of the tree :-D

But I've found something to keep me busy: QA warnings!
I have so many logfiles to grep through ... I've filed something on the order of 20 bugs and have managed to get all pre-stripped files from the first half of the first of ten logfiles bugged. There are other QA things listed in the logfiles, I'll look at those when I'm done with this list.

I've managed to make another dev happy by giving lisa access to my server, she can compile and test AMD64 packages much easier now. That shows again that we seem to lack dev-boxen in gentooland ... might be nice to find some sponsors. But sponsors ... that's a very touchy subject. I was told that due to inactivity a sponsored box for Gentoo Infra was cancelled today, that's bug 108379 for those that want to see the history of that box. Quite a shame to see such a powerful box be decomissioned ... and that sponsor will most likely avoid Gentoo now.
Due to the time I personally invested I might be biased, but that was something that could have been handled a bit better. And with better I mean somehow at all, letting a machine rot for 6 months after demanding enterprisey features etc. shows a certain arrogance and also a lack of manpower. Not that I haven't offered to help ...

On a funny sidenote, about 85 bugs directly and 35 through CC:s have been orphaned as my bugzilla account has been locked. Jakub (our chief bugwrangler, I'd say) reacted with a big "WTF?" when I told him, seems that the normal policy is to change the email and take away rights from a bugzie account for retired devs, not kill it. But ... no longer my bugs, what do I care. Right?

With the many bugs I've filed I have also been asked to test new versions of packages, that has shown itself to be more time-consuming than expected. Tinderbox can run for days without needing attention, version bumps aren't that easy to script. Makes me wonder if I can automate that too :-) That might be appreciated by our amazing ATs too.

As a sideeffect of tinderboxing I get lots of binary packages for free. Right now I have 2540 for AMD64 and 3327 for i686 - that should be roughly 30% of the tree compiled (in fact it should be a bit less as there's a small chance of multiple versions per package).
I'm building off an almost stock profile (needed to add a few useflags to work around silly defaults). So now I'm wondering - do I want to add a "minimal" profile? Do I add a profile that has all the mad stuff I use locally so that I don't have to compile on my "slow" box?
There's a distinct difference between tinderboxing (trying to build everything) and binpkgs for fast installation - those should be customized to fit the need. Otherwise you'd have to recompile a few packages locally ... two options: You either get enough processing power to churn out multiple profiles or you do a default profile and people have to compile a few bits themselves.
So the critical variable here is change speed - how fast does new stuff get added to the tree? Of course, it's not that simple - a new gcc might trigger a full rebuild of everything, that would take "long". Things like a new KDE release would also be annoying :-)
If anyone wants to have a look at my binpkgs, they are still here, with a big thanks to the people at my preferred french webhosting company.

... that's the percentage of all bugs I filed this week. Roughly 18% - and most of them quite simple things that everyone should be able to find.
A bunch of interactive stuff, masked locally
4 infinite builds that just won't finish
A few gcc porting bugs
Quite some totally broken stuff from, like, 2001
A few false positives which I need to track down
About 30 test-requests for bumps

I'm up to kde-base on one box - gnome stuff is mildly absent because of b0rked deps. Quite silly when it can't build out of the box ...
To be more precise: gtk-2 needs cairo with USE="X", libgnome has missing automake dep and gnome-libs is missing an xpm dep. So that's, again, quite unconvincing, but I guess I'm personally biased as Gnome is very ...

Ok, different topic. That is too emotional :-)
I'm at home because I have a nice fever and am unable to work. So I'm relaxing and doing not much ... I've hacked up a few code fragments to mangle ebuilds with pkgcore, that turns out to be quite easier than expected. Big thanks to ferringb and all his helpers for creating such a beast that is faster and easier to tame than portage. What it still lacks is some nice UI bits and docs from my point of view - the rest seems to be quite feature-complete and stable. Haven't managed to really make it barf yet.
Ah yes. I'm still running a simple naive nanoblogger instance for my blog, so no comments. Haven't found the motivation to work around all the spam issues etc. So, unless I work through my nice backlog in the next 24h (which is somewhere between unlikely and impossible) you won't be able to comment on my blog.

My primary build machine has quit tinderboxing for now, I forgot about one tiny bug and was too lazy to inject my overlay with the "fixed" ebuilds. The second machine went up to games-action and then went interactive. So it seems that the games.eclass calls the eutils.eclass to show a license. Only problem:

It's interactive unless you do some tomfoolery with ACCEPT_LICENSES. The sanest way to handle it would be to abort, tell the user to RTFL and add it to ACCEPT_LICENSES in make.conf. But no, that is too smart, instead we break lots of stuff by asking for keyboard input.
Stabstabstabstab.

I feel tempted to focus on games-* just to find all the horrible QA mistakes and policy violations. That stuff should just go into an overlay ... (or at least that's my opinion).
My bugfiling rate has gone down a lot with AMD64 being relatively clean and categories like dev-java giving many false positives. Also I'm at home now since I have a nice fever and can't work.
My estimates look quite nice, I think I can build a full tree in about three weeks and do update cycles of 5-8 days. That's not too bad for one machine, but then again I'm still using a "naive" compile script that does a few bits slightly redundant.

The dynamic-deltup-server I've made available is churning out deltas at a nice rate, but I still find that idea mildly flawed. GLEP 25 was much more fun :-)
The estimates from long long time ago were about 3 days for initial delta generation and then just a low amount of processing power to keep it updated. Shouldn't be that hard to resurrect ... and it would be nice to save 50-95% of traffic just because.

The last few hours of compilation were quite boring, .netmono doesn't compile because of missing pkg-config deps, so that goes nowhere. Java is tricky - as I found out with the help of Caster all gen 1 Java ebuilds will fail. There's not much I can do about it, but I don't care enough. Still so many other things to compile :-)

Work is slowly driving me insane. Sometimes it is cheaper to just buy a replacement part for 500Eur ... but that would be logical, and we can't have none of that here.
Package of the day is dev-lang/smlnj - it's up to 1 CPU-hour and 1,8G during compilation. Yes, that's right. Over 1800 Megabytes of my precioussss RAM used for one process. Really nice and WTFy.

The second box is online and crunching packages. I'll have to setup an rsync job to sync binpkgs to the other server ... but I have to say that AMD64 is marginally cleaner than x86. More stuff compiles, much of the same stuff is just so broken it should be punted. All in all quite interesting to see some categories compile close to 100% and others ... well ... infinite compilation processes etc. Let's not discuss this :-) Too frustrating.

Still have a few ideas to test, but I'm lacking processing power. The old AthlonXP 2600+ won't boot at full speed and is down to 2000+ since quite some time, and it overheats after ~12h of compilation. But a new box is too expensive - at least for what I want. That would be an X2 with so much memory that I can't complain anymore :-)
Anyways, need to get a few things beaten into shape. Every day seems to be an uphill struggle against so many things that need to be done, but I won't give up. I'm too dumb for that ...

I've reached app-emacs on x86. One of the weirdest categories until now ... Looking at CPU load it seems to wait on _something_ as it averages maybe 10% during emacs-lisp code compilation. Really weird :-)

Looking at bugs it's going really well, it's quite nice to see stuff finally compiling in the logs. Also finding many totally abandoned packages ... treecleaners material :-)
Box number two is up and running, I'll have to take it down for now as it is too loud. But now I have x86 and AMD64 dedicated buildmachines!

After a look in the logfiles I put in a robots.txt ... and guess what, traffic dropped down. A lot. Friggin MSNBot and Googlebot eating through my binpkgs ...

As a response to my complaints about interactive ebuilds we now have GLEP 52, that is most excellent. But ... look at this quote from the bug:
"...and? Of *course* they're interactive. They pull data from a CD."
That coming from a developer is scary. I mean ... wow. Even better:
"They are interactive by design and will stay that way. What exactly is this proposal about?"

Dude... this proposal is about not waiting an eternity per ebuild. I'm a little short on eternities and infinites. Argh ...

I've hit a few interesting conditions:
- java-config seems to need manual configuration, so most java ebuilds fail right now (but no bugs files as long as I don't understand why/how)
- stuff eating lots of memory - dev-lisp/sbcl takes ~850M
- stuff running infinite loops. Against all my hopes and wishes I have a quite large package.mask already

Apart from that ... working on a few things when not sleeping. Work is slowly killing me, I averages about 16h of sleep per day for the weekend. Need to get back to a sane sleep rhythm (but getting up at 8am is definitely insane for me)
Reading seemants blog about his fasting / diet, I feel tempted to add such stuff to my blog too. But that's nothing interesting I guess.

Noticing I'm in braindump mode. Tired. So much to do, so little time. Need to insulttalk to a few people. Need to do more automation. Need sleep.

Yesterday I found out that gerrnynjr has resigned. Teh suck :-(
It's quite frustrating to see so many people being bled off (but in this case it is mostly due to time reasons, so it's not that bad)

On the bugfiling front I've managed to file more than one bug per hour for a week. I've finished app-portage, so I'm through about 22 of 161 categories with a moderate coverage of the rest. OpenOffice went amazingly fast on the compile box - big thanks to my sponsors for providing such a monster. I really need to get these guys more publicity :-)
I've been punted from the planet.g.o feeds, that was quite nasty - Kugelfang had to poke beandog to remove me and the "nonsense" I'm writing. So, ok, Kugelfang, you're a banana, don't let facts get in your way.

My sleep patterns are becoming really messed up thanks to work, so it's time for plan B (top sekrit). So much to do these days ... But it's worth it. World domination, here we come ...

from Wikipedia:
"A cohort selected for punishment by decimation was divided into groups of ten; each group cast lots, and the soldier on whom the lot fell was executed by his nine comrades, often by stoning or clubbing. The remaining soldiers were given rations of barley instead of wheat and forced to sleep outside of the Roman encampment."

I've managed to file 30 bugs since yesterday evening, which gives me a current rate of about 3 bugs per hour. Flameeyes is closing many simple dep bugs at a stunning rate, that's quite motivating. Many other devs also respond really fast, so a big thanks to them.
Nelcheal even thanked me for filing about 70 bugs for desktop-misc, that's a bit weird :-)

As I'm approaching app-office/openoffice I expect a heavy slowdown in bug generation, but on the upside I've managed to munch my way through about 1/3rd of the tree in about 10 days - my estimate of 3 weeks for the initial run wasn't that far off. I might be able to increase the speed a bit by making the compile script smarter, but that'll be quite messy I think as it involves some depgraph mangling.
I hope to get the second AMD64 working soon, then I'll be able to work on 2 arches at the same time. Binpkgs are growing nicely, so I'm thinking about doing some cleanup so that I provide a good binpkg repo.

Catalyst dies in a different place now, building stages is still very fragile. Nothing unfixable, but another slowdown.
The dynamic-deltup server I'm providing for blackpenguin seems to be crunching out diffs at a really nice rate, but I'm mildly unhappy with the design. So I might try to implement a GLEP25 prototype just because ;-)

A big thanks to those people that still motivate me to continue, I hope I can help you at some point in the future.

So Stuart has responded to my blog posts. Quite unexpected, but there's one thing I find really interesting. Quote:
"Oh, and whilst we're dealing with wild claims on Planet Gentoo ... for the record, it's Daniel we have to thank for Planet Gentoo, not Patrick. He's the one who took the Planet Gentoo idea and worked with Infra to deliver it."
Stuart, that is correct. But also not the complete picture: Before infra accepted the planet.g.o idea there were long discussions wether it is worth it, how much administrative work it would be etc. etc. And it was really amazing to see the prototype running ... before infra did anything.
I really appreciate the work people did - but from my gentoo-core mail archives (and any Gentoo dev from that time can confirm) the first mention of planet.gentooexperimental.org was 23 Jan 2005 by karltk. The first announcement of the official planet.gentoo.org I could locate was 07 Mar 2005 by dsd. So say what you want, planet.gentooexperimental.org was first.

Today has been rather slow on bugs for me (only about 15 I think), but I blame the muppet who put cdrom_get_cds into the eutils.eclass. It violates all policies about interactive ebuilds very nicely and breaks scripts (hint: I can't press ctrl-c if my script gets the input before portage does, so don't do that ...) Since about 20 ebuilds are affected I think my stats for today are acceptable, especially since I lost about 10 hours waiting for someone to insert a cd ...
I hope the suggestion to RESTRICT="fetch" these ebuilds gets implemented, otherwise I'll have to grow a big package.mask. And that I wish to avoid because then I'm no longer compiling stuff from the tree as it is shipped ...
I've also resumed toying around with catalyst, I hope this time it manages a run without failures (small perl bug killed the last two or three tries) - would be nice to have up-to-date stage3 available (and ~arch stages for teh ricers, but that might be unpopular)
That's all for today, I hope y'all have fun and don't take life to seriously. It's much too short for that!

So ... this will most likely the last communication from me syndicated to planet.gentoo.org etc. Thanks to all who read, responded and discussed my random rants with me.
I'll be back.

So ...
I'm slightly annoyed at the behaviour of some people. Obviously. But that's to be expected ... anyways, there's a few other people that got retired without being consulted - that's very not nice, especially since over 20 people got removed retired at once. That's ... what percentage?

I've had to improvise a bit since my bugzilla-foo has been punted, so now bugwranglers have to handle each and every bug I file. Only 26 (or 27?) today, slower than expected. But I think I can keep up that rate of 1 bug per hour for quite some time :-)
Slightly annoying that my mail account is locked. There's not much of importance there, only all my gentoo mailing list subscriptions and my bugspam. *shrug* makes it hard to respond to bugs (especially your removalretirement bug ... ), but not my problem I guess.

So. erm. Something completely different.
There's people that are not Gentoo devs that have access to Gentoo CVS.
OMGNOES WTFBBQ !!!!!1!!!
so, like, someone forgot to disable them after they retired quite some time ago ... but I get removed with such speed and accuracy that I feel honoured to be so important.
Not to worry though, the tree signing GLEP that people have been working on is almost done. I'd guess a week or two until it is released, but that may be optimistic. Evil committs will be mildly more difficult when that stuff is implemented.

On a sidenote, I must really thank all the users and devs that expressed their support. You guys rock.
Even when devrel members close bugs about them (hello? I'd appreciate some unbiased forum to express complaints) there are people that remind me why Gentoo used to be teh bestest!!!111!.
So ... many things not going right. People violating policy for their egos. People getting devrelled just because, and other people being forgotten. Very annoying. But who cares, I'm going to continue what I'm doing. I still have good ideas for regexpen I want to run over the tree, but they have too many false positives. Tinderbox running. Work sucking me dry. Stuff happening. Me ranting.
I do remember one funny bit right now - planet.gentoo.org was stalled because people were unsure if it was worth the effort. So ... what happened was that karltk got fed up, spent seven and a half minutes installing and configuring planetplanet and made planet.gentooexperimental.org available to the public.
The rest, as they say, is history ...

As the avid readers of my blog have noticed I've been really really inactive, so kloeri decided to retire me. And because our infra people are usually fast as lightning ramereth has already kill0red my accounts, taking down my email adress in the process. So I'm not getting the bugspam. Haha. ha.

So I'm a bit scratching my head, I mean ... people ... can we not communicate with each other? I'd appreciate to be asked about things involving me. We'll see what happens next ... but infra won't get any further servers donated through me. After 6 months that machine (bug 108379 for anyone who cares) is not configured, but deactivating my account takes all of 3 days or so.
So a big thanks to all the people who stop my life from getting boring ... and at least I have a reason now to not file a few hundred bugs. *shrug* I think I don't have to understand that, but that's the way the banana peels.

After fixing a few of the remaining bugs (like not catching all exceptions) the tinderbox seems to be running very nicely now. There's a few ebuilds it won't compile as it is now, but I've found less than 5 that fit this little parser error until now. I think I can live with that ...

I think I want to rewrite the logfile handling a bit to put all failed package logs in one place. Right now it's a bit annoying to look through the directories with many succesful compiles and finding the few that fail. Also I need to make it smarter so it won't try to recompile a package that already failed, that is silly and only takes lots of time.

After 5 days of running I'm close to 100 bugs filed. Many missing deps - in fact about 80% of the bugreports are either pkg-config or automake missing. The rest is mostly gcc4 being stricter or Xaw/xpm deps missing, but there are a few totally b0rked packages that just can't ever work like this. Devs are mostly responsive and fix the simple dependency errors quite fast, that is good.

Because of a design oddity (I'm walking the packages alphabetically backwards) most of the bugs until now were in x11- (I'm at sys-apps now). The desktop-misc herd is most likely really frustrated because they got at least three quarters of the bug reports until now ... sorry guys, I hope I'm done with filing bugs for you.

Now I really hope to be able to compile on more arches soon, a few people have offered me access to their boxen, but one machine is offline and one person is really busy these days, so nothing yet.

My first estimates were around 3 weeks, I think it'll take a bit longer for a "full" compile-it-all run. So what I do now is to sync and re-start the tinderbox every now and then (as it used to abort on some ebuilds I haven't had it running for more than 24h continuously). The tinderbox code lacks comments, is not very clean and annoys me in some places, so if I find some motivation I'll make it presentable and hope that other people have fun with it too ...

More or less by accident I stumbled over some quite interesting things in ebuilds. All the things listed below were found in an unmodified /usr/portage. I think it shows quite well that even Gentoo devs are only human ...

The Binary Decision:

yesno() { useq $1 && echo yes || echo no; }

yesno() { if $@ ; then echo y ; else echo n ; fi ; }

We need more yesno() functions!
A nice substitution:
echo_setup() {
# All values must be double-quoted, so escape the inner quotes.
echo "${1}=\"${2}\"" >> ${3}
}

We also need more informative optimistic scripts:
echo "Perl script! Woohoo! No need to compile!"

Now ... let's not use einfo/elog so that people looking at the logfiles won't ever figure out what to do:
eerror "Please do:"
echo FRITZCAPI_CARDS="fcpci" emerge fritzcapi

Also nice: interactive buildsystems from upstream

echo 'y' | ./unixmake config || die
That's silly - what if you ever need to disable one option?

echo "" >> answer
echo "" >> answer
echo "" >> answer
echo "" >> answer
echo "" >> answer
echo "" >> answer
echo "" >> answer
make < answer || die

If you look around you'll most likely find more cases of such absurdity. But then there's also the "We don't care" camp:

echo /opt/kylix3 >> /etc/ld.so.conf

Or the NIH people that don't like EINFO and friends:
NO=$'\x1b[0;0m'
RD=$'\x1b[31;01m'
GR=$'\x1b[32;01m'
YL=$'\x1b[33;01m'

echo "${GR} *${NO} Checking Configuration."

So I hope noone feels attacked by this, just pointing out some unconventional bits I've found :-) Maybe we should have a forum like TheDailyWTF.com, one of the bestest websites I've found (and which motivated me to search around in the tree myself) ...