Sordid (well, not really) tales of Brian Harring, aka ferringb

Whee, another confcache bugfix release. Version 0.3.3 is up in usual location. Adding it to portage shortly, since the bugfixes against 0.3 have been pretty minor, plus 0.3.x will sit for a bit while some refactoring occurs to better handle doing tricks to avoid invalidation of the cache.

Many thanks to Ed Catmur, Christian Lemke, and Diego 'Flameeyes' Pettenò for feedback and bug reports.

Confcache, version 0.3. Fixed up a bunch of ass biter issues in dealing with the sandbox when trying to start it up (versus confcache being called within a sandbox env); pretty confident it now works fine under userpriv, and standalone as of version 0.2

Wound up having to introduce a lovely little bit of twisty code- certain sandbox vars defined prior to starting the sandbox result in the sandbox not appending the usual sane defaults (allowing write to /dev/tty for example). The solution? Well, we shift those vars to the side, spawn an indirection script that appends them to the sandbox default, then execs the actual configure call.

Kind of nuts, but does the trick.

Aside from the quicky version 0.2 release, 0.3 was released a while after. Closed up some nasty gaps in staleness detection. At this point, I don't know any way to trip up the detection code- still get some occasional failures (mono for example), but the failures so far haven't been confcache's fault, issue has been that the package doesn't properly support --cache (which confcache uses).

Aside from that, rewrote the portage integration patch- back out the original if you have it, and raid the v3 from the usual place.

Addressing a couple of questions-

Why was configure complaining about stale build_alias/host_alias?
Bug, namely. Version 0.3 now digs through the args passed to configure, and (should) properly handle staleness detection for that facet of configure caches.

Why isn't configure caching all configure tests?
Because the configure script has to be written to cache results. A lot of default checks are cached, but a lot of package specific checks aren't written to cache (yell at upstream in other words).

Why isn't confcache working for ebuild xyz?
The current integration is active only for econf (it's the only place I can force it). You can do a nasty little trick however to have confcache step in in the meantime. Add

function ./configure() { $CONFCACHE ./configure; }

to your /etc/portage/bashrc. Shouldn't break stuff, but if it does it will be obvious- additionally, don't blame me. It's an evil hack, you've been duly warned :) .

Why is the global cache invalidated every run when I have cpufreq enabled?
Cpufreq changes the proc frequency (good thing), this info is exposed in /proc/cpuinfo, a file configure scripts check. The md5 changes every time the proc changes frequency, thus invalidating the cache each time (bad thing). I'm aware of it, and intend to address this in 0.4.

Offhand, probably end of the weekend for 0.4- need to introduce a framework for registering triggers to override the normal md5 checks for certain files, so it'll require a bit of expansion to confcache.

Continuing to pillage year old patches out of the ebd portage branch, parallel-fetch.patch is available.

Roughly, this patch adds FEATURES="parallel-fetch", which when enabled (and merging more then one package), splits the buildplan execution into two processes- one building, the other doing strictly fetching.

Equivalent to (emerge -f target &> /dev/null < /dev/null &); emerge target , just saner (verifies distlocks are available for example).

Testing and feedback, as always, is desired and appreciated.

Got a little ticked off at the runtime cost of configure calls for package updates, so I wound up rewriting confcache from the ebd portage line as a stand alone tool (score, I can use it locally for development), and split a patch for portage stable integrating it as FEATURES="confcache". Ebuild, and portage patch are available in my devspace.

Rough example of the improvement run time wise, is a reduction of php's configure call from 80s to 25s. About what ebd was getting, if I recall correctly.

For those unfamiliar with what this feature/code is actually doing- autoconf scripts have the option to cache settings in a file, and reuse that file next time around. Save the result of checks/tests, basically, which can result in pretty massive speed ups sometimes.

The problem is that autoconf doesn't offer a way to quickly/cleanly verify the cache- for portage, this is a major issue since a glibc update should most likely invalidate the cache; you don't want configure tests using old results that are no longer correct.

Enter the sandbox. :)

Via the sandbox, we can track what the configure script accesses for it's tests. Store a list of the files checked, and their md5 checksums, and you have a way to check if the cache is stale or not. It's actually a bit more complicated (need to track env vars also), but that's the basic jist of it.

Either way, looking for testers and feedback- the gain from it should be obvious, so I'll avoid the usual pleading to get people to test stuff ;)

For the non-gentoo folk, well, grab our sandbox, install it, and install confcache and you will be able to use it to manage a global cache for configure calls. Still useful, although y'all will have a few more hoops to jump through ;)