So, RIP marty :-/

Many thanks to Ed Catmur, Christian Lemke, and Diego 'Flameeyes' Pettenò for feedback and bug reports.

Wound up having to introduce a lovely little bit of twisty code- certain sandbox vars defined prior to starting the sandbox result in the sandbox not appending the usual sane defaults (allowing write to /dev/tty for example). The solution? Well, we shift those vars to the side, spawn an indirection script that appends them to the sandbox default, then execs the actual configure call.

Kind of nuts, but does the trick.

Aside from the quicky version 0.2 release, 0.3 was released a while after. Closed up some nasty gaps in staleness detection. At this point, I don't know any way to trip up the detection code- still get some occasional failures (mono for example), but the failures so far haven't been confcache's fault, issue has been that the package doesn't properly support --cache (which confcache uses).

Aside from that, rewrote the portage integration patch- back out the original if you have it, and raid the v3 from the usual place.

Addressing a couple of questions-

Why was configure complaining about stale build_alias/host_alias?
Bug, namely. Version 0.3 now digs through the args passed to configure, and (should) properly handle staleness detection for that facet of configure caches.

Why isn't configure caching all configure tests?
Because the configure script has to be written to cache results. A lot of default checks are cached, but a lot of package specific checks aren't written to cache (yell at upstream in other words).

Why isn't confcache working for ebuild xyz?
The current integration is active only for econf (it's the only place I can force it). You can do a nasty little trick however to have confcache step in in the meantime. Add

function ./configure() { $CONFCACHE ./configure; }

Why is the global cache invalidated every run when I have cpufreq enabled?
Cpufreq changes the proc frequency (good thing), this info is exposed in /proc/cpuinfo, a file configure scripts check. The md5 changes every time the proc changes frequency, thus invalidating the cache each time (bad thing). I'm aware of it, and intend to address this in 0.4.

Offhand, probably end of the weekend for 0.4- need to introduce a framework for registering triggers to override the normal md5 checks for certain files, so it'll require a bit of expansion to confcache.

Roughly, this patch adds FEATURES="parallel-fetch", which when enabled (and merging more then one package), splits the buildplan execution into two processes- one building, the other doing strictly fetching.

Equivalent to (emerge -f target &> /dev/null < /dev/null &); emerge target , just saner (verifies distlocks are available for example).

Testing and feedback, as always, is desired and appreciated.

Rough example of the improvement run time wise, is a reduction of php's configure call from 80s to 25s. About what ebd was getting, if I recall correctly.

For those unfamiliar with what this feature/code is actually doing- autoconf scripts have the option to cache settings in a file, and reuse that file next time around. Save the result of checks/tests, basically, which can result in pretty massive speed ups sometimes.

The problem is that autoconf doesn't offer a way to quickly/cleanly verify the cache- for portage, this is a major issue since a glibc update should most likely invalidate the cache; you don't want configure tests using old results that are no longer correct.

Enter the sandbox. :)

Via the sandbox, we can track what the configure script accesses for it's tests. Store a list of the files checked, and their md5 checksums, and you have a way to check if the cache is stale or not. It's actually a bit more complicated (need to track env vars also), but that's the basic jist of it.

Either way, looking for testers and feedback- the gain from it should be obvious, so I'll avoid the usual pleading to get people to test stuff ;)

For the non-gentoo folk, well, grab our sandbox, install it, and install confcache and you will be able to use it to manage a global cache for configure calls. Still useful, although y'all will have a few more hoops to jump through ;)

Received a request earlier, which wound up with diffball 0.7 being released, and a large amount of lib work being done so that people can get access to a rather simple high level api.

At this point, the lib exposes the exact same set of runs that compromises the differ binaries algo, and exposes a reconstruct func that, again, pretty much comprises the patcher binaries internal reconstruction calls. An example of using libdiffball for doing delta compression between two files in memory, dumping the results to another array (note that the code is being anal about dealloc, hence the structure)-

#include 
#include 
#include 

// returns 0 on success, non zero on failure
int diff_it(char *src_array, int src_len, char *ver_array, int ver_len, 
  char **out_array, int *out_len)
{
  cfile src_cfh, ver_cfh, out_cfh;
  int ret = -1;

  if((ret = copen_mem(&src_cfh, src_array, src_len, NO_COMPRESSOR, CFILE_RONLY)) 
    == 0) {

    if((ret = copen_mem(&ver_cfh, ver_array, ver_len, NO_COMPRESSOR, CFILE_RONLY)) 
      == 0) {

      if((ret = copen_mem(&out_cfh, NULL, 0, NO_COMPRESSOR, CFILE_WONLY)) == 0) {
      
        if((ret = simple_difference(&src_cfh, &ver_cfh, &out_cfh, 0, 0, 0, 0)) 
          == 0) {
          *out_array = out_cfh.data.buff;
    	  *out_len = out_cfh.data.write_end;
        } else {
          free(out_cfh.data.buff);
        }
        cclose(&out_cfh);
      }
      cclose(&ver_cfh);
    }
    cclose(&src_cfh);
  }
  return ret;
}

To head off concerns about buffers of buffers, preferred access to cfile structs is via a page api, that exposes the underlying buffer, so as to cut down on unneccessarily cread memcpy's (for example).

The copen_mem above is configuring the cfile struct so that it's working directly from the passed in array- for CFILE_RONLY, obviously, doesn't modify the buffer. :) I'm opening the out_cfh with a null buffer, so that it allocs it's own.

From there... the simple_difference call is pretty straightforward. The trailing zero args are just telling it to use defaults for a couple of tunables (in this case, I have no interest in fooling with them).

What's all the nonsense involving pulling a buffer? Well, that's because I haven't yet thought up a good, *clean* api for raiding the buffer + size out; till that's in place, management of data.buff (namely free'ing), is left to the caller- this is also why I choose this example, since would welcome suggestions for it. Remaining chunk of the code is pretty much just free'ing stuff on the way out, error or otherwise, then returning either the error or zero.

That's the beast. Might seem a bit complex, but aside from the CFILE_WONLY + free design decision, it's about as simple as I can make it. The api reconstruct equivalent, 'simple_reconstruct' is roughly on par for straightforwardness-

int simple_reconstruct(cfile *src_cfh, cfile *patch_cfh[], unsigned char patch_count, 
  cfile *out_cfh, unsigned int force_patch_id, unsigned int max_buff_size);

So, enough whoring of that. Needed an easy way to test the new code from above, so I also wrote out some quick python bindings via pyrex- pydiffball-0.1 being the result. Exposed functionality is cfile creation (both memory and usual file based), simple_difference, and simple_reconstruct. The python module is quite a bit more friendly, since it'll generate the cfile instances on it's own for all api calls, assuming it's an on disk path passed in when the arg is a basestring derivative.

Hopefully people find some use in it; personally, the python bindings will at least make my life a helluva lot easier for some extra code I occasionaly work with.

Did some further improvements to it, lifting a couple of classes out of 3.x so as to cut down on unnecessary io overhead. Stats via FuzzyRay (Paul Varner), a 233mhz pentium with 256 megs and an amazing disk access speed of 9.74MB/sec. All tests are with 2.0.53_rc5 as base.

emerge --metadata with existing full cache

Version	real	user	sys
vanilla	9m30.580s	4m9.230s	0m17.850s
patched	5m43.876s	2m39.730s	0m13.610s

emerge --metadata without cache

Version	real	user	sys
vanilla	35m30.164s	26m49.250s	1m23.410s
patched	11m59.595s	5m6.890s	0m36.930s

Not too shabby. Stats source available here. Patch is available here, and further feedback would be appreciated.

Be aware if you test it out, you're going to need to run a emerge --metadata after applying the patch to /usr/lib/portage/pym. CVS $PORTDIR users, you're stuck running a regen.

1. EAPI awareness. Unset EAPI in an ebuild is EAPI=0, so no massive tree wide changes needed (grandfathered the existing tree in).
2. Support for upcoming rsync metadata changes, transparent switch over.
3. Glep31 checks, and enforcement.
4. CDEPEND metadata key is no longer looked at by the resolver (wasn't used).
5. SetUID/SetGID installed files will have their o+w yanked automatically on merging now, rather then complaining about it.
6. No more has_version/portageq in global scope. Do it, and portage intentionally pukes on your ebuild during sourcing.
7. prelink md5 calculation optimization via zmedico.
8. Good collection of bug fixes, plus cleanup of a couple of messages emerge spits out so it's saner.

Not the full list (generate the diff if you're after it, or raid the ChangeLog), but that's the notables. Please test it :)

Now, I thought the func was fairly tight, speedy enough. Simple little sucker-

def iter_tokens(s, splitter=" "):
    """iterable yielding of splitting of a string"""
    pos = 0
    l = len(s)
    while pos < l:
        if s[pos] in splitter:
            pos += 1
            continue
        next_pos = pos + 1
        while next_pos < l and s[next_pos] not in splitter:
            next_pos+=1
        yield s[pos:next_pos]
        pos = next_pos + 1

python -m timeit -s 'x="mamma said knock you out\n mama \t knocked\t you out\n";x*=10000;' 'list(iter_tokens(x, " \t\n"))'

What massively gets my goat is that it's actually pretty damn slow in most real world usage compared to a seemingly primitive, and butt ugly (imo) aproach.

from itertools import ifilter
def iter_tokens(s, splitter=" "):
    l = len(splitter)
    if l > 1:
        if l == 3 and " " in splitter and "\t" in splitter and "\n" in splitter:
            return iter(s.split())
        for x in splitter[:-1]:
            s = s.replace(x, splitter[-1])
    return ifilter(None, s.split(splitter[-1]))

python -m timeit -s 'x="mamma said knock you out\n mama \t knocked\t you out\n";x*=10000;' 'list(iter_tokens(x, " \t\n"))';

If it were a single split, still the replace hack is faster (although the difference between the two is minor enough). So... that's weird, and bugged the hell out of me last night :)

Zac Medico's comments about the yield instantiating and returning another string instance probably are fairly on par. Either way, it's not intuitive to me :)

Final comment on it, downside to the faster approach is that you have to do the processing up front, rather then JIT as the generator does- in the case of the code that uses this, it's not an issue though. Haven't dug into the underlying python source to figure out why there's such a difference, so if someone knows kindly tell me so I spend my time doing something else ;)

Update: Tweaked the replace func and updated it's runtime since it was brain dead from experimentation at 3am, saner/simpler version of the replace loop is courtesy Andy Dustman for the replace cleanup. The check for " \t\n" is a quicky addition from me, mainly since that is even faster.
Note also that the faster approach I don't have issue with, I'm just rather amazed at the major difference in runtime for the two approaches.

Detection of $PORTDIR/metadata/cache format- currently portage stable uses an ordered list of (implicit) key -> value; this makes it essentially impossible to ever remove a key, and makes addition of keys have a hard limit. Bad. So... the new format is an old format I hacked out a year back, flat_hash, (explicit) key -> value unordered. Nothing hugely fancy, but does allow us to jam stuff in without issue.

Increased flexibility requires us to version the cache entry in some way, so that we know if entries are incompatible with the version of portage reading it. Additionally, we should have been versioning the expected ebuild env (how it will be called, what funcs are available, etc) long ago. EAPI is that; additions/extensions to the ebuild spec result in a new EAPI standard, for example, src_configure addition is part of what EAPI=1 is. With EAPI in the cache, we can know whether or not the local portage version is capable of properly handling that cache entry. A higher EAPI (later portage release) may add new metadata; any portage version that doesn't support that EAPI must in some way mark the entry as "I know of it, but I can't use this ebuild".

So... in that jumble, essentially the rsync metadata/cache auto-detection allows us to move over to a more flexible format without causing cache horkages every time we change stuff (as has happened often enough), and EAPI allows us to to version those entries, so that EAPI aware portage versions can protect themselves from doing something stupid.

That and a lot of emerge --metadata cleanups got stuck in, hopefully killing off any remaining failures during cache transfer ;)

Also dropped root requirement for emerge --metadata. That always bugged the heck out of me, since it wasn't needed...