Sordid (well, not really) tales of Brian Harring, aka ferringb

Quicky, anyone who has interested, tarsync.tar.bz2 was updated with --exclude support. Globbing rules, if the glob spans multiple directories (app-accessibility/SphinxTrain) you get to be the lucky person to test it, since my main concern was blocking deletion of distfiles and packages.

Meanwhile, sped up it's updates a bit, but probably not noticable due to the exclusion support. Unless I write some code to collapse all passed in globs to a single glob or regex, their will still be overhead.

Either way, poke at it a bit. Should be useful for anyone stuck in snapshot land.

Few weeks back, started fooling around with syncing an ebuild tree directly from a tarball. Dodge the the untar to tmp, then rsync from tmp to the live dir, basically. Originally wrote a prototype in python, although hit a few issues- the tarfile module is pretty nifty, except it doesn't seem to be designed to work with tarball's with a large amount of entries. We have roughly around 110,000 files in our tree, and using tarfile to work with said tarball wound up using well over 120mB of memory on my box. Yikes.

Issue came down to a lot of extra stuff they do, tracking inodes for tar entries for example, which is useful if dealing with hardlinks. Also jacks up the memory usage pretty heavily though. Either way, moving on...

Rewrote the bugger in c, lifted a fair chunk of diffball's io, tarball, and option handling code (screw reinventing the wheel), and glued together a slightly frankenstein like implementation. Source is available here, and is mostly complete exempting a couple of warts I need to remove. First and foremost, lack of --exclude support. Kind of need that, otherwise the bugger would delete your portage/distfiles directory on you, which isn't friendly. Needs slightly better owner/group handling- it tries to enforce what the tarball states as owners by default, rather then trying to work within what the user is capable of. Can override the owner/group it tries to assign via commandline options, but a saner default would be useful.

So... at the moment, if you're not storing your distfiles in $PORTDIR, it's a possibility. If you are, don't try tarsync against your live $PORTDIR till I add exclude support :) . Bit of code cleanup, and removal of warts needed, but once that's done intending on adding it into emerge-webrsync and emerge-delta-webrsync as options. It's much faster avoiding the whole untarring, then rsync'ing the dumped tree to the live tree. That and the use of 190mB+ for the tree for untarring is a serious drag, so dodging it is nice.

Either way, might be of use to others, bugs/suggestions/whatever feel free to send my way.

For those who have been following the 'cabal censorship' claims, it has morphed into a statement of no proof ("I don't log"), and an attempt to justify antisocial behaviour towards others via the claim of having the stones to state your mind. Relevant quote-

I should start keeping IRC logs. Would be fun to post all the incriminating material and allegations that are made behind people's backs. But then, I'm not that kind of person. If people don't have the guts to say in public what they think about certain Gentoo developers in public, that's their decision.

For those who aren't aware, sadly there is a gentoo ettiquete guide that applies to devs; either you can view it's existance as proof of another step towards beuracracy, or as a reactionary measure to incidents requiring that at least general common courtesy/decent attitude towards others has to be spelled out.

The basic notion behind the guide is that by becoming a dev, like it or not said dev becomes a representative of gentoo. People don't recall their experience with a group as a whole, their interaction with specific members of the group serve as the basis for their opinion of the group. Bluntly, one bad apple in a group casts a bad shadow on the group as a whole.

Continuing, by becoming a dev you become a representative of the group in whole- doesn't matter if it's volunteer work or not, it still is a basic social group, and has it's own conventions/requirements. In laymens terms, by joining the group you agree to the implicit (now explicit via ettiquete guide) rules of the road. In gentoo's case, courtesy towards others is required; as the guide states, and common sense would dictate, it's not hard and fast rules, the rules are essentially between the lines. Play nice if you want to play with us is the implicit assertion of that document.

Having the balls to stand up for what you believe in is one thing, but it does not justify abuse of others. There is a very large difference between stating

"Dev xyz doesn't always deliver on what he has talked up"
and
"Dev xyz is only capable of hyped up bullshit that will never get anywhere"

Both state your opinion. The difference is that the latter is an example of flagrant disrespect for others. This is justified via this claim of "having the guts to say in public what they think about certain devs". This is the assertion being used to try and justify trollish behaviour.

Something to consider. I'm "having the guts to say in public what I think about certain devs", and I'm managing to do it without resorting to catcalling, flagrant attacks, or nebulous claims about supposed censorship. I'm simply pointing out that ciaran as a general rule of thumb has trollish tendencies, and aparently just levelled an unsupported accusation of 'censorship' at devrel.

No swearing needed, nor namecalling. The point stands on it's own.

So, according to ciaranm he's been told to not state that pathspec/gentoo opensolaris are vapourware, posted via this entry. Stuart picked up on it, and did a bit of digging into if it's vapourware or not, here. Personally, I stated about 2 months back that pathspec was functionally vapourware (no code at the time), although bug 87877 was opened since, providing the basic prefix modifications required for installation akin to fink's /sw layout.

Ultimately, doesn't matter if the projects are vapourware or not, the thing devs should be concerned about is whether or not the accusation is true. Specifically, that devrel in no unclear terms (implied via his post) told him not to comment on the projects, in effect, that they attempted censorship. The implication of this is not pretty.

Thing is, we've heard strictly ciaran's take on it, and he's known for being heavily against devrel (cabal namecalling anyone?). Perhaps the reality of his claim is that he was told to attempt basic civility towards our users, devs, and projects, rather then the type of attitude displayed in this thread that just happens to touch upon all the things he was supposedly told not to speak ill of.

Jochen Maes (Sejo)'s statement of "stop bashing people!" within the thread lends a bit more credence to the possibility the claim of censorship is a bit misleading, and the request was to lay off the bashing/attacking of others. While the statements made in said thread might be true, they certainly weren't stated in anything but the most disparaging terms possible.

So... finally got my ass moving and did a fair amount of maintenance/upgrade on emerge-delta-webrsync (v2 is in the tree now also). Added in a few command line opts also; -h (guess what that does), -u (sync only if a new snapshot was found), -k (don't cleanse stale snapshots).

Aside from happy command line opts, cleansing of stale snapshots is now handled, and bzip2 v1.03 vs bzip2 v1.02 issues should now be addressed. Basically it tracks it's own local md5 if it detects a difference, and verifies the generated tarball to discern if it's md5 correct uncompressed (which it best be), and then verifies if the md5 after compression matches. If it doesn't, resorts to tracking the new md5 itself. Should work pretty much seamlessly, sans being a bit slower due to extra md5 checks.

So... eagle, which generates the rsync snapshots, lost it's marbles slightly on 04/29, and didn't generate a snapshot. Not an issue for emerge-webrsync, because it just grabs full version and works from current day, decrementing by a day till it finds a snapshot it can pull. A missing snapshot doesn't screw with it, just the users- no snapshot for that day.

Patches + webrsync, however gets screwed with, heavily. It's a break in the upgrade path for lack of a better description. No 04/28->04/29 patch, and no 04/29->04/30 patch (all are based on a 04/29 snapshot). Emerge-delta-webrsync's algo for identifying a base version to use, and patches to use doesn't like that gap (even if a 04/28->04/30 patch existed), since it goes day by day. Bleh.

Server side, I can mangle the bugger so it automatically compensates for missing snapshots. To do it properly, emerge-delta-webrsync needs some modifications (another release), yay. Meanwhile, generated a 04/28->04/29 patch (it's actually 04/28->04/30), and generated a 23 byte patch for 04/29->04/30 that is actually 04/30->04/30. It's basically just a big copy command. Compressing the sucker adds more overhead then the copy command (the command should be 12 bytes iirc).

Yeah, it's a hack, but it's the right type of hack- a working hack :-) . Meanwhile, once I'm back in mke, I'll get cracking on a more permenant/resistant solution after mirror-dist cruft is straightened out (namely, going live).

Sidenote, been generating patches for glep25, distfile diffing. Still ironing kinks out of 'file line' identification (I'll write a lil intro/blog entry about the whole problem another day), but have generated around 2gb of patches thus far, which is a lot, but not doing any filtering currently of when it's worth it, just generating patches for the file line. Re: "when it's worth it", if the delta is some percentage of the download, note that file-lines size issue, and down the line, adapt and stop generating patches for it if the patch is too large (larger then what would be fetched, or more likely, larger then some percentage of the target file). Case in point, OOo_1.1.4_LinuxIntel_install.tar.gz -> OOo_1.9.95_LinuxIntel_install.tar.gz patch weighs in larger then the target, as such shouldn't be using the patch. Short version, auto identification prior is tricky.