Sordid (well, not really) tales of Brian Harring, aka ferringb

Lot of work, and bug squashing leading up to 0.6. Several new options-

• --purgatory-dir ; instead of deleting a dead distfile, transfer it to a directory that is checked prior to fetching. Think of it as a way to hold onto all older distfiles for a long time, and still have them accessible
• --max-attempt; used to control max attempts for a specific uri against a mirror
• --disable-central-pid-manager; This is a concession to osprey.gentoo.org. The default design is a bit more efficient, polls the pids once every 0.25 seconds from a thread, notifying waiting fetchers if the pid they're watching has died. Osprey.gentoo.org, for whatever whacked out reason (likely my fault), doesn't seem to like reaping any zombie kids via waitpid(-1, WNOHANG). So... fetching is completely horked via that, but if you wait on a specific pid, instead of -1, it returns. This option reverts back to the older design of having N threads all polling, and checking on the pids themselves. It's less efficient, but I am a loss identifying what's going on with osprey.g.o, so this option exists...

Aside from new opts, continued code cleanup, and correction of a few oddities. Help output is properly wrapped under 80 chars for example (minor, but it bugged me). Main noticable issue fixed is ensuring that when sigint'd, mirror-dist shuts down as fast as it can (figure a second or two). Signal it a second time, and it kills itself, and all of it's kids. This is useful if you modify it, and the kids get away from you (which happened to me a few times to me, nothing like trying to forkbomb yourself :).

Meanwhile, it's available at http://gentooexperimental.org/~ferringb/mirror-dist/mirror-dist-0.6.tar.bz2, and now carries with it a rather crappy cvs2cl ChangeLog dumping.

Enjoy.

Fixed bugs people pointed out, mainly dealing with the code being written against portage cvs head, and not playing nice with portage stable (I hate you stable), version 0.5.2 of mirror-dist is available here. Enjoy.

Took a look through mirror.py that is distributed with portage, and did one of my usual "bah, I can do that better"... 36 hours later, a massive head ache from python's global interpretter lock + threading issues, and it's finished.

Available here, tentatively titled mirror-dist, it has a bunch of features that make it better then the default portage mirror script.

• Uses command line args, instead of relying on env vars to control features/settings
• Killable in under .5s. This is important- existing mirror script has no way to kill it immediately, threads will just run till they finish their current task, and notice they're supposed to be dead. Seppuku of threads is pretty much under .5s, so as soon as the switch has been 'flipped' stating, "shutdown", it bails pretty quickly.
• Mirror overrides via command line- you can specify a text dict file to override portage's thirdpartymirrors. This is useful for the actual gentoo mirrors, since trying to fetch an update from another master mirror doesn't make much sense.
• distfiles cleansing support
• death watch db. Basically, if a distfile isn't associated with an ebuild, you can leave the distfile alone for a configurable period of time. That distfile is recorded in a db. If an ebuild re-claims the distfile prior to the specified interval, the distfile is removed from the db. If no ebuild claims the distfile, and the waiting period has passed, it gets waxed. Think of it as delayed cleaning support
• Debug mode- acts as if it fetched and deleted what it would have normally. Useful for trying it out without having any actual action undertaken (eg, you don't trust it yet).
• Command line configurable verification modes of the existing distfiles on disk- either do quick size checks (default), or md5 all distfiles to identify if a distfile on disk has become corrupt, and needs an update- a more likely occurence would be that the ebuild's digest for that file has changed, so the distfile on disk is no longer valid

Host of other goodies. Code is pretty clean, has --usage, and --help output too.
Enjoy.