Patrick's playground

KDE 4.13 was released with a new indexer, named "Baloo". It mostly replaces the 'old' Akonadi indexer, which at first glance appears to be a good idea. It seems to work, so that's quite swell. There's only a problem. Or rather, some little problems, and upstream is one of them as they don't want to acknowledge that these issues exist. So let me try to explain ...

• There are times when I just need the indexer to not run. For example when I'm watching a movie (IO activity -> stutter), doing a presentation (random lag?!) etc. And there are times (e.g. at night) when the indexer can run as much as it wants.
• There are times when the indexer interferes with normal operation - e.g. when using firefox, the added IO activity causes the FF UI to lag severely, as if the machine was swapping. Partially also because the IO activity evacuates the filesystem cache, which is quite funny. And fsync plus lots of reads means the latency goes up to multiple seconds or even multiple tens of seconds for a single IO activity
• The indexer claims to not interfere with normal operation. It limits itself to 10% CPU usage - which is the wrong metric, since I have lots of CPU and very little IO, relatively speaking. Thus it takes 100% of available IO bandwidth. Akonadi used up to 4 CPUs for longer amounts of time, but as it didn't hurt IO much I could just ignore it.
• The indexer takes a LONG time. On boot it needs about 20 minutes walltime just to figure out if anything has changed. During that time service quality is severely degraded.
• The indexer takes a long time. The initial scan of my home directory takes about, hmm, 36-48h I think, during which time service quality is severely degraded
• The indexer isn't polite, it auto-respawns if you just kill the baloo_file_indexer process. You have to kill its parent too, otherwise it'll just respawn and bother you some more
• [Fixed in next release] Removing a directory from the index causes an index cleaner to run, which is even more severe than the indexer itself

So, to summarize: As much as I like the indexer, it prevents me from working normally, so I have to insist that it has a simple "off" button. A lesson that akonadi learned, that gnome's tracker learned, is that you need to nice yourself down. It would be very much appreciated if baloo were to nice and ionice itself down to idle, which usually avoids the severe lag that foreground tasks may experience.

An extra bonus would be this: The indexer should do a microbenchmark on startup (or let the user provide a guesstimate) to figure out IO capacity in IO/s, and then limit itself to a configurable amount of that. If it takes 1/10th of my IO bandwidth (about 10-15 IO/s with a single SATA disk) it wouldn't even bother me more than, say, Firefox running in the background.

Another interesting glitch is that most indexers use inotify listeners to see if anything in a directory changes. This has the funny effect that it only works on small data sets - on my desktop I get random popups that an application wants to change system limits. Well, /proc/sys/fs/inotify/max_user_watches is already set to "262144" by default, and that's still not enough? This also takes memory, and it can't scale up. I "only" have a few million files, that's not even a lot.

So, to summarize:
Simple fixes:

• Nice and ionice the indexer on startup
• Provide users with a simple on/off mechanism

Advanced fixes:

• Throttle on IO instead of CPU
• Delay indexer startup for a little while on boot. Maybe 120sec grace period
• Figure out system limits and fail gracefully instead of annoying users with popups

Well, dear upstream, don't accuse me of not being constructive ...

<DrEeevil> people complain about user-hostile behaviour, and you tell them to ... be nicer and not complain so loud?
<unormal> DrEeevil: To be honest. The only things I see from you in here since hours and days is hostile behaviour. I really would like to ask you to stop this and be constructive or otherwise leave
<DrEeevil> unormal: well, if I didn't have to remove binaries and kill processes I'd be a lot happier
<DrEeevil> since upstream hasn't shown any understanding I'll rather escalate until the bugs are resolved
<DrEeevil> constructive: give me an off button so I can stop the indexer when it hurts me, give me a rate limit so I can run it while using the computer
<DrEeevil> (using 99% of available IO bandwidth for up to 72h is just not acceptable in normal use)
<DrEeevil> I don't want to remove the indexer, but I want to control how much resource usage it has
<DrEeevil> (bonus: ionice + nice it down to lowest/idle, then it doesn't bother that much)
<DrEeevil> it's not THAT hard to figure that out ...
<unormal> DrEeevil: Ok and now explicitely: Please do us a favor and leave the channel.
<DrEeevil> unormal: once I can have baloo installed and working as described above you'll never hear from me again
<DrEeevil> just every time I get a local DoS you get a complaint, so that you don't lose the motivation to fix the bugs
<unormal> That's not how you motivate people, please leave.
<DrEeevil> that's not how you write software, please fix
<DrEeevil> I'll be the stone in your shoe until you stop being the one in mine
<DrEeevil> heck, I'll even test patches once they are provided!
<krop> and ultimately, you'll roll on the floor crying until something happens ? how can gentoo accept immature people in their staff ?
--> seaLne (~seaLne@kde/kenny) has joined #kde-baloo
*** Mode #kde-baloo +o seaLne by ChanServ
<DrEeevil> krop: how can kde have releases with such serious regressions?
<DrEeevil> sorry, I don't deal with C++, in this case I'm just a QA tool
<krop> no, you just behave like a stubborn child
<DrEeevil> because I actually would like to USE kde
<DrEeevil> not sure how you see that, but it's kinda nice usually, except when someone staples in a DoS and then tells me that's all fine and dandy
<DrEeevil> maybe I should use git HEAD again to catch regressions earlier
*** Mode #kde-baloo +b DrEeevil!*@* by seaLne
*** Mode #kde-baloo +b not!*@* by seaLne
*** Mode #kde-baloo +b being!*@* by seaLne
*** Mode #kde-baloo +b constructive!*@* by seaLne
<DrEeevil> heh
<-* seaLne has kicked DrEeevil from #kde-baloo (DrEeevil)