Unknown reporter writes

When indexing, recoll sucks up all the disk I/O on my reasonably new desktop system, making it painfully slow. I’m talking 8 seconds to create a new terminal, for instance. It’s not swapping: it’s just doing full-throttle I/O: it’s greedy.

Running recoll with ionice -c2 -n7 nice -19 recoll solves the indexing problem nicely, though it can make the UI a bit slow. The appropriate solution is to put a call to nice and ionice at the point where the indexing gets forked off from the user interface. It should be a 2-line fix, and it will stop a lot of grumbling.

medoc writes

Thanks for reporting this. This is actually the first time someone complains about Recoll being io-greedy (which it certainly is, there is nothing to prevent it). This all depends on the balance of cpu/memory/io on each system.

The problem with using ionice on the recoll command is that it’s going to slow the searches as well. I think that the best approach would be to use the command-line recollindex to perform the indexing instead (running it with ionice). It does exactly the same thing, the only thing you’ll be losing is the status line updates.

recollindex already runs at low cpu priority. I’ll try to use ionice for the next release, but:

  • ionice seems to be operating at the process, not at the thread level, so that using the command line will remain preferable.

  • ionice is fairly recent and doesn’t even seem to be exported by libc yet, I don’t know how to use it from a program.

I may be missing something, if you can point me to more information about ionice, this will be welcomed…

medoc writes

Closing this: I’ll add an ionice call to recollindex as soon as there is a libc interface, but I’ve never seen anything close to the effects reported, there must be something special with the system.

r_mano writes

Hi, I’d like to chime in after 2 years --- I am coming back to recoll after a time, and I still have a problem with recollindex.

I start it with the ionice and nice option as above, and even I try to limit it in a cgroup to limit its memory and cpu usage, but still --- on my two machines, running Ubuntu gnome 14.04, when recollindex is running all the system is painfully slow (for example, now I am writing this and having the feedback on firefox after 10-15 seconds at time --- it seems that recollindex stop dead every other minute during some second and with it all the system).

One of the machines is a i5/8G and the other one an i5/16G ram.

Running last version form the PPA / 1.19.14p2-1ppa1trusty1

Thanks!

medoc writes

The periodicity that you are seeing probably corresponds to the Xapian "flushes" when index data is sent to disk after a period of in-memory updating.

The weird thing is that I never see this ! And, even if Recoll is not an extremely current application, it still probably has at least a few thousands regular users, and I should hear more complaints if this system-freezing was common.

recollindex ionices itself, so there should be no need of an external command.

I would suspect a hardware or close-to-hardware issue, but you are seeing this on 2 different systems so this seems unlikely.

Do you do any system tuning or are these vanilla Trusty installs ?

I have a full indexing going on while writing this, and I just could not say when the flushes happen, the system just seems normally responsive all the time.

r_mano writes

Hi --- first of all thank you very much.

This are quite vanilla system --- Ubuntu Trusty 14.04. I am now on my "home" system, reindexing recoll (I just updated this one from the Ubuntu standard to the PPA system). This one sports 16G of ram, but the "hiccups" are here too. I have recollindex running in a limited-memory and limited-cpu cgroup, but still sometime it stops all the rest of the system for two-three seconds.

It happen mostly when it is indexing very big files (I have some big mbox with thousands of messages stored on my thunderbird directory); and your guess about the flush is probably correct.

After it’ll finish I’ll check the incremental thing --- if it happens only on a full rebuild I do not mind.

This is the ps status now:

#!python

⌂87% [romano:~] % ps augxwm | grep recoll
romano     622  0.0  0.0  30096  5028 pts/13   -    19:05   0:00 python /usr/share/recoll/filters/rclaudio
romano    6230  0.0  0.0  45856  3396 pts/13   -    19:15   0:00 python /usr/share/recoll/filters/rclchm
romano    6231  0.0  0.0  45392  3400 pts/13   -    19:15   0:00 python /usr/share/recoll/filters/rclchm
romano    6232  0.0  0.0  45364  3356 pts/13   -    19:15   0:00 python /usr/share/recoll/filters/rclchm
romano   12405 32.5  0.8 641364 133764 pts/13  -    19:03  26:55 recollindex
romano   12418  0.0  0.0  30108  5468 pts/13   -    19:03   0:00 python /usr/share/recoll/filters/rclaudio
romano   12421  0.0  0.0  30080  4588 pts/13   -    19:03   0:00 python /usr/share/recoll/filters/rclaudio
romano   12424  0.0  0.0  35416  8468 pts/13   -    19:03   0:01 python /usr/share/recoll/filters/rclzip
romano   12431  0.0  0.1  46464 17296 pts/13   -    19:03   0:01 python /usr/share/recoll/filters/rclzip
romano   12432  0.8  0.1  76852 24904 pts/13   -    19:03   0:40 /usr/bin/perl -w /usr/share/recoll/filters/rclimg
romano   12506  0.0  0.0  42180  4716 pts/13   -    19:03   0:00 python /usr/share/recoll/filters/rclzip
romano   12530  0.0  0.0  28740  2380 pts/13   -    19:03   0:00 python /usr/share/recoll/filters/rclics
romano   12578  0.0  0.0  67408 10652 pts/13   -    19:03   0:00 python /usr/share/recoll/filters/rclzip
romano   12934  0.0  0.0  56876 10220 pts/13   -    19:03   0:01 python /usr/share/recoll/filters/rclzip
romano   13130  0.0  0.0  37884  8160 pts/13   -    19:03   0:00 python /usr/share/recoll/filters/rclzip
romano   13267  0.0  0.0  33332  5676 pts/13   -    20:21   0:00 python /usr/share/recoll/filters/rcldia
romano   13564  0.0  0.0  68144  5924 pts/13   -    19:03   0:01 python /usr/share/recoll/filters/rclzip
romano   14197  0.7  0.1  76848 25912 pts/13   -    19:03   0:39 /usr/bin/perl -w /usr/share/recoll/filters/rclimg
romano   14790  0.7  0.1  78132 25132 pts/13   -    19:03   0:39 /usr/bin/perl -w /usr/share/recoll/filters/rclimg
romano   14791  0.7  0.1  82428 29804 pts/13   -    19:03   0:38 /usr/bin/perl -w /usr/share/recoll/filters/rclimg
romano   17836  0.0  0.0  28872  5532 pts/13   -    20:24   0:00 python /usr/share/recoll/filters/rclinfo
romano   23212  0.0  0.0  15988   896 pts/16   -    20:26   0:00 grep recoll
romano   24088  0.0  0.0  30036  4800 pts/13   -    19:09   0:00 python /usr/share/recoll/filters/rclaudio
romano   25088  0.0  0.0  28996  2896 pts/13   -    19:10   0:00 python /usr/share/recoll/filters/rclics
romano   25101  0.0  0.0  28740  2896 pts/13   -    19:10   0:00 python /usr/share/recoll/filters/rclics
romano   30123  0.0  0.0  28872  3436 pts/13   -    19:14   0:00 python /usr/share/recoll/filters/rclinfo

If I can help dubugging, please tell me, I’ll try to do my best.

medoc writes

Big mboxes should not be particularly an issue, except that they get fully reindexed if anything changes (there is no reasonable way to determine which messages need reindexing). But only the current message is in memory, the mbox itself is streamed as a file.

But this made me think of another possibility: individual documents are copied to memory while being indexed, and the memory used can be a multiple of the document size. normally, there are safeguards to avoid extreme situations (for example a maximum of 20 MBS by default for text/plain files to avoid issues with big logs), but maybe you have some documents which trigger extreme memory usage. Anything you can think of which could be in this category ?

r_mano writes

Hmmm… well, I have some big file (binary) that are really raw data from measurements, some of them compressed. But not so many, really.

#!bash

⌂66% [romano:~] 1 % find . -size +20M | wc -l
230

Another possibility could be some interaction with gnome-shell? I do not know how does the "recent documents" things in the dash works, but maybe there could be some strange interaction?

Hmmm. no. Now what was completely locked was firefox, because some javascript was taking too longo to run. Maybe some lock on some directory where different programs are trying to write? Should I try to move the recoll db on a different partiton?

medoc writes

I don’t think that recoll locks any directory. The measurement files are an interesting possibility though. Esp. if they fail indexing, they would be retried at each run. Do they have some distinctive file name suffix which you could add to the skippedNames ? Or maybe you can skip their location to see if it changes anything ?

r_mano writes

Yep --- the are .rwa .rwb etc… I will try to blacklist them. Next week I’ll try and comment back.

medoc writes

no feedback