Issue 352: Excessive memory use during recoll startup

Environment: Running OpenSuSE 42.1, KDE desktop, using Recoll 1.22.4-3.1-x86_64 from their KDE:Extra repository. 6 Gb RAM, 4 Gb Swap. Been using recoll and very happy with it for years; problem started since upgrade to OpenSuSE 42.1.

Description: On starting, recoll memory usage as shown by KSysGuard rises to maximum available, peaking around 5 Gb within the first 5-10 seconds. The desktop is unresponsive untill the recoll GUI appears after about 10-15 minutes. In the mean time the usually empty swap file had grown some. Once the recoll GUI appears, recoll memory use drops a bit to around 4.5 Gb; recoll then works perfectly normally and other applications are once again responsive, albeit sluggish. CPU load remains normal throughout.

This is probably not an index, configuration or xapian issue.The attached screenshot shows a new install with default recoll.conf and no xapiandb directory, and no active search, about 15 minutes after launching recoll when the GUI first appeared.

At no point in the two weeks that I have been struggling with this has recoll ever crashed, it just hogs memory and freezes the machine. It seems that something that happens during recoll startup is misbehaving, but stops doing so once the GUI is loaded, although the memory is not being released. I will be grateful if you could look into this issue that is making recoll essentially unusable, and am happy to provide the output of any diagnostics you might require.

What I have tried to do already: Uninstall and re-install: no effect Recreate index with recollindex -z: no effect Reduce index size by half by blocking largest directories and re-indexing: no effect Tried various index configuration options: no effect Delete all indices: no effect Start recoll with: ionice -c2 -n7 nice -19 recoll: no effect Attempt to limit memory use with: ulimit -v 1024000; recoll results in: terminate called after throwing an instance of St9bad_alloc what(): std::bad_alloc The same for ulimit with 2 Gb and 3 Gb.

Follow-up:

Installed 1.23.1 from source on recoll.org. No change.

Installed the earliest qt4 version I could find: 1.16.2. It starts off the same: memory use climbs to about 5.5 Gb in the first 10 seconds, everything hangs for a few seconds more, but then suddenly memory drops to 268Mb, the GUI opens, and everything else works normally. When I close recoll, the memory use again shoots up and reaches 1Gb where it again hangs a few seconds before it actually closes. This is specific to recoll: nothing else that I use does anything similar. Very strange, but at least I can live with a 15 second wait instead of a 15 minute one. Busy rebuilding indices and will see what happens then.

I tried to reproduce the problem, here is what I did:

Started recoll, came up instantly, nothing special to report (it would be too easy).

Maybe try to delete ~/.config/Recoll.org/recoll.conf, this holds the GUI preferences, and I quite regularly see cases of corruption with miscellaneous consequences.

If this does not change anything, and if you installed 1.23.1 from source, maybe you could try to build it with qt5 to see if it changes anything. It should be something like the following (depending on qmake naming on openSuse):

Good morning. Thank you for your reply and your efforts. As always, the biggest problems have the simplest solutions. My /.config/Recoll.org/recoll.conf was 126 Mb in size, caused by a prefs\adv\clauseList that contained the same short sequence of numbers repeated over and over again. I do not know how it became this way, but when I deleted it the memory problem disappeared, like magic.

I was unaware of /.config/Recoll.org/recoll.conf, only knew of /.recoll/recoll.conf. Since it seems to be auto-generated with default values after deletion, it might be a good idea to add this trick to the wiki pages (unless it is already there and I just missed it).

I appreciate you taking the time to help me. Recoll is a great product. Keep up the good work. Thank you.

There was a bug some time ago which could cause corruption while loading the clause list from the GUI configuration file. In most cases, this resulted in a program crash, but it’s always difficult to predict exactly what may happen in this kind of situation.

The bug was fixed around April 2016, in 1.21.7, and is fixed in all 1.22 and 1.23 releases, so I am not too sure of what triggered your issue. We’ll leave it at that for the moment, but please report any reoccurrence (rather than just deleting the config file :) )

I’ll add a mention of this issue and its workaround to the list of know problems.