Issue 42: recoll frequently stalls indexing for a considerable time and then crashes.

Running recoll for the first time to index nearly 1TB of data, it stalls whenever it hits certain file types (mainly graphic and icon files). It pauses for a long time but finally manages to some what to continue. However, not only this makes the indexing process very lengthy but also eventually it crashes. It never succeeded in finishing indexing all my data.

Recoll has been compiled with these flags (minus means it has not been compiled with that). I’m on Gentoo: app-misc/recoll-1.14.4 USE="djvu flac inotify mp3 msdoc msppt msxls ogg pdf qt4 rtf session spell xattr -camelcase -chm -dvi -exif -fam -gs -ics -lyx -tex -wordperfect -zip"

this the log in the terminal within which recoll has run and finally it crashed:

:

/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:544: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop589" / > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:552: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop590" / >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:560: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop591" / > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:597: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop596" / >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:605: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop597" / > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:613: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop598" / >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:650: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop603" / > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:658: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop604" / >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:666: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop605" / > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:703: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop610" / >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:711: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop611" / > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:719: namespace error : Namespace prefix a on midPointStop is not defined id="midPointStop612" / >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:863: namespace error : Namespace prefix x for xmptk on xmpmeta is not defined id="xmpmeta12" > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:863: namespace error : Namespace prefix x on xmpmeta is not defined id="xmpmeta12" >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:873: namespace error : Namespace prefix pdf on Producer is not defined id="Producer18" > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:902: namespace error : Namespace prefix xap on CreateDate is not defined id="CreateDate28" >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:909: namespace error : Namespace prefix xap on ModifyDate is not defined id="ModifyDate31" > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:916: namespace error : Namespace prefix xap on CreatorTool is not defined id="CreatorTool34" >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:923: namespace error : Namespace prefix xap on MetadataDate is not defined id="MetadataDate37" > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:930: namespace error : Namespace prefix xap on Thumbnails is not defined id="Thumbnails40" >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:937: namespace error : Namespace prefix xapGImg on format is not defined id="format46" > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:944: namespace error : Namespace prefix xapGImg on width is not defined id="width49" >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:951: namespace error : Namespace prefix xapGImg on height is not defined id="height52" > ^{/home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:958: namespace error : Namespace prefix xapGImg on image is not defined id="image55" >} /home/sisu/.icons/dlg-etiquette/scalable/apps/display-capplet.svg:1200: namespace error : Namespace prefix xapMM on DocumentID is not defined id="DocumentID64" > ^ * glibc detected * recoll: double free or corruption (fasttop): 0x00007fda800f9bb0 * ======= Backtrace: ========= /lib/libc.so.6(+0x72b56)[0x7fda91582b56] /lib/libc.so.6(cfree+0x6c)[0x7fda9158790c] ======= Memory map: ======== 00400000-005bd000 r-xp 00000000 08:08 423830 /usr/bin/recoll 007bc000-007bd000 r—p 001bc000 08:08 423830 /usr/bin/recoll 007bd000-007cd000 rw-p 001bd000 08:08 423830 /usr/bin/recoll 007cd000-007cf000 rw-p 00000000 00:00 0 02065000-086d5000 rw-p 00000000 00:00 0 [heap] 3b30c00000-3b30c46000 r-xp 00000000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b30c46000-3b30e45000 ---p 00046000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b30e45000-3b30e48000 r—p 00045000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b30e48000-3b30e49000 rw-p 00048000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b32c00000-3b32c15000 r-xp 00000000 08:08 1048955 /lib64/libz.so.1.2.3 3b32c15000-3b32e14000 ---p 00015000 08:08 1048955 /lib64/libz.so.1.2.3 3b32e14000-3b32e15000 r—p 00014000 08:08 1048955 /lib64/libz.so.1.2.3 3b32e15000-3b32e16000 rw-p 00015000 08:08 1048955 /lib64/libz.so.1.2.3 3b38e00000-3b38e27000 r-xp 00000000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b38e27000-3b39027000 ---p 00027000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b39027000-3b39029000 r—p 00027000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b39029000-3b3902a000 rw-p 00029000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b39200000-3b3923e000 r-xp 00000000 08:08 308566 /usr/lib64/libpng12.so.0 3b3923e000-3b3943d000 ---p 0003e000 08:08 308566 /usr/lib64/libpng12.so.0 3b3943d000-3b3943e000 r—p 0003d000 08:08 308566 /usr/lib64/libpng12.so.0 3b3943e000-3b3943f000 rw-p 0003e000 08:08 308566 /usr/lib64/libpng12.so.0 3b39e00000-3b39e3b000 r-xp 00000000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b39e3b000-3b3a03a000 ---p 0003b000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b3a03a000-3b3a03c000 r—p 0003a000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b3a03c000-3b3a03d000 rw-p 0003c000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b3aa00000-3b3aa1a000 r-xp 00000000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3aa1a000-3b3ac19000 ---p 0001a000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3ac19000-3b3ac1a000 r—p 00019000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3ac1a000-3b3ac1b000 rw-p 0001a000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3ae00000-3b3aedd000 r-xp 00000000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3aedd000-3b3b0dd000 ---p 000dd000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3b0dd000-3b3b0e0000 r—p 000dd000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3b0e0000-3b3b0e3000 rw-p 000e0000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3b0e3000-3b3b0e4000 rw-p 00000000 00:00 0 3b3b200000-3b3b209000 r-xp 00000000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3b209000-3b3b408000 ---p 00009000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3b408000-3b3b409000 r—p 00008000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3b409000-3b3b40a000 rw-p 00009000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3ba00000-3b3ba30000 r-xp 00000000 08:08 1858162 /usr/lib64/libglitz.so.1.0.0 3b3ba30000-3b3bc30000 ---p 00030000 08:08 1858162 /usr/lib64/libglitz.so.1.0.0 3b3bc30000-3b3bc31000 r—p 00030000 08:08 1858162 /usr/lib64/libglitz.so.1.0.0 3b3bc31000-3b3bc32000 rw-p 00031000 08:08 1858162 /usr/lib64/libglitz.so.1.0.0 3b3d800000-3b3d804000 r-xp 00000000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b3d804000-3b3da03000 ---p 00004000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b3da03000-3b3da04000 r—p 00003000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b3da04000-3b3da05000 rw-p 00004000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b3e400000-3b3e406000 r-xp 00000000 08:08 1858168 /usr/lib64/libglitz-glx.so.1.0.0 3b3e406000-3b3e606000 ---p 00006000 08:08 1858168 /usr/lib64/libglitz-glx.so.1.0.0 3b3e606000-3b3e607000 r—p 00006000 08:08 1858168 /usr/lib64/libglitz-glx.so.1.0.0 3b3e607000-3b3e608000 rw-p 00007000 08:08 1858168 /usr/lib64/libglitz-glx.so.1.0.0 3b40c00000-3b40c03000 r-xp 00000000 08:08 1858219 /usr/lib64/libgpg-error.so.0.5.0 3b40c03000-3b40e02000 ---p 00003000 08:08 1858219 /usr/lib64/libgpg-error.so.0.5.0 3b40e02000-3b40e03000 r—p 00002000 08:08 1858219 /usr/lib64/libgpg-error.so.0.5.0 3b40e03000-3b40e04000 rw-p 00003000 08:08 1858219 /usr/lib64/libgpg-error.so.0.5.0 3b41c00000-3b41d4e000 r-xp 00000000 08:08 1858196 /usr/lib64/libxml2.so.2.7.7 3b41d4e000-3b41f4d000 ---p 0014e000 08:08 1858196 /usr/lib64/libxml2.so.2.7.7 3b41f4d000-3b41f55000 r—p 0014d000 08:08 1858196 /usr/lib64/libxml2.so.2.7.7 3b41f55000-3b41f57000 rw-p 00155000 08:08 1858196 /usr/lib64/libxml2.so.2.7.7 3b41f57000-3b41f58000 rw-p 00000000 00:00 0 3b42800000-3b42860000 r-xp 00000000 08:08 1858192 /usr/lib64/libORBit-2.so.0.1.0 3b42860000-3b42a60000 ---p 00060000 08:08 1858192 /usr/lib64/libORBit-2.so.0.1.0 3b42a60000-3b42a70000 r—p 00060000 08:08 1858192 /usr/lib64/libORBit-2.so.0.1.0 3b42a70000-3b42a72000 rw-p 00070000 08:08 1858192 /usr/lib64/libORBit-2.so.0.1.0 3b42a72000-3b42a73000 rw-p 00000000 00:00 0 3b43000000-3b4303a000 r-xp 00000000 08:08 1858195 /usr/

Hi and thanks for reporting the problems.

About the stalls: these are quite probably the pauses which occurs when Xapian data is flushed from memory to disk. To confirm, set the debug level to 4. You should then see a message like the following at each pause:

4:Db::add: text size >= xxx Mb, flushing

There is a parameter to adjust how much data is kept in memory before a flush occurs: idxflushmb (to be set in ~~/.recoll/recoll.conf). By default, this is set to 10 MB (idxflushmb = 10) which is quite conservative (the actual amount of ram used is a few times the value). You could try to increase it (ie: 100 if you have a few GBytes of ram) and see if the pauses become less frequent. This should help performance too, but of course, the program will use more memory.

About the crash. Both the log and the stack dump are very unhelpful. You need to set the log level to at least 4 (loglevel = 4 in recoll.conf), and maybe send the messages to some file (logfilename=/tmp/somefilename). Like this, we will be able to see if the crash always happens for the same file. The error messages I see in the log are, I think, completely irrelevant: errors on image files because of the missing exiftool (normal), and some error messages from the svg filter, also uninteresting.

It would also be quite interesting to try to obtain a core dump and a meaningful stack trace (https://bitbucket.org/medoc/recoll/wiki/GettingAStackTrace).

Hello Recoll Developer(s), Thank you very much for your swift reply. Indeed, first I added exif and ghostscript support and then I took your advice on changing the configuration and somehow it worked. Finally it could index all my huge data and finished successfully. The program now shows resilient behavior and it never crashed again. However, I have noticed whenever I start recoll from a terminal (where it can write the log of its actions) the stall is more noticeable. Only through its GUI it is rock and solid and does the desired tasks without any lag. I am sure that it demonstrates that whenever it reaches indexing a file which does not know well, the lag appears. In fact, I considered adding support for exif, etc. to omit these unknown questions of recoll. Anyway, I think this bug could be closed or at least its severity could be lowered. Still, I would be grateful if the "lag problem" walks away. I should really thank you for your nice software. I am desperate for a stable desktop search tool and I use it more than often. Although beagle and tracker are very nice tools, they failed to index and monitor my frequently changing over 1TB data properly. Your tool seems to get the job well done. As a user who frequently uses desktop search tools under sever circumstances, I would report any misbehavior, optionnal feature, enhancement or problem. Thank you very much for your efforts.

I am really inclined to believe that the stall you are seeing is not related to the file types but to the index flushes to disk. This appears differently in the GUI and terminal messages, because the information printed is different: a sampling of indexed files on the GUI status line, errors in the log file (at verbosity 2). I can’t see why recoll would pause when encountering an unknown file. The way to confirm this is to set the verbosity to 4 and see if the lag comes with the data flush debug messages. I’m quite interested in fixing the issue if it is real, but I need a little more convincing :)

I’m more worried about the indexer crash, and would be most interested by more information if it happens again. Thanks a lot for reporting this, and don’t hesitate to contact me !

Hello again, First I am willing to thank you for your new version of Recoll. It’s nice with new features. However, even thought I has recoll 1.14 installed the new version 1.15 couldn’t make use of the old database; no matter how much I fiddled with it. I had to rebuild the whole database again, and still it’s rebuilding. This time first I changed verbosity to level 5 then started recoll from a terminal. During indexing a file which is descried bellow, it stalled. I waited so long but in vain so I kept the log from terminal which is as follows:

:4:../internfile/internfile.cpp:736:FileInterner::internfile. ipath [] :4:../utils/execmd.cpp:188:ExecCmd::startExec: (0|1) /usr/share/recoll/filters/rcltext {/home/sisu/mydownloads/LinDL/Crossover Linux Pro 8 32&64[deb-rpm-sh]/install-crossover-pro-8.0.0.sh} :5:../utils/netcon.cpp:268:Netcon::selectloop: fd 38 has 0x0 mask, erasing :5:../utils/execmd.cpp:368:ExecCmd::doexec: selectloop returned 0 :4:../utils/execmd.cpp:468:ExecCmd::wait: got status 0x0 :4:../internfile/internfile.cpp:652:FileInterner::addHandler: next_doc is text/html :4:../internfile/mh_html.cpp:78:MHHtml::next_doc.: default supposed input charset: [ANSI_X3.4-1968] :4:../internfile/mh_html.cpp:97:Html::mkDoc: pass 0 :4:../utils/transcode.cpp:129:transcode: [ANSI_X3.4-1968]- >[UTF-8] 15004147 errors :4:../internfile/mh_html.cpp:113:textHtmlToDoc: init transcode had 15004147 errors for [unknown]

I stopped it from indexing and started it again. Again it stalled at that point. This time neither I could stop it nor kill it. Simply even the kill command didn’t work and I had to reboot. After reboot when I started recoll again, it refused to index the data. Clicking on "update index" nothing happened. Now, I’m going to erase .recoll directory and rebuild it again! Do you have any idea about what is going wrong?

Update: It’s almost 4 hours that it is stuck with indexing some svg files. Here is the log: /gartoon/scalable/apps/gnome-setting.svg :4:../internfile/internfile.cpp:207:FileInterner:: [/home/sisu/.icons/gartoon/scalable/apps/gnome-setting.svg] mime [(null)] preview 0 :4:../internfile/internfile.cpp:281:FileInterner:: init ok image/svg+xml [/home/sisu/.icons/gartoon/scalable/apps/gnome-setting.svg] :4:../internfile/internfile.cpp:736:FileInterner::internfile. ipath [] :4:../utils/execmd.cpp:188:ExecCmd::startExec: (0|1) /usr/share/recoll/filters/rclsvg {/home/sisu/.icons/gartoon/scalable/apps/gnome-setting.svg} :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1 :4:../utils/execmd.cpp:360:ExecCmd::doexec: selectloop returned 1

The last line just appears for several times then it starts over again and again. It does advance but almost after spending several minutes on indexing every svg file.

Another update on the issue: I just excluded the whole of .icons folder inside my home directory. After I saved the preferences, recoll crashed outputting these lines:

Netcon::selectloop: fd 57 has 0x0 mask, erasing :5:../utils/execmd.cpp:368:ExecCmd::doexec: selectloop returned 0 :4:../utils/execmd.cpp:468:ExecCmd::wait: got status 0x0 * glibc detected * recoll: double free or corruption (fasttop): 0x0000000007d76ce0 * ======= Backtrace: ========= /lib/libc.so.6(+0x72b56)[0x7f9ba2177b56] /lib/libc.so.6(cfree+0x6c)[0x7f9ba217c90c] ======= Memory map: ======== 00400000-005c7000 r-xp 00000000 08:08 1319051 /usr/bin/recoll 007c7000-007c8000 r—p 001c7000 08:08 1319051 /usr/bin/recoll 007c8000-007d8000 rw-p 001c8000 08:08 1319051 /usr/bin/recoll 007d8000-007da000 rw-p 00000000 00:00 0 01219000-16a39000 rw-p 00000000 00:00 0 [heap] 3b30c00000-3b30c46000 r-xp 00000000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b30c46000-3b30e45000 ---p 00046000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b30e45000-3b30e48000 r—p 00045000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b30e48000-3b30e49000 rw-p 00048000 08:08 1859015 /usr/lib64/libeggdbus-1.so.0.0.0 3b32c00000-3b32c15000 r-xp 00000000 08:08 1048955 /lib64/libz.so.1.2.3 3b32c15000-3b32e14000 ---p 00015000 08:08 1048955 /lib64/libz.so.1.2.3 3b32e14000-3b32e15000 r—p 00014000 08:08 1048955 /lib64/libz.so.1.2.3 3b32e15000-3b32e16000 rw-p 00015000 08:08 1048955 /lib64/libz.so.1.2.3 3b38e00000-3b38e27000 r-xp 00000000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b38e27000-3b39027000 ---p 00027000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b39027000-3b39029000 r—p 00027000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b39029000-3b3902a000 rw-p 00029000 08:08 1858157 /usr/lib64/libexpat.so.1.5.2 3b39e00000-3b39e3b000 r-xp 00000000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b39e3b000-3b3a03a000 ---p 0003b000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b3a03a000-3b3a03c000 r—p 0003a000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b3a03c000-3b3a03d000 rw-p 0003c000 08:08 1858158 /usr/lib64/libfontconfig.so.1.4.4 3b3aa00000-3b3aa1a000 r-xp 00000000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3aa1a000-3b3ac19000 ---p 0001a000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3ac19000-3b3ac1a000 r—p 00019000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3ac1a000-3b3ac1b000 rw-p 0001a000 08:08 1858159 /usr/lib64/libdirect-1.4.so.0.3.0 3b3ae00000-3b3aedd000 r-xp 00000000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3aedd000-3b3b0dd000 ---p 000dd000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3b0dd000-3b3b0e0000 r—p 000dd000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3b0e0000-3b3b0e3000 rw-p 000e0000 08:08 1858161 /usr/lib64/libdirectfb-1.4.so.0.3.0 3b3b0e3000-3b3b0e4000 rw-p 00000000 00:00 0 3b3b200000-3b3b209000 r-xp 00000000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3b209000-3b3b408000 ---p 00009000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3b408000-3b3b409000 r—p 00008000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3b409000-3b3b40a000 rw-p 00009000 08:08 1858160 /usr/lib64/libfusion-1.4.so.0.3.0 3b3d800000-3b3d804000 r-xp 00000000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b3d804000-3b3da03000 ---p 00004000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b3da03000-3b3da04000 r—p 00003000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b3da04000-3b3da05000 rw-p 00004000 08:08 1858171 /usr/lib64/libxcb-render-util.so.0.0.0 3b43000000-3b4303a000 r-xp 00000000 08:08 1858195 /usr/lib64/libgconf-2.so.4.1.5 3b4303a000-3b43239000 ---p 0003a000 08:08 1858195 /usr/lib64/libgconf-2.so.4.1.5 3b43239000-3b4323c000 r—p 00039000 08:08 1858195 /usr/lib64/libgconf-2.so.4.1.5 3b4323c000-3b4323e000 rw-p 0003c000 08:08 1858195 /usr/lib64/libgconf-2.so.4.1.5 3b46800000-3b4686a000 r-xp 00000000 08:08 1858527 /usr/lib64/libgnomevfs-2.so.0.2400.3 3b4686a000-3b46a69000 ---p 0006a000 08:08 1858527 /usr/lib64/libgnomevfs-2.so.0.2400.3 3b46a69000-3b46a6d000 r—p 00069000 08:08 1858527 /usr/lib64/libgnomevfs-2.so.0.2400.3 3b46a6d000-3b46a6e000 rw-p 0006d000 08:08 1858527 /usr/lib64/libgnomevfs-2.so.0.2400.3 3b46a6e000-3b46a6f000 rw-p 00000000 00:00 0 3b48800000-3b48827000 r-xp 00000000 08:08 1858537 /usr/lib64/libart_lgpl_2.so.2.3.21 3b48827000-3b48a27000 ---p 00027000 08:08 1858537 /usr/lib64/libart_lgpl_2.so.2.3.21 3b48a27000-3b48a28000 r—p 00027000 08:08 1858537 /usr/lib64/libart_lgpl_2.so.2.3.21 3b48a28000-3b48a29000 rw-p 00028000 08:08 1858537 /usr/lib64/libart_lgpl_2.so.2.3.21 7f9b78000000-7f9b7bfcf000 rw-p 00000000 00:00 0 7f9b7bfcf000-7f9b7c000000 ---p 00000000 00:00 0 7f9b8b769000-7f9b8b771000 rw-p 00000000 00:00 0 7f9b8e771000-7f9b8e781000 rw-p 00000000 00:00 0 7f9b8ef81000-7f9b8ef89000 rw-p 00000000 00:00 0 7f9b8f113000-7f9b8f123000 rw-p 00000000 00:00 0 7f9b8f306000-7f9b8f44e000 rw-p 00000000 00:00 0 7f9b8f451000-7f9b8f709000 rw-p 00000000 00:00 0 7f9b8f709000-7f9b8f786000 r-xp 00000000 08:08 453142 /usr/lib64/libtiff.so.3.9.4 7f9b8f786000-7f9b8f985000 ---p 0007d000 08:08 453142 /usr/lib64/libtiff.so.3.9.4 7f9b8f985000-7f9b8f988000 r—p 0007c000 08:08 453142 /usr/lib64/libtiff.so.3.9.4 7f9b8f988000-7f9b8f989000 rw-p 0007f000 08:08 453142 /usr/lib64/libtiff.so.3.9.4 7f9b8f989000-7f9b8f991000 r-xp 00000000 08:08 419439 /usr/lib64/qt4/plugins/imageformats/libqtiff.so 7f9b8f991000-7f9b8fb90000 ---p 00008000 08:08 419439 /usr/lib64/qt4/plugins/imageformats/libqtiff.so 7f9b8fb90000-7f9b8fb91000 r—p 00007000 08:08 419439 /usr/lib64/qt4/plugins/imageformats/libqtiff.so 7f9b8fb91000-7f9b8fb92000 rw-p 00008000 08:08 419439 /usr/lib64/qt4/plugins/imageformats/libqtiff.so

About 1.15 not reading 1.14 indexes: I really think it usually does, and I would have been interested by some details (ie: a log) about the "couldn’t make use")

About the .sh file. New in 1.15, and following user requests, some script program files (ie: .sh, .pl, …) are now indexed as text. I’d venture that the problem file is some kind of shar-like archive with a big bunch of data following the initial shell text. I’d be interested by what the data looks like by the way, because it was apparently text-like enough to interest recoll :( What is the size of this file ? This issue is difficult to handle. One possible approach would be to lower the "textfilemaxmbs" limit which is by default quite high (20 - > 20 MegaBytes).

For quickly killing an indexing recoll, you can use kill -9. This needs to be followed by a full indexing though (or remove the xapiandb directory), because the index files could be in bad shape. If this does not work (reboot actually needed), this has to be a system issue, recollindex is a normal user command, and there is no way it can ignore a kill -9.

In your case, maybe the reboot leaved a lock file around, or some other issue that imposed removing .recoll (or more likely .recoll/xapiandb would have been enough).

Add the offending file to the skippedPaths variable to avoid more issues.

About the svg files: recoll uses xsltproc to extract text from svg files. I hadn’t noticed it was that slow, but if it’s too slow and you’re not interested by a possible text content in the files, just add the directory to skippedPaths.

The selectloop debug message signals that recoll is waiting for some command (rclsvg here). I have no way to know if rclsvg is looping here or if it’s just very slow. You can directly try "/usr/share/recoll/filters/rclsvg somesvg.svg" to see what happens. If it actually loops I’d be more than interested by a sample file. Were you able to verify if the files were indexed or if the recoll timed out on the command ?

About the final crash: no idea what happened from what I see here. If you can find a way to reliably reproduce the issue, of course, I’d be much interested to fix it !

Please keep me updated. I’m not currently at home, so you may not get prompt answers, but I’d like to work on these as soon as I’m back next week if you can provide a bit more details and data. You can also directly email me at jfd@recoll.org, maybe it will be more convenient.

Hopefuly fixed by commits 2175-1280

Added video types to the media category. closes issue #42

→ <<cset 684ec8ee05ee > >