Unknown reporter writes

Hello!

I use Recoll 1.19.14 with XapianDB 1.2.17

I start recoll-webui this command: "nohup python ./webui-standalone.py >/dev/null 2 >&1&" I get this error message after !!202!! searching (I test many times and I always get error message after 202. webui search):

Sorry, the requested URL 'http://X.X.X.X/results?query=asdf&dir=%3Call%3E&after=&before=&sort=relevancyrating&ascending=0&page=1' caused an error:

Internal Server Error

Exception:

EnvironmentError("Can't open index",)

Traceback:

Traceback (most recent call last):
File "/opt/recoll/recoll-webui-master/bottle.py", line 744, in _handle
return route.call(**args)
File "/opt/recoll/recoll-webui-master/bottle.py", line 1479, in wrapper
rv = callback(*a, **ka)
File "/opt/recoll/recoll-webui-master/bottle.py", line 2850, in wrapper
result = func(*args, **kwargs)
File "/opt/recoll/recoll-webui-master/webui.py", line 241, in results
res, nres, timer = recoll_search(query)
File "/opt/recoll/recoll-webui-master/webui.py", line 180, in recoll_search
query = recoll_initsearch(q)
File "/opt/recoll/recoll-webui-master/webui.py", line 157, in recoll_initsearch
db = recoll.connect(config['confdir'])
EnvironmentError: Can't open index

Thomas01 writes

I got this message on the console:

Couldn’t open /recoll-index/position.baseA: Too many open files Couldn’t open /recoll-index/position.baseB: Too many open files

Traceback (most recent call last): File "/opt/recoll/recoll-webui-master/bottle.py", line 744, in _handle return route.call(args) File "/opt/recoll/recoll-webui-master/bottle.py", line 1479, in wrapper rv = callback(*a, ka) File "/opt/recoll/recoll-webui-master/bottle.py", line 2850, in wrapper result = func(*args, **kwargs) File "/opt/recoll/recoll-webui-master/webui.py", line 241, in results res, nres, timer = recoll_search(query) File "/opt/recoll/recoll-webui-master/webui.py", line 180, in recoll_search query = recoll_initsearch(q) File "/opt/recoll/recoll-webui-master/webui.py", line 157, in recoll_initsearch db = recoll.connect(config[confdir]) EnvironmentError: Can’t open index

medoc writes

Hi,

Thanks from reporting this, this may be a bug in the webui or the Python module (the latter is more probable). I’ll take a better look tomorrow (long day).

Thomas01 writes

I got this message from XapianDB admins:

"The issue is what it says - there are too many open files. We need some free file descriptors to open the index, but there are a finite number available - e.g. on modern Linux the limit is 65536. So it sounds like recoll isn’t closing files in some case, as it’s unlikely it really needs so many files open at once.

This is a recoll issue, not a Xapian one. There’s nothing we can really do if there aren’t file handles available, short of randomly closing some which wouldn’t be helpful overall."

medoc writes

Hi,

Yes, as I wrote above, this is most likely a Recoll issue, probably the Recoll Python module is not destroying its objects. It was not necessary to bother the Xapian admins at this point … Taking a better look this morning.

Thomas01 writes

I think, we must inject code to recoll-webui: db.close() or query.close()

medoc writes

No, on the Python side, object destruction is automatic. I found one bug in the python module which caused the db object not to be deallocated. This solves the descriptors issue. I’m searching for other leaks while I am at it, and I’ll then release a fix.

What platform are you running this on ? (I’ll arrange to build a package for you).

jf

Thomas01 writes

I use ubuntu 14.04 x64, but actually I try with kali linux 1.07 with sereveral version of recoll: 1.19.14-13-12 and XapianDB 1.2.18-17 and so on. Thank you again for everything you’ve done. I am really waiting your email and I will test immediately.

medoc writes

I have fixed a couple of bugs in the Python module: db objects were never de-allocated, db.close() and query.close() sometimes caused crashes.

There is still a memory leak, the webui server seems to be losing around 500 bytes per request. This only becomes significant for high request rates, and if you needed some time to go up to 200, maybe you can find a time slot to restart the server from time to time ? Correcting this is on my todo. If this is an urgent issue for you, tell me.

The fix will be in recoll 1.19.14p1 which exists only on the experimental PPA for now. Here follow the links to the source tar file and the built pyrecoll .deb:

I’ll move this to the web site and the normal PPA if you confirm that this fixes the issue for you.

Thomas01 writes

It is working after 35000 recoll-webui search. Your estimate is correct: loose 14MB memory after ~35000 query search

You’re a lifesaver. Literally. And thanks for believing in someone you don’t even know.

medoc writes

Well it’s not hard to believe a bug report with an obvious error log :) Thansk for helping me improve Recoll.

Don’t hesitate to contact me rapidly if you have other issues, don’t lose too much time testing on your side before initial contact. If more testing is needed, I’ll ask.

Cheers,

Jf

medoc writes

Solved by 502d35c6148c