Recoll overview

Recoll uses the Xapian information retrieval library as its storage and retrieval engine. Xapian is a very mature package using a sophisticated probabilistic ranking model.

The Xapian library manages an index database which describes where terms appear in your document files. It efficiently processes the complex queries which are produced by the Recoll query expansion mechanism, and is in charge of the all-important relevance computation task.

Recoll provides the mechanisms and interface to get data into and out of the index. This includes translating the many possible document formats into pure text, handling term variations (using Xapian stemmers), and spelling approximations (using the aspell speller), interpreting user queries and presenting results.

In a shorter way, Recoll does the dirty footwork, Xapian deals with the intelligent parts of the process.

The Xapian index can be big (roughly the size of the original document set), but it is not a document archive. Recoll can only display documents that still exist at the place from which they were indexed. (Actually, there is a way to reconstruct a document from the information in the index, but the result is not nice, as all formatting, punctuation and capitalization are lost).

Recoll stores all internal data in Unicode UTF-8 format, and it can index files of many types with different character sets, encodings, and languages into the same index. It can process documents embedded inside other documents (for example a pdf document stored inside a Zip archive sent as an email attachment...), down to an arbitrary depth.

Stemming is the process by which Recoll reduces words to their radicals so that searching does not depend, for example, on a word being singular or plural (floor, floors), or on a verb tense (flooring, floored). Because the mechanisms used for stemming depend on the specific grammatical rules for each language, there is a separate Xapian stemmer module for most common languages where stemming makes sense.

Recoll stores the unstemmed versions of terms in the main index and uses auxiliary databases for term expansion (one for each stemming language), which means that you can switch stemming languages between searches, or add a language without needing a full reindex.

Storing documents written in different languages in the same index is possible, and commonly done. In this situation, you can specify several stemming languages for the index.

Recoll currently makes no attempt at automatic language recognition, which means that the stemmer will sometimes be applied to terms from other languages with potentially strange results. In practise, even if this introduces possibilities of confusion, this approach has been proven quite useful, and it is much less cumbersome than separating your documents according to what language they are written in.

By default, Recoll strips most accents and diacritics from terms, and converts them to lower case before either storing them in the index or searching for them. As a consequence, it is impossible to search for a particular capitalization of a term (US / us), or to discriminate two terms based on diacritics (sake / saké, mate / maté).

Recoll versions 1.18 and newer can optionally store the raw terms, without accent stripping or case conversion. In this configuration, default searches will behave as before, but it is possible to perform searches sensitive to case and diacritics. This is described in more detail in the section about index case and diacritics sensitivity.

Recoll has many parameters which define exactly what to index, and how to classify and decode the source documents. These are kept in configuration files. A default configuration is copied into a standard location (usually something like /usr/share/recoll/examples) during installation. The default values set by the configuration files in this directory may be overridden by values set inside your personal configuration, found by default in the .recoll sub-directory of your home directory. The default configuration will index your home directory with default parameters and should be sufficient for giving Recoll a try, but you may want to adjust it later, which can be done either by editing the text files or by using configuration menus in the recoll GUI. Some other parameters affecting only the recoll GUI are stored in the standard location defined by Qt.

The indexing process is started automatically (after asking permission), the first time you execute the recoll GUI. Indexing can also be performed by executing the recollindex command. Recoll indexing is multithreaded by default when appropriate hardware resources are available, and can perform in parallel multiple tasks for text extraction, segmentation and index updates.

Searches are usually performed inside the recoll GUI, which has many options to help you find what you are looking for. However, there are other ways to perform Recoll searches: