KinoSearch::Docs::FileFormat 0.15

KinoSearch::Docs::FileFormat Perl module contains an overview of invindex file format.

It is not necessary to understand the guts of the Lucene-derived "invindex" file format in order to use KinoSearch, but it may be helpful if you are interested in tweaking for high performance, exotic usage, or debugging and development.
On a file system, all the files in an invindex exist in one, flat directory.

Conceptually, the files have a hierarchical relationship: an invindex is made up of "segments", each of which is an independent inverted index, and each segment is made up of several subsections.

[invindex]--|
|-"segments" file
|
|-[segments]------|
|--[seg _0]--|
| |--[postings]
| |--[stored fields]
| |--[deletions]
|
|--[seg _1]--|
| |--[postings]
| |--[stored fields]
| |--[deletions]
|
|--[ ... ]---|

The "segments" file keeps a list of the segments that make up an invindex. When a new segment is being written, KinoSearch may put files into the directory, but until the segments file is updated, a Searcher reading the index wont know about them.

Each segment is an independent inverted index. All the files which belong to a given segment share a common prefix which consists of an underscore followed by 1 or more decimal digits: _0, _67, _1058. A fully optimized index has only a single segment.

In theory there are many files which make up each segment. However, when you look inside an invindex not in the process of being updated, youll probably see only the segments file and files with either a .cfs or .del extension. The .cfs file, a "compound" file which is consolidated when a segment is finalized, "contains" all the other per-segment files.

Segments are written once, and with the exception of the deletions file, are never modified once written. They are deleted when their data is written to new segments during the process of optimization.