Motivation

Currently index updates and (re)build operations are not very efficient for multiple reasons. The goal of this IEP is to aggregate all possible improvements to our indexes and track their progress. Implementation of these fixes and features are critical to product success because a lot of our users struggle with slow data ingestion and slow updates when {{indexing}} module is used.

Proposed changes

The following changes are proposed:

• Iterate over data pages rather than partition BTree when building an index
• Fix index inline logic
• Sort index entries by (indexed value + link) rather than (indexed value + _KEY) to minimize number of data page lookups
• Fix PK index which is currently built over _KEY object rather than concrete columns
• Implement OFFLINE index create procedure using bottom-up approach
• Implement batched index update
• Skip secondary index updates when possible
• Avoid unnecessary lookup through data tree for every key
• Optionally disable WAL during index re-build

Tickets