Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: 

JIRA or Github Issue: 

Released: <Doris Version>

Google Doc: <If the design in question is unclear or needs to be discussed and reviewed, a Google Doc can be used first to facilitate comments from others.>

Motivation

For text data, such as log, commodity description, OLAP users need to search in the text. For example search error logs that contains 'ERROR', 'Exception' keywords.

Currently, users use LIKE SQL function for text pattern matching in Doris and most OLAP databases. But LIKE is not slow since all rows need to be checked against the search pattern.

Related Research

In information retrieve area, fulltext search is a mature solution to find the content that match a given query. It's widely used by web search engine.

Elasticsearch support distributed fulltext search capability based on the open source search library lucene.

Some database, eg. MySQL, PostgreSQL, also add inverted index to support fulltext search.

In fact, the BITMAP index in Doris is a simple inverted index. But it lack text tokenization, efficient dictionary, search query syntax to support mature fulltext search.

Detailed Design

the detailed design of the function.

Scheduling

specific implementation steps and approximate scheduling.