...

Google Doc: <If the design in question is unclear or needs to be discussed and reviewed, a Google Doc can be used first to facilitate comments from others.>

Motivation

For text data, such as log, commodity description, OLAP users need to search in the text. For example search error logs that contains 'ERROR', 'Exception' keywords.

Currently, users use LIKE SQL function for text pattern matching in Doris and most OLAP databases. But LIKE is not slow since all rows need to be checked against the search patternDescribe the problems you are trying to solve.

Related Research

In information retrieve area, fulltext search is a mature solution to find the content that match a given query. It's widely used by web search engine.

Elasticsearch support distributed fulltext search capability based on the open source search library lucene.

Some database, eg. MySQL, PostgreSQL, also add inverted index to support fulltext search.

In fact, the BITMAP index in Doris is a simple inverted index. But it lack text tokenization, efficient dictionary, search query syntax to support mature fulltext searchsome research related to the function, such as the advantages and disadvantages of the design, related considerations, etc.

Detailed Design

the detailed design of the function.

...