Manticore Search has implemented automatic chunking for vector search

The Manticore Search team has added built-in support for automatic document chunking when performing vector search. The new functionality is activated through the chunk_strategy parameter and allows processing long texts that could previously lose relevance when converted to embeddings as a whole.

Developers have access to five segmentation strategies: fixed, sentence, paragraph, word, and none. The fixed method allows setting the exact chunk size in characters, while sentence and paragraph use logical text boundaries to form fragments. This allows the system to index smaller and semantically more accurate blocks of information.

Using chunking directly within the search engine improves the recall (completeness of search) for large documents. Now users do not need to pre-process data on the application side: Manticore Search takes care of text slicing and subsequent vector representation of each fragment within a single query.

The update is aimed at simplifying the architecture of RAG systems (Retrieval-Augmented Generation). Integrating the chunking logic into the database reduces the load on client-side code and ensures uniform document processing during indexing and search operations.