count + grouping on text index #118385

You are right to question that recommendation: my earlier suggestion to reduce the index GRANULARITY was incorrect. ClickHouse text indexes deliberately use “infinite granularity” (100 million), so that value is not a tuning knob like it is for conventional skip indexes: https://clickhouse.com/docs/reference/engines/table-engines/mergetree-family/textindexes#index-granularity.

The 5-gram index can still return exact LIKE results for needles shorter than five characters because ClickHouse verifies the predicate against the rows, but the index cannot derive a 5-gram from such a needle, so it cannot make that case selective. Moving to 3-grams may increase recall so much that it reads more rows, exactly as you observed.

You also did not miss a generic projection optimization. A projection definition cannot contain a WHERE clause, and an aggregate projection or materialized view can only pre-aggregate dimensions known at insert time. An arbitrary runtime LIKE/hasAllTokens substring is not such a dimension; after the text index identifies matching rows, ClickHouse still has to compute the grouping sets over those rows. A materialized view helps only if the search vocabulary can be normalized into a bounded token dimension at ingestion and aggregates are stored by token plus each facet. Expanding arbitrary n-grams for 140 billion rows would usually be prohibitively large.

For this workload, I would keep the tokenizer that gives the best measured selectivity, apply any date/partition/tenant predicates before the text predicate, and compare read_rows/read_bytes in system.query_log. Common three-character searches that occur in a large fraction of rows are inherently expensive; there is no projection that can precompute exact facets for every possible substring without effectively materializing the search space.