Eh? The tokenizers are fully pluggable, the Unicode and ICU tokenizers will be good for many purposes and you can always drop something specific in.

Eg https://github.com/wangfenjin/simple

I know it’s extensible, with original work. ICU isn’t helpful for some languages

For what languages? Why ICU isn't helpful? My naive guess would be "it works for anything supported by Unicode".

languages that don't use spaces like Japanese

Of all/many opensource projects I looked at, only meilisearch is good for non-English languages: https://www.meilisearch.com/docs/learn/resources/language

Unfortunately it still works very poorly for languages with compound words (German, Swedish, Finnish) where you would usually want to surface search results if someone uses the second or third part of the compound word, otherwise the user has to enumerate the (often too many) variants. Meilisearch doesn’t support suffix search and their tokenizer can’t handle these languages.

For instance searching for ‘___missilestrike’ in Kyiv would ideally surface all incidents, regardless of launch mode (surface-to-surface, air-to-surface…), range, propulsion, warhead, or guidance system. In these languages a number of these would form the prefix of that compound word.

Japanese support looks good. But it looks like I can’t deploy this inside an iOS app :(

I need an offline client side db/ fts (for this language learning app: https://reader.manabi.io )

To my knowledge, meili is free software (MIT license) so theoretically, it should be possible to embed it to an IOS app.

it's more about the tech it uses... it's oriented toward docker based deploy and is made in rust so maybe it can be ported to iOS, I'd have to spend time forking the packaging. looks like it's geared toward server environments