I'd like to outline the challenges in doing this to explain why I don't expect to support this any time soon, or really ever.

Tokenization around emojis is absurdly difficult. When processing a message to be counted for frequentcy, I explicitly exclude several objects which are difficult to tokenize- Emoji, codeblocks, URLs, Discord's emotes and mentions, and [[references]]. Each of these is relatively easy to detect in a stream of characters via regex, but are unreasonable to detect after tokenization. My tokenizer assumes that all tokens can be divided into word-candidates (contiguous writing characters) and non-candidates (contiguous "punctuation"). Granted, I did add an exception for some kinds of intra-word punctuation, but that was very specific and small in scope. Emoji "punctuation" is nightmarish.

Variation selectors break a core assumption of my tokenizer, because they are considered a non-spacing mark according to Unicode's character classification, which means it is inherited into the \p{Punct} regex character class. But they follow the character they correspond to, rather than being between other characters. This makes sense for the character, but makes them hard to deliver accurately.
Zero width space is also considered punctuation, insofar as being a format character.
Regional Indicators, by definition, don't have any joiners and don't necessarily have a single representation.

I can do some experiments with including these characters as writing characters, since they don't generally appear with toki pona's latin script, but that still isn't a near-future thing. I have higher priorities for now.

Doesn't support Emoji #18

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions