Doing the conversion only if necessary sounds like a great idea 👍

Hi :D
How long do you think you'll need to finish this feature?

It would be awesome if you can get it ready soon-ish, I'd love if luasnip immediately handles unicode correctly.
If it takes some more time, that's alright too, not supporting transforming unicode is not a huge loss IMO

I think I does work correctly, but I don't have a good idea on what to test yet. There are some lists like https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt and https://www.cl.cam.ac.uk/~mgk25/ucs/examples/quickbrown.txt. I would also like some malformed utf-8. This is arguably more important than being correct because a crash here will crash neovim.

Good point!

I guess quickjs's applying a regex to unicode is well tested, so there would need to be tests for transforming utf8 to utf16 (and back)?

@L3MON4D3 Ok, feeding random bytes into lre_compile will crash when repeated tens of thousands of times. I think checking for valid utf8 can be done if we don't trust the user. Stopping the conversion from utf8 to utf16 prevents crashing during matching. Would you expect some kind of error in that case? Or just no matches?

An error sounds appropriate 👍
Better to distinguish between error and no match (we should just log errors).

I went with returning nil, err on error.

For the crashing, it looks like lre_compile segfaults if the input string contains 0xfd. This byte used to indicate the beginning of a six byte sequence which is unneeded and was removed. Valid unicode can not contain 0xfd so we can check for that.

Nice, that sounds good
Looking at the spec, 0xfcindicate six bytes as well, does it also crash?

No, only 0xfd, and only in combination with other non-ascii chars. I also cannot reproduce it outside of the lua/luajit context. I think I will merge #8 and then this PR later today, seems easier in this order.

No, only 0xfd, and only in combination with other non-ascii chars.

Okay, that's good I guess :D

I think I will merge #8 and then this PR later today, seems easier in this order.

Niiice, TY for getting it done this quickly ❤️