Bug in beta tokenizer #178
Description
Just looking at the code I think there is a bug here:
This only matches litteral periods .. I assume that you meant to match any character /(?<unknown>.+?)/.
Without this change (now that it's using .matchAll()) anything in the input SQL string that does not match the regex will be removed from the results and not visible to the user. I assume that is undesired.
(Also of note that this approach will at most match a single character in every unknown group match.)
And a bit off topic, but I'm using this highlighter with a non-standard SQL flavor, which uses several additional keywords. These keywords will now be highlighted as identifiers. I hope the highlighting for those is not too disturbing. 😅