-
Notifications
You must be signed in to change notification settings - Fork 87
Add _safe variants of convert_utf16_to_utf8 #725
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
cc @anonrig who's probably interested. |
Also somewhat related #689 |
@syg Yes. That's reasonable. |
Hi, any news? @lemire |
@paaspaas00 Would you be interested in providing a pull request for this feature? |
I can't unfortunately, just meant to ask if anyone was working on the thing |
@paaspaas00 The simdutf library is being developed actively but we have limited ressources and we tackle issues one by one, not all at once. |
It'd be nice to have
convert_utf16_to_utf8_{with_errors}_safe
that takes an output size and only writes as much as that size. I noticed that there's a output size-taking variant for Latin1,convert_latin1_to_utf8_safe
, so there's precedent for this. Understandably this will be slower than assuming the output is sufficiently large in that there'll be a bunch of additional bounds checks.The full context is that related to #688, another missing piece to using simdutf in V8 is that V8's
String::WriteUtf8V2
API tries to be safe and takes an output size. If the required capacity exceeds the actual capacity during transcoding, we try to write as much as can fit into the actual capacity.That is, while V8's API is also intended to be used in a 2-phase mode of first computing the needed length via
String::Utf8LengthV2
, there's extra flexibility into allowing people to do partial transcoding bounded by output size.simdutf
only allows partial transcoding bounded by input size.The text was updated successfully, but these errors were encountered: