Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add _safe variants of convert_utf16_to_utf8 #725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
syg opened this issue Mar 21, 2025 · 7 comments
Open

Add _safe variants of convert_utf16_to_utf8 #725

syg opened this issue Mar 21, 2025 · 7 comments
Labels
enhancement New feature or request

Comments

@syg
Copy link

syg commented Mar 21, 2025

It'd be nice to have convert_utf16_to_utf8_{with_errors}_safe that takes an output size and only writes as much as that size. I noticed that there's a output size-taking variant for Latin1, convert_latin1_to_utf8_safe, so there's precedent for this. Understandably this will be slower than assuming the output is sufficiently large in that there'll be a bunch of additional bounds checks.

The full context is that related to #688, another missing piece to using simdutf in V8 is that V8's String::WriteUtf8V2 API tries to be safe and takes an output size. If the required capacity exceeds the actual capacity during transcoding, we try to write as much as can fit into the actual capacity.

That is, while V8's API is also intended to be used in a 2-phase mode of first computing the needed length via String::Utf8LengthV2, there's extra flexibility into allowing people to do partial transcoding bounded by output size. simdutf only allows partial transcoding bounded by input size.

@syg
Copy link
Author

syg commented Mar 21, 2025

cc @anonrig who's probably interested.

@anonrig
Copy link
Member

anonrig commented Mar 21, 2025

Also somewhat related #689

@anonrig anonrig added the enhancement New feature or request label Mar 21, 2025
@lemire
Copy link
Member

lemire commented Mar 22, 2025

@syg Yes. That's reasonable.

@paaspaas00
Copy link

Hi, any news? @lemire

@lemire
Copy link
Member

lemire commented Apr 21, 2025

@paaspaas00 Would you be interested in providing a pull request for this feature?

@paaspaas00
Copy link

I can't unfortunately, just meant to ask if anyone was working on the thing

@lemire
Copy link
Member

lemire commented Apr 22, 2025

@paaspaas00 The simdutf library is being developed actively but we have limited ressources and we tackle issues one by one, not all at once.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants