fix: escape control characters in TOML string renderers#3402
Open
marcelsafin wants to merge 2 commits into
Open
fix: escape control characters in TOML string renderers#3402marcelsafin wants to merge 2 commits into
marcelsafin wants to merge 2 commits into
Conversation
TOML forbids raw U+0000-U+0008, U+000B, U+000C, U+000E-U+001F, U+007F in all string forms, and bare CR outside CRLF. Both TOML renderers passed these through, producing files tomllib rejects. Route strings containing forbidden characters to an escaped basic string via a shared toml_escape_basic helper. Fixes github#3340 Co-authored-by: Copilot <[email protected]>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes TOML command generation by ensuring raw control characters (and bare \r not in a CRLF pair) are never emitted directly into TOML strings, preventing tomllib parse failures for affected descriptions/prompts/bodies.
Changes:
- Added
toml_escape_basic()plus a shared forbidden-control detection regex, and routedTomlIntegration._render_toml_string()through the helper when control chars/bare CR are present. - Updated
CommandRegistrar.render_toml_command()/_render_basic_toml_string()to use the shared escaping helper and to avoid multiline TOML forms when forbidden control chars are present. - Added a dedicated regression test suite that round-trips problematic inputs through
tomllibfor both renderer code paths.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/integrations/test_toml_control_chars.py | Adds regression tests covering control chars + bare CR handling for both TOML renderers. |
| src/specify_cli/integrations/base.py | Introduces shared TOML basic-string escaping and ensures _render_toml_string() switches to escaped basic strings when required. |
| src/specify_cli/agents.py | Uses shared escaping for basic-string rendering and avoids multiline forms when TOML-forbidden control characters are present. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+261
to
+266
| # Control characters TOML forbids in raw form (and bare CR) cannot | ||
| # appear in either multiline form, so those bodies also route to the | ||
| # escaped basic string. | ||
| from specify_cli.integrations.base import _TOML_FORBIDDEN_CTRL | ||
|
|
||
| has_forbidden_ctrl = bool(_TOML_FORBIDDEN_CTRL.search(body)) |
Contributor
Author
There was a problem hiding this comment.
Good point. Added a public toml_contains_forbidden_ctrl helper in integrations.base and switched both call sites to it; the regex stays private.
Avoids importing the module-private regex across modules. Co-authored-by: Copilot <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3340
TOML forbids raw control characters (U+0000-U+0008, U+000B, U+000C, U+000E-U+001F, U+007F) in every string form, and bare CR outside a CRLF pair. Both TOML renderers (
TomlIntegration._render_toml_stringandCommandRegistrar.render_toml_command) passed these through untouched, so a description or command body containing e.g. a NUL or escape byte produced a TOML file thattomllibrejects.The only valid representation for these characters is an escaped basic string, so both renderers now route such strings through a shared
toml_escape_basichelper inintegrations/base.py. It escapes backslash, quote, \n, \r, \t, and emits the remaining forbidden characters as\uXXXX. Clean strings keep their current rendering (multiline/literal forms), so existing output is unchanged.Testing
uv run specify --helpuv sync && uv run pytest(3848 passed)New test file
tests/integrations/test_toml_control_chars.py(39 tests) round-trips every affected character class throughtomllib.loadsfor both renderers, and verifies clean bodies still use the multiline form.AI Disclosure
Fix and tests written with GitHub Copilot CLI; I reviewed the diff and test results.