Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

ShaharNaveh
Copy link
Collaborator

@ShaharNaveh ShaharNaveh commented Sep 3, 2025

Follow up on #6121 (review)

Summary by CodeRabbit

  • Refactor

    • Replaced the internal expression unparser with an integrated AST-to-source generator, updating internal usage accordingly.
    • Removed the obsolete unparser module to streamline code generation paths.
  • Chores

    • Added new workspace dependencies to support parsing, code generation, and source handling.
    • Updated build configuration to include the new components.

No user-facing behavior changes are expected.

Copy link
Contributor

coderabbitai bot commented Sep 3, 2025

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (1)
  • Lib/test/test_typing.py is excluded by !Lib/**

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This change removes the local unparser and switches code generation to use Ruff’s codegen stack. Cargo manifests add ruff-related dependencies. compile.rs introduces a new private helper to unparse expressions via ruff generators and updates call sites. The old unparse module and its public UnparseExpr type are deleted.

Changes

Cohort / File(s) Summary
Workspace dependencies
Cargo.toml
Adds ruff_python_codegen (git: Astral-Sh/ruff, tag 0.11.0) to workspace dependencies.
Codegen crate dependencies
compiler/codegen/Cargo.toml
Adds ruff_python_parser, ruff_python_codegen, ruff_source_file under [dependencies]; adds ruff_python_parser under [dev-dependencies].
Replace local unparser with Ruff codegen
compiler/codegen/src/compile.rs
Adds private fn unparse_expr(&Expr) -> String using Ruff’s Generator and LineEnding; replaces previous UnparseExpr call sites for mapping-pattern keys and future-annotation strings; imports LineEnding; removes UnparseExpr import.
Module cleanup
compiler/codegen/src/lib.rs
Removes mod unparse; declaration.
Remove unparser implementation
compiler/codegen/src/unparse.rs
Deletes the entire local unparser, including public UnparseExpr type, its constructor, Display impl, and all helper logic.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    actor Compiler as Codegen (compile.rs)
    participant Helper as unparse_expr (new)
    participant RuffGen as ruff_python_codegen::Generator
    participant Stylist as ruff_python_parser::Stylist (derived)
    note over Compiler: When needing a string form of an Expr

    Compiler->>Helper: unparse_expr(expr)
    Helper->>Stylist: derive stylist (via dummy parse)
    Helper->>RuffGen: configure with LineEnding + stylist
    RuffGen-->>Helper: generate source string
    Helper-->>Compiler: String

    note over Compiler: Used for mapping keys and future annotations
Loading
sequenceDiagram
    autonumber
    actor Compiler as Codegen
    participant Old as UnparseExpr (removed)
    participant New as unparse_expr + Ruff Generator

    rect rgb(245,245,255)
    note right of Old: Old flow (removed)
    Compiler->>Old: UnparseExpr::new(expr, &source_file)
    Old-->>Compiler: to_string()
    end

    rect rgb(245,255,245)
    note right of New: New flow
    Compiler->>New: unparse_expr(expr)
    New-->>Compiler: String
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • youknowone

Poem

A nibble of code, a twitch of nose,
I swapped our quill for Ruff-er prose.
Old unparser hops away,
New generator leads the way.
With gentle paws I press “compile”—
Strings unspool in tidy style. 🐇✨

✨ Finishing Touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@ShaharNaveh ShaharNaveh changed the title Use ruff for Expr nparsing Use ruff for Expr unparsing Sep 3, 2025
@@ -147,6 +147,19 @@ enum ComprehensionType {
Dict,
}

fn unparse_expr(expr: &Expr) -> String {
// Hack, because we can't do `ruff_python_codegen::Indentation::default()`
// https://github.com/astral-sh/ruff/pull/20216
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that it was already merged astral-sh/ruff#20216 (that was fast 😅 )

I'll wait for it be released, and I wanted to upgrade ruff anyway so it will be a good excuse to do so

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
compiler/codegen/Cargo.toml (1)

34-34: Remove redundant dev-dependency on ruff_python_parser

It’s already a normal dependency due to compile-time usage; duplicating it in dev-deps is unnecessary.

 [dev-dependencies]
-ruff_python_parser = { workspace = true }
 insta = { workspace = true }
compiler/codegen/src/compile.rs (2)

150-162: Avoid reparsing for every unparse and stabilize line endings

Unconditionally reparsing "x = 1" on every call is unnecessary. Cache the computed indentation with OnceLock and force LF for reproducible output across platforms.

-fn unparse_expr(expr: &Expr) -> String {
-    // Hack, because we can't do `ruff_python_codegen::Indentation::default()`
-    // https://github.com/astral-sh/ruff/pull/20216
-    let indentation = {
-        let contents = r"x = 1";
-        let module = ruff_python_parser::parse_module(contents).unwrap();
-        let stylist = ruff_python_codegen::Stylist::from_tokens(module.tokens(), contents);
-        stylist.indentation().clone()
-    };
-
-    ruff_python_codegen::Generator::new(&indentation, LineEnding::default()).expr(expr)
-}
+fn unparse_expr(expr: &Expr) -> String {
+    static INDENT: std::sync::OnceLock<ruff_python_codegen::Indentation> =
+        std::sync::OnceLock::new();
+    let indentation = INDENT.get_or_init(|| {
+        // Hack, until Indentation::default() is available (see ruff PR #20216)
+        let contents = "x = 1";
+        let module = ruff_python_parser::parse_module(contents)
+            .expect("parse_module('x = 1') should never fail");
+        let stylist = ruff_python_codegen::Stylist::from_tokens(module.tokens(), contents);
+        stylist.indentation().clone()
+    });
+    ruff_python_codegen::Generator::new(indentation, LineEnding::Lf).expr(expr)
+}

3608-3616: Duplicate-key detection now relies on Ruff unparse; add coverage for semantic equivalence

Switching to Ruff’s generator likely normalizes literals (e.g., 1_0 vs 10, different string quotes). Please add/adjust tests to ensure duplicates are rejected based on semantic equality, not just textual quirks.

Would you like me to add regression tests covering numbers with underscores, different string quote styles, and mixed-case hex literals?

📜 Review details

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 056795e and 87dd525.

⛔ Files ignored due to path filters (12)
  • Cargo.lock is excluded by !**/*.lock
  • Lib/test/test_future_stmt/badsyntax_future.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/badsyntax_future3.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/badsyntax_future4.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/badsyntax_future5.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/badsyntax_future6.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/badsyntax_future7.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/badsyntax_future8.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/badsyntax_future9.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/import_nested_scope_twice.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/nested_scope.py is excluded by !Lib/**
  • Lib/test/test_future_stmt/test_future.py is excluded by !Lib/**
📒 Files selected for processing (5)
  • Cargo.toml (1 hunks)
  • compiler/codegen/Cargo.toml (1 hunks)
  • compiler/codegen/src/compile.rs (4 hunks)
  • compiler/codegen/src/lib.rs (0 hunks)
  • compiler/codegen/src/unparse.rs (0 hunks)
💤 Files with no reviewable changes (2)
  • compiler/codegen/src/lib.rs
  • compiler/codegen/src/unparse.rs
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.rs: Follow the default rustfmt code style (cargo fmt to format)
Always run clippy to lint code (cargo clippy) before completing tasks. Fix any warnings or lints that are introduced by your changes
Follow Rust best practices for error handling and memory management
Use the macro system (pyclass, pymodule, pyfunction, etc.) when implementing Python functionality in Rust

Files:

  • compiler/codegen/src/compile.rs
🧠 Learnings (2)
📓 Common learnings
Learnt from: CR
PR: RustPython/RustPython#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T03:16:56.511Z
Learning: Applies to **/*.py : Use ruff for linting Python code
📚 Learning: 2025-07-19T03:16:56.511Z
Learnt from: CR
PR: RustPython/RustPython#0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-07-19T03:16:56.511Z
Learning: Applies to **/*.py : Use ruff for linting Python code

Applied to files:

  • Cargo.toml
  • compiler/codegen/Cargo.toml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
  • GitHub Check: Run snippets and cpython tests (windows-latest)
  • GitHub Check: Run snippets and cpython tests on wasm-wasi
  • GitHub Check: Check the WASM package and demo
  • GitHub Check: Check Rust code with rustfmt and clippy
  • GitHub Check: Run snippets and cpython tests (ubuntu-latest)
  • GitHub Check: Run rust tests (macos-latest)
  • GitHub Check: Run rust tests (ubuntu-latest)
  • GitHub Check: Run snippets and cpython tests (macos-latest)
  • GitHub Check: Run rust tests (windows-latest)
  • GitHub Check: Ensure compilation on various targets
🔇 Additional comments (4)
compiler/codegen/Cargo.toml (1)

16-18: Ruff parser/codegen/source_file deps: LGTM

These additions align with the new unparse path in compile.rs.

Cargo.toml (1)

165-165: Workspace adds ruff_python_codegen: LGTM

Keeps all ruff crates pinned to the same tag; good for API coherence.

compiler/codegen/src/compile.rs (2)

33-33: Import of LineEnding: LGTM

Needed for Generator construction.


4160-4163: Future annotations stringification parity

Using Ruff for annotation strings is fine; please verify parity with CPython across edge cases (unions, generics, nested subscripts, parentheses, and starred PEP 646 forms) to avoid subtle mismatches in annotations.

I can draft insta tests comparing against CPython’s unparse for a corpus of annotations if helpful.

@@ -4952,6 +4952,7 @@ def barfoo(x: AT): ...
def barfoo2(x: CT): ...
self.assertIs(get_type_hints(barfoo2, globals(), locals())['x'], CT)

@unittest.expectedFailure # TODO: RUSTPYTHON; 'List[list["C2"]]' != "List[list['C2']]"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any option to change the quote style?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No:/ not from the Generator directly, I think that this needs to be fixed upstream (over Ruff's side)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once Ruff forked rustpython-parser, we patched unparse to decide quote by option. That might be removed during refactoring. I have no idea if Ruff maintainers are interested in adding feature only for downstream user. Probably asking if possible be worth. Otherwise fork? but in easier form to sync.

@ShaharNaveh
Copy link
Collaborator Author

ngl, I have mixed feelings about this PR. on one hand it's a major code reduction, and on the other hand there is a regression when it comes to the test cases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants