Codestin Search App

0oshowero0 · 2025-12-24T12:00:57Z

Provide async_clear_partition(previous clear) and async_clear_samples method for client. Through the async_clear_samples interface, we can let user to clear specific samples rather than determine the whole partition.

    async def async_clear_samples(self, metadata: BatchMeta):
        """Asynchronously clear samples from all storage units and controller.

        Args:
            metadata: The BatchMeta of the corresponding data to be clear

        Raises:
            RuntimeError: If clear operation fails
        """

Modify the default value of env var TQ_NUM_THREADS to 8.
Simplify some duplicated logs.

Summary by CodeRabbit

New Features
- Added clear_samples() method to selectively clear specific data samples within a partition.
Documentation
- Updated tutorials to reflect new partition-clearing API usage.
Chores
- Renamed clear() method to clear_partition() for improved clarity on partition-level operations.
- Adjusted default worker thread concurrency from 16 to 8.
- Enhanced test coverage for partition and sample clearing operations.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Signed-off-by: 0oshowero0 <[email protected]>

coderabbitai · 2025-12-24T12:01:03Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

This PR refactors the transfer queue client and controller APIs, renaming clear() to clear_partition(), introducing new clear_samples() and async_clear_samples() methods for batch-level sample clearing, and updating corresponding internal methods and ZMQ request types. Default thread concurrency reduced from 16 to 8.

Changes

Cohort / File(s)	Summary
Core Client API `transfer_queue/client.py`	Renamed `async_clear()` → `async_clear_partition()` and `_get_clear_meta()` → `_get_partition_meta()`. Added new `async_clear_samples(metadata)` method to clear specific samples by batch. Added `_clear_meta_in_controller()` to handle partition metadata clearing. Renamed `_clear_controller()` → `_clear_partition_in_controller()`. Introduced sync wrappers `clear_partition()` and `clear_samples()`. Updated controller request types to use `CLEAR_PARTITION` and `GET_PARTITION_META`. Reduced default thread count from 16 to 8.
Core Controller API `transfer_queue/controller.py`	Renamed `release_indexes(partition_id)` → `release_partition(partition_id)`. Added new `release_indexes(partition_id, indexes_to_release)` for selective index release. Renamed `clear(partition_id, ...)` → `clear_partition(partition_id, ...)`. Added `clear_meta(global_indexes, partition_ids, ...)` method for clearing specific samples across partitions using index manager. Updated `DataPartitionStatus.clear_data()` signature from `global_indexes_range` to `indexes_to_release`. Updated request handling for `CLEAR_PARTITION` and `GET_PARTITION_META`. Imported `groupby` from itertools for partition-based batch processing.
Message Protocol `transfer_queue/utils/zmq_utils.py`	Renamed enum members `GET_CLEAR_META` → `GET_PARTITION_META` and `GET_CLEAR_META_RESPONSE` → `GET_PARTITION_META_RESPONSE`. Added new enum members `CLEAR_PARTITION` and `CLEAR_PARTITION_RESPONSE`.
Storage Configuration `transfer_queue/storage/simple_backend.py`	Reduced default `TQ_NUM_THREADS` from 16 to 8.
Test Suite `tests/test_client.py`	Removed `GET_CLEAR_META` handling. Updated `CLEAR_META` response from "clear ok" to "clear meta ok". Added `CLEAR_PARTITION` and `GET_PARTITION_META` request type handling with mock responses. Expanded test coverage with `async_clear_partition()`, `clear_partition()`, `async_clear_samples()`, `clear_samples()`, and `async_clear_samples_with_empty_metadata()` test cases. Added multi-partition and multi-storage scenario tests. Removed `test_clear_operation`.
Controller Tests `tests/test_controller.py`	Replaced `clear()` calls with `clear_partition()` in two test locations. Added new `test_controller_clear_meta()` to validate `clear_meta()` functionality for selective sample clearing, including metadata creation, status updates, and verification.
Data Partition Tests `tests/test_controller_data_partitions.py`	Removed explicit return value capture and assertion on `clear_data()` call in `test_data_partition_status_advanced`.
Demo Scripts `recipe/simple_use_case/async_demo.py`, `recipe/simple_use_case/sync_demo.py`	Updated API calls from `async_clear(partition_id=...)` → `async_clear_partition(partition_id=...)` and `clear(partition_id=...)` → `clear_partition(partition_id=...)`.
Tutorial Updates `tutorial/01_core_components.py`, `tutorial/02_metadata_concepts.py`, `tutorial/03_understanding_controller.py`, `tutorial/04_custom_sampler.py`	Updated all cleanup finalization calls from `clear(partition_id=...)` → `clear_partition(partition_id=...)` across multiple demo workflows. Tutorial 01 adds message mentioning alternative `clear_samples()` option.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 The clearing dance is now refined,
Partitions and samples, clearly designed,
Less threads but more control we find,
Eight paths forward, metadata aligned!
Hop along with our new API, so kind! 🎉

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Copilot

Pull request overview

This PR introduces a breaking change that adds fine-grained clearing capabilities to the TransferQueue system. The main change renames the existing clear() method to clear_partition() for clarity and adds a new clear_samples() method that allows clearing specific samples while preserving the partition.

Key changes:

Renamed clear() to clear_partition() in both client and controller
Added clear_samples() method for fine-grained sample clearing
Refactored ZMQ request types (renamed GET_CLEAR_META to GET_PARTITION_META, added CLEAR_PARTITION separate from CLEAR_META)
Updated PartitionIndexManager with new release_partition() and release_indexes() methods
Reduced default thread count from 16 to 8 in TQ_NUM_THREADS

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
transfer_queue/controller.py	Added `clear_meta()` method for sample-level clearing, renamed `clear()` to `clear_partition()`, refactored index manager with separate release methods
transfer_queue/client.py	Renamed `clear()` to `clear_partition()`, added `clear_samples()` method, refactored internal helper methods to distinguish partition vs sample clearing
transfer_queue/utils/zmq_utils.py	Renamed and added ZMQ request types to distinguish partition-level vs sample-level operations
transfer_queue/storage/simple_backend.py	Changed default `TQ_NUM_THREADS` from 16 to 8
tests/test_controller.py	Added test for `clear_meta()` functionality, updated tests to use renamed `clear_partition()`
tests/test_client.py	Added comprehensive tests for both `clear_partition()` and `clear_samples()` methods, updated mock controller to handle new request types
tutorial/*.py	Updated all tutorial examples to use renamed `clear_partition()` method
recipe/simple_use_case/*.py	Updated demo files to use renamed `clear_partition()` method

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-24T12:05:19Z

+            await self._clear_partition_in_controller(partition_id)

            # Clear storage unit data
            await self.storage_manager.clear_data(metadata)


Operation ordering issue: The partition is cleared in the controller (line 413) before clearing storage (line 416). If the storage clear operation fails, the controller metadata has already been cleared, leaving the system in an inconsistent state. Consider either clearing storage first, or implementing a rollback mechanism, or at minimum documenting this behavior in the method's docstring.

Copilot · 2025-12-24T12:05:19Z

+            # Clear the controller metadata
+            await self._clear_meta_in_controller(metadata)
+
+            # Clear storage unit data
+            await self.storage_manager.clear_data(metadata)
+


Operation ordering issue: Samples are cleared in the controller (line 442) before clearing storage (line 445). If the storage clear operation fails, the controller metadata has already been cleared, leaving the system in an inconsistent state. Consider either clearing storage first, or implementing a rollback mechanism, or at minimum documenting this behavior in the method's docstring.

Suggested change

# Clear the controller metadata

await self._clear_meta_in_controller(metadata)

# Clear storage unit data

await self.storage_manager.clear_data(metadata)

# Clear storage unit data first

await self.storage_manager.clear_data(metadata)

# Clear the controller metadata

await self._clear_meta_in_controller(metadata)

Signed-off-by: 0oshowero0 <[email protected]>

0oshowero0 · 2025-12-24T12:20:51Z

@coderabbitai review

coderabbitai · 2025-12-24T12:20:56Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-24T12:24:53Z

+
+            if not self._controller:
+                raise RuntimeError("No controller registered")
+


Consider adding validation to check if metadata is empty (no samples) before proceeding with the clear operation. While the controller's clear_meta can handle empty lists safely, it's more efficient and clearer to validate and potentially short-circuit early or log a warning when clearing with empty metadata.

Suggested change

# Short-circuit if metadata contains no samples to clear

if (

not getattr(metadata, "global_indexes", None)

and not getattr(metadata, "partition_ids", None)

):

logger.debug(

f"[{self.client_id}]: Skipping clear_samples; received empty metadata."

)

return

Signed-off-by: 0oshowero0 <[email protected]>

0oshowero0 added 4 commits December 24, 2025 17:17

add clear methods

10a87ab

Signed-off-by: 0oshowero0 <[email protected]>

update

358a68a

Signed-off-by: 0oshowero0 <[email protected]>

fix ut

598acc5

Signed-off-by: 0oshowero0 <[email protected]>

update other parts

a099b82

Signed-off-by: 0oshowero0 <[email protected]>

Copilot AI review requested due to automatic review settings December 24, 2025 12:00

Copilot started reviewing on behalf of 0oshowero0 December 24, 2025 12:01 View session

Copilot AI reviewed Dec 24, 2025

View reviewed changes

0oshowero0 changed the title ~~[BREAKING][Feat] Add fine-grained clear method~~ [BREAKING][Feat] Add fine-grained clear method & optimize log Dec 24, 2025

fix

e4fa55c

Signed-off-by: 0oshowero0 <[email protected]>

0oshowero0 requested a review from Copilot December 24, 2025 12:21

Copilot started reviewing on behalf of 0oshowero0 December 24, 2025 12:21 View session

Copilot AI reviewed Dec 24, 2025

View reviewed changes

0oshowero0 added 2 commits December 24, 2025 20:29

fix

7104ad0

Signed-off-by: 0oshowero0 <[email protected]>

fix

38b3af9

Signed-off-by: 0oshowero0 <[email protected]>

0oshowero0 merged commit c5dfd6a into dev Dec 24, 2025
3 checks passed

0oshowero0 mentioned this pull request Dec 31, 2025

[TODO] TODO List #128

Open

22 tasks

0oshowero0 mentioned this pull request Jan 14, 2026

[TODO] Long-term TODO list Ascend/TransferQueue#2

Closed

22 tasks


		if not self._controller:
		raise RuntimeError("No controller registered")

+            # Short-circuit if metadata contains no samples to clear
+            if (
+                not getattr(metadata, "global_indexes", None)
+                and not getattr(metadata, "partition_ids", None)
+            ):
+                logger.debug(
+                    f"[{self.client_id}]: Skipping clear_samples; received empty metadata."
+                )
+                return

Uh oh!

Conversation

0oshowero0 commented Dec 24, 2025 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Other AI code review bot(s) detected

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0oshowero0 commented Dec 24, 2025

Uh oh!

coderabbitai Bot commented Dec 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

0oshowero0 commented Dec 24, 2025 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Dec 24, 2025 •

edited

Loading