Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Jan 21, 2026. It is now read-only.

[BREAKING][Refactor] Simplify get_meta loop logics#158

Merged
0oshowero0 merged 2 commits into
devfrom
han/sampler
Dec 23, 2025
Merged

[BREAKING][Refactor] Simplify get_meta loop logics#158
0oshowero0 merged 2 commits into
devfrom
han/sampler

Conversation

@0oshowero0

@0oshowero0 0oshowero0 commented Dec 23, 2025

Copy link
Copy Markdown
Member
  1. Simplify scan_data_status: just return all the samples that meets the requirement. No while loop is needed here.
    def scan_data_status(
        self,
        partition_id: str,
        data_fields: list[str],
        task_name: str,
    ) -> list[int]:
        """
        Find samples that are ready for consumption in a specific partition.
        Delegates scanning functionality to the partition's own method.

        Args:
            partition_id: ID of the partition
            data_fields: List of required field names
            task_name: Name of the consumer task
            batch_size: Number of samples needed

        Returns:
            List of sample indices that are ready for consumption
        """

        partition = self._get_partition(partition_id)
        if not partition:
            return []

        # Use partition's own scanning method
        ready_sample_indices = partition.scan_data_status(data_fields, task_name)

        return ready_sample_indices
  1. Optimize get_metadata loop. We do not need to sample during each loop, just try to scan_data_status.
  2. Reduce duplicated logs.
        if mode == "fetch":
            # Find ready samples within current data partition and package into BatchMeta when reading

            start_time = time.time()
            while True:
                # ready_for_consume_indexes: samples where all required fields are produced
                # (production status is ready) and not yet consumed
                ready_for_consume_indexes = self.scan_data_status(partition_id, data_fields, task_name)

                if len(ready_for_consume_indexes) < batch_size:
                    if time.time() - start_time > TQ_CONTROLLER_GET_METADATA_TIMEOUT:
                        # TODO: non_blocking related logics here @ningbenzhe
                        # if self.non_blocking:
                        #     logger.info()
                        #     return BatchMeta.empty()
                        raise TimeoutError(
                            f"Timeout while waiting for sufficient data for task {task_name}. "
                            f"Required: {batch_size}, Available: {len(ready_for_consume_indexes)}"
                        )
                    logger.warning(
                        f"Insufficient data for task {task_name}. Required: {batch_size} samples with "
                        f"fields {data_fields} in partition {partition_id}, but only have "
                        f"{len(ready_for_consume_indexes)} samples meeting the criteria. "
                        f"Retrying in {TQ_CONTROLLER_GET_METADATA_CHECK_INTERVAL}s..."
                    )
                    time.sleep(TQ_CONTROLLER_GET_METADATA_CHECK_INTERVAL)
                else:
                    break

            if len(ready_for_consume_indexes) < batch_size:
                raise RuntimeError(
                    "Unexpected error: ready_for_consume_indexes has insufficient samples before sampling. "
                )

            batch_global_indexes, consumed_indexes = self.sampler(
                ready_for_consume_indexes,
                batch_size,
                **(sampling_config or {}),
            )

            # Check if we got valid results from the sampler
            if len(batch_global_indexes) != batch_size:
                raise RuntimeError(
                    f"Sampler returned insufficient samples. Please check the sampler logic. "
                    f"Expected: {batch_size}, before sampling: {len(ready_for_consume_indexes)}, "
                    f"after sampling: {len(batch_global_indexes)}"
                )

CC @NINGBENZHE

Summary by CodeRabbit

Release Notes

  • Refactor

    • Improved data readiness verification for transfer queue partitions by switching to partition-scoped checks instead of polling mechanisms. Simplified the public API by removing unnecessary configuration parameters.
  • Bug Fixes

    • Enhanced error messages to include additional context information (task names, partition identifiers, and field details) for improved troubleshooting.

✏️ Tip: You can customize this high-level summary in your review settings.

Copilot AI review requested due to automatic review settings December 23, 2025 07:37
@coderabbitai

coderabbitai Bot commented Dec 23, 2025

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

The scan_data_status method in the controller is simplified by removing batch_size and timeout parameters. The implementation switches from batch-based polling with fixed batch sizes to partition-scoped readiness checks via DataPartitionStatus, with adjusted loop break conditions and error handling.

Changes

Cohort / File(s) Summary
Controller scan_data_status logic
transfer_queue/controller.py
Method signature changed: removed batch_size and timeout parameters. Fetch mode now breaks from polling loop when sufficient ready indices exist. Replaced sampling/retry loop with direct sampler call. Added runtime error validation if sampler returns incorrect batch size. Updated error messages to include task_name, partition_id, and field details.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes


Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

Comment @coderabbitai help to get the list of available commands and usage tips.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the get_metadata method's loop logic by moving retry/timeout handling from scan_data_status to its caller, simplifying the control flow and improving error messages.

Key Changes:

  • Removed retry and timeout logic from scan_data_status, making it a simple delegation function
  • Changed get_metadata loop from using continue to using break with else clause for clearer control flow
  • Improved error messages to be more descriptive with task names, partition IDs, and field information

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread transfer_queue/controller.py Outdated
Comment thread transfer_queue/controller.py Outdated
Comment on lines +864 to 868
if len(ready_for_consume_indexes) < batch_size:
raise RuntimeError(
"Unexpected error: ready_for_consume_indexes has insufficient samples before sampling. "
)

Copilot AI Dec 23, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error check is unreachable code. The loop above at line 839 only breaks when len(ready_for_consume_indexes) >= batch_size, so this condition can never be true. This check should be removed.

Suggested change
if len(ready_for_consume_indexes) < batch_size:
raise RuntimeError(
"Unexpected error: ready_for_consume_indexes has insufficient samples before sampling. "
)

Copilot uses AI. Check for mistakes.
Comment thread transfer_queue/controller.py Outdated
@0oshowero0

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Dec 23, 2025

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Signed-off-by: 0oshowero0 <[email protected]>
@0oshowero0 0oshowero0 changed the title [Refactor] Simplify get_meta loop logics [BREAKING][Refactor] Simplify get_meta loop logics Dec 23, 2025
@0oshowero0 0oshowero0 merged commit 6ee7ae9 into dev Dec 23, 2025
3 checks passed
@0oshowero0 0oshowero0 deleted the han/sampler branch December 23, 2025 08:49
NINGBENZHE pushed a commit that referenced this pull request Dec 23, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants