Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@rustatian
Copy link
Member

@rustatian rustatian commented Apr 25, 2025

Reason for This PR

ref: https://github.com/orgs/roadrunner-server/discussions/2012
closes: roadrunner-server/roadrunner#2091

Description of Changes

  • Handle graceful channel close operations.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the MIT license.

PR Checklist

[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

  • All commits in this PR are signed (git commit -s).
  • The reason for this PR is clearly provided (issue no. or explanation).
  • The description of changes is clear and encompassing.
  • Any required documentation changes (code and docs) are included in this PR.
  • Any user-facing changes are mentioned in CHANGELOG.md.
  • All added/changed functionality is tested.

Summary by CodeRabbit

  • Bug Fixes

    • Improved handling of channel and connection closures to prevent redundant or erroneous close attempts.
    • Enhanced detection and logging for graceful versus unexpected channel closures.
  • Refactor

    • Added more detailed debug logs to clarify the source and outcome of redial events.
    • Updated internal checks to ensure connections and channels are only closed when appropriate.
  • Chores

    • Removed obsolete configuration settings related to relay timeout and log encoding.
    • Updated test expectations to reflect changes in connection closure event logging.

@rustatian rustatian added bug Something isn't working enhancement New feature or request labels Apr 25, 2025
@rustatian rustatian requested a review from Copilot April 25, 2025 19:13
@rustatian rustatian self-assigned this Apr 25, 2025
@coderabbitai
Copy link

coderabbitai bot commented Apr 25, 2025

Walkthrough

The changes in the amqpjobs/redial.go file revise the handling of channel close notifications within a redialer goroutine. The select cases now receive both an error and a boolean indicating if the channel is closed, allowing the logic to distinguish between graceful closes and errors. Logging has been enhanced to provide more detailed information about channel states and redial attempts. Additional safety checks were introduced to prevent attempts to close already closed or nil channels and connections. Comments and debug messages were updated for clarity and precision. In the test configuration .rr-amqp-durability-no-queue.yaml, the relay_timeout and encoding settings were removed. The test TestDurabilityAMQP_NoQueue was updated to expect zero occurrences of a specific log message instead of one.

Changes

File(s) Change Summary
amqpjobs/redial.go Updated select cases to handle both error and closed state; improved logging; added safety checks before closing; clarified comments and debug messages; refined logic for graceful close handling.
tests/configs/.rr-amqp-durability-no-queue.yaml Removed relay_timeout under server and encoding under logs.
tests/jobs_amqp_durability_test.go Modified test assertion in TestDurabilityAMQP_NoQueue to expect zero occurrences of "amqp connection closed" log message snippet.

Poem

In the warren of code, where channels flow,
A rabbit hops softly, ensuring we know:
When channels close gently, don’t leap in alarm,
Just log it, stay calm, and avoid any harm.
With checks and with logs, our tunnels are clear,
The redialer’s watchful—no bugs to fear! 🐇

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (1.64.8)

Error: you are using a configuration file for golangci-lint v2 with golangci-lint v1: please use golangci-lint v2
Failed executing command with error: you are using a configuration file for golangci-lint v2 with golangci-lint v1: please use golangci-lint v2


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 672ee7b and 038c822.

📒 Files selected for processing (2)
  • tests/configs/.rr-amqp-durability-no-queue.yaml (0 hunks)
  • tests/jobs_amqp_durability_test.go (1 hunks)
💤 Files with no reviewable changes (1)
  • tests/configs/.rr-amqp-durability-no-queue.yaml
⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: RabbitMQ plugin (Go stable, PHP 8.4, OS ubuntu-latest)
  • GitHub Check: Analyze (go)
  • GitHub Check: RabbitMQ plugin durability (Go stable, PHP 8.4, OS ubuntu-latest)
🔇 Additional comments (1)
tests/jobs_amqp_durability_test.go (1)

218-218: Assertion adjustment correctly reflects improved connection handling.

This change aligns with the PR objective of improving graceful close handling for AMQP channels. By expecting zero occurrences of the "amqp connection closed" log message instead of one, this test now properly validates that the refined connection handling logic correctly distinguishes between graceful closes and errors.

✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the AMQP driver's redialer to better handle graceful closures of various AMQP channels by checking for both error and closed states. The changes include reading a tuple (err, closed) from the notify channels, refining the clean-up logic with IsClosed() checks, and updating log messages to better reflect the connection state.

Files not reviewed (1)
  • go.work.sum: Language not supported
Comments suppressed due to low confidence (1)

amqpjobs/redial.go:86

  • [nitpick] There is an inconsistency in handling graceful close notifications: connection and consume channels use 'continue' while publish and statistic channels use 'return'. Please confirm if this behavior difference is intentional and consider adding inline comments to clarify the design.
if closed && err == nil { d.log.Debug("[notify close publish channel]: channel is closed")

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (4)
amqpjobs/redial.go (4)

80-81: Incorrect log message for consume channel

The log message indicates the driver is stopped, but it should indicate that the redial channel is full (similar to other handlers).

-					d.log.Debug("[notify close consume channel]: channel is not closed, but driver is stopped")
+					d.log.Debug("[notify close consume channel]: redial channel is full (or redial in progress)")

118-119: Minor formatting issue in log message

There's an extra closing bracket at the end of the log message.

-					d.log.Debug("[notify close statistic channel]: channel is not closed, but driver is stopped]")
+					d.log.Debug("[notify close statistic channel]: channel is not closed, but driver is stopped")

130-131: Inconsistent log message for statistics channel

The statistic channel log message says "redialer stopped" while other handlers use "redial channel is full". For consistency, consider using the same message pattern.

-					d.log.Debug("[notify close statistic channel]: redialer stopped")
+					d.log.Debug("[notify close statistic channel]: redial channel is full (or redial in progress)")

34-132: Suggestion: Extract common patterns into helper methods

The four channel notification handlers follow very similar patterns with repeated code. Consider extracting this pattern into a helper method to improve maintainability and reduce duplication.

+ func (d *Driver) handleChannelClose(channelType string, err *amqp.Error, closed bool, notifySourceName string) bool {
+     // exit on a graceful close
+     if closed && err == nil {
+         d.log.Debug(fmt.Sprintf("[notify close %s channel]: channel is closed", notifySourceName))
+         return true // continue the loop
+     }
+ 
+     // stopped
+     if atomic.LoadUint64(&d.stopped) == 1 {
+         d.log.Debug(fmt.Sprintf("[notify close %s channel]: channel is not closed, but driver is stopped", notifySourceName))
+         return false // return to exit goroutine
+     }
+ 
+     select {
+     case d.redialCh <- &redialMsg{
+         t:   channelType,
+         err: err,
+     }:
+         d.log.Debug(fmt.Sprintf("[notify close %s channel]: redial message was sent", notifySourceName))
+         return false
+     default:
+         d.log.Debug(fmt.Sprintf("[notify close %s channel]: redial channel is full (or redial in progress)", notifySourceName))
+         return false
+     }
+ }

Usage would look like:

 case err, closed := <-d.notifyCloseConnCh:
-    // exit on a graceful close
-    if closed && err == nil {
-        d.log.Debug("[notify close connection channel]: channel is closed")
-        continue
-    }
-
-    // stopped
-    if atomic.LoadUint64(&d.stopped) == 1 {
-        d.log.Debug("[notify close connection channel]: channel is not closed, but driver is stopped")
-        return
-    }
-
-    select {
-    case d.redialCh <- &redialMsg{
-        t:   ConnCloseType,
-        err: err,
-    }:
-        d.log.Debug("[notify close connection channel]: redial message was sent")
-        return
-    default:
-        d.log.Debug("[notify close connection channel]: redial channel is full (or redial in progress)")
-        return
-    }
+    if d.handleChannelClose(ConnCloseType, err, closed, "connection") {
+        continue
+    }
+    return
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e3f1cd7 and 3e225c9.

⛔ Files ignored due to path filters (1)
  • go.work.sum is excluded by !**/*.sum
📒 Files selected for processing (1)
  • amqpjobs/redial.go (5 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: RabbitMQ plugin (Go stable, PHP 8.4, OS ubuntu-latest)
  • GitHub Check: RabbitMQ plugin durability (Go stable, PHP 8.4, OS ubuntu-latest)
  • GitHub Check: Analyze (go)
🔇 Additional comments (8)
amqpjobs/redial.go (8)

34-39: Good addition for handling graceful closes

The change to receive both the error and a closed boolean from the channel is a good improvement. This properly distinguishes between graceful closes (closed && err == nil) and actual errors that require redial.


42-45: Clearer logging for driver stopped state

The updated debug log message is more descriptive, making it clear that while the channel isn't closed, the driver is stopped. This helps with troubleshooting connection states.


52-56: Enhanced logging for redial operations

The log messages now clearly indicate whether a redial message was sent or if the channel is full. This improvement helps track the redial process flow.


59-64: Consistent pattern for graceful close handling

The consume channel handler correctly follows the same pattern established for connection channels, improving consistency in error handling across different channel types.


172-177: Good safety check for consumeChan

Adding the nil check and IsClosed() verification before attempting to close the channel prevents potential errors when operating on already closed or nil channels. This is a good defensive programming practice.


179-184: Good safety check for connection

Similar to the channel check, this safety check for the connection prevents attempts to close already closed connections, which is a good practice.


193-194: Improved reset method with better logging and connection checks

The reset method now includes better debug logging and properly checks if the connection is not closed before attempting to close it, which is a good safety measure.

Also applies to: 213-220


235-236: Improved comment clarity

The updated comment more clearly describes what the redial function does - it trashes broken publishing channels and closes the connection.

rustatian and others added 4 commits April 25, 2025 21:26
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
Signed-off-by: Valery Piashchynski <[email protected]>
@rustatian rustatian merged commit 971b2c7 into master Apr 26, 2025
9 checks passed
@rustatian rustatian deleted the fix/redialer-sometimes-does-not-resume-pipelines branch April 26, 2025 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[🐛 BUG]: Redialer does not resume pipeline after delivery channel was closed by RabbitMQ

2 participants