HDFS-16642. [SBN read] Moving the selecting inputstream from journalnode in EditLogTailer out of FSNLock #4497
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
Looks thread-safe to me. Has this patch been tested in a production environment? |
@jojochuang Thanks for your review and sorry sorry for replaying so late. We have release it in our prod environment and it works well. After checking some jstacks of ObserverNamenodes, I found that there are no handlers blocked at @ferhui can you help me review this patch? |
|
It looks good. The failed case TestDiskBalancerCommand seems unrelated to changes. @ZanderXu could you confirm? |
ferhui
approved these changes
Aug 3, 2022
|
@ferhui Sir, thank you very much for your helping me to review this patch.
|
|
The PR has hung there for 10 days. @ZanderXu could you please push an empty commit to trigger CI? We can avoid potential conflicts and merge it after CI runs. |
|
Copy, sir. I will rebase it on the latest the trunk. |
…LogTailer outof FSNLock
|
💔 -1 overall
This message was automatically generated. |
|
emm... cc @ferhui |
|
@ZanderXu It is unrelated to this PR. can create a ticket to track it. |
|
@ZanderXu Thanks for your contribution. @jojochuang Thanks for your review! Merged |
|
@jojochuang @ferhui Sir, thank you very much for helping review. |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…LogTailer outof FSNLock (apache#4497)
NyteKnight
pushed a commit
to NyteKnight/hadoop
that referenced
this pull request
Jun 25, 2024
…itioning to Active with in-progress tailer (apache#194) * HDFS-16642. Moving the selecting inputstream from journalnode in EditLogTailer outof FSNLock (apache#4497) * HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (apache#4744) * remove annotation introduced in merge wrongly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
In our prod environment, 8020 Handler of Observer NameNode also blocked by EditlogTailer.
And EditlogTailer cost a long time for selecting InputStreams from Journalnode with holding the FSNLock.
During this period, 8020 handlers of Observer NameNode will be blocked by the FSN Lock.
In theory, selecting inputstreams from JournalNode does not involve changing memory information in NameNode, so we can move the selecting out of the FSN Lock, and it can improve the throughput of Observer NameNode.
After reading the relevant code in depth, I found that it is feasible to move the selecting inputStream out of the FSNLock.