[ADBDEV-6855] Rebalance automation #1236
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
63301bc to
59ff18b
Compare
59ff18b to
01a6b13
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Member
Still managed to successfully launch rebalance in one container! Had to fix paths and ports.
But why so slow?! Even on an empty cluster!
gprebalance -n 1
20250227:10:03:30:473322 gprebalance:cdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 7.2.0_arenadata7+dev.30.gba7c7afd9fc build dev'
20250227:10:03:30:473322 gprebalance:cdw:gpadmin-[INFO]:-coordinator Greenplum Version: 'PostgreSQL 12.12 (Greenplum Database 7.2.0_arenadata7+dev.30.gba7c7afd9fc build dev) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit compiled on Feb 27 2025 04:40:54 (with assert checking) Bhuvnesh C.'
20250227:10:03:30:473322 gprebalance:cdw:gpadmin-[INFO]:-Querying gprebalance schema for current state
20250227:10:03:30:473322 gprebalance:cdw:gpadmin-[INFO]:-Validation of rebalance possibility...
You haven't specified desirable mirroring strategy. Spread mirroring places
a given hosts mirrored segments each on a separate host. You must be
using more hosts than the number of segments per host for spread mirroring.
Grouped mirroring places all of a given hosts segments on a single
mirrored host. You must be using at least 2 hosts for grouped strategy.
What type of mirroring strategy would you like?
spread|grouped (default=grouped):
>
20250227:10:03:32:473322 gprebalance:cdw:gpadmin-[INFO]:-Validation passed. Preparing rebalance...
20250227:10:03:32:473322 gprebalance:cdw:gpadmin-[INFO]:-Planning rebalance...
20250227:10:03:32:473322 gprebalance:cdw:gpadmin-[INFO]:-Creating expansion schema
20250227:10:03:36:473322 gprebalance:cdw:gpadmin-[INFO]:-About to run gprecoverseg for move dbid=15
20250227:10:05:01:473322 gprebalance:cdw:gpadmin-[INFO]:-About to run gprecoverseg for move dbid=16
20250227:10:06:06:473322 gprebalance:cdw:gpadmin-[INFO]:-Executing role swaps for 2 segments
20250227:10:06:16:473322 gprebalance:cdw:gpadmin-[INFO]:-About to run gprecoverseg for move dbid=18
20250227:10:07:20:473322 gprebalance:cdw:gpadmin-[INFO]:-About to run gprecoverseg for move dbid=19
20250227:10:08:26:473322 gprebalance:cdw:gpadmin-[INFO]:-Dropping rebalance schema
20250227:10:08:26:473322 gprebalance:cdw:gpadmin-[INFO]:-Shutting down gprebalance...Is this expected behavior? |
Member
Author
Now the parallel execution is fixed, but all movements are done via gprecoverseg, which launches pg_basebackup, so at the moment I am running out of ideas how to speed up things. It's really likely that rebalance on loaded cluster will take hours |
Member
or days/weeks/months |
red1452
reviewed
Feb 27, 2025
|
|
||
| EXECNAME = os.path.split(__file__)[-1] | ||
| MAX_BATCH_SIZE = 128 | ||
| MAX_PARALLEL_WORKERS = 96 |
There was a problem hiding this comment.
Why did you move it, instead of changing at the first place?
Member
Author
There was a problem hiding this comment.
For mvp i suggest to pay attention to general and intrinsic aspects of rebalance feature rather than considering the constants definition. I can make such cosmetics improvements later, after there will be no crucial questions left.
| parser.add_option('-f', '--target-hosts', metavar='<hosts_file>', dest='filename', | ||
| help='yaml containing target hosts configuration') | ||
| parser.add_option('-p', '--show-plan', dest='show_plan', action='store_true', default=False, | ||
| help='show rebalance plan') |
Member
Author
There was a problem hiding this comment.
It's the restult of conflicts resolution during the preparation of current brunch for PR. A we can see, the -p option is still present in the code
| parser.add_option('-e', '--end', type='datetime', metavar='datetime', | ||
| help="ending date and time in the format 'YYYY-MM-DD hh:mm:ss'.") | ||
| parser.add_option('-n', '--parallel', type="int", default=1, metavar="<parallel_processes>", | ||
| parser.add_option('-n', '--parallel', type="int", default=4, metavar="<parallel_processes>", |
Member
Author
There was a problem hiding this comment.
it seems better to have default parallel degree different from 1
| return balancer.getPlan(balancer.balance()) | ||
|
|
||
| def save_plan(self, plan: Plan): | ||
| # picke the plan in conf directory |
This comment was marked as outdated.
This comment was marked as outdated.
|
Is it Ok that after Full log: |
Member
Author
Didn't reproduce. What is in gprecoverseg log files? |
Member
Author
Fixed |
Member
How do you run rebalance? In docker? In one container or in several? I have provided a working config for running in one container above. |
Member
Member
Author
Yep, sorry. I 've misinterpreted |
I guess it is some issue with my local setup. Maybe some resource related issue. I tested with 4 containers (1 for coordinator and 3 for segments). When I tried to do the same on other machine (in the cloud), I didn't reproduce the problem. So'll keep testing with the cloud machine. |
|
When I tried to rebalance a cluster with following configuration: I got an error: Full configuration file used for cluster initFull log of gprebalance |
Member
Author
fixed |
My expectation as the user was that the tool would change the original configuration: into smth like: But I got:
|
whitehawk
reviewed
Mar 4, 2025
This comment was marked as resolved.
This comment was marked as resolved.
This patch implements the --clean option of the gprebalance utility. It cleans the status-file, the gprebalance schema in the database, and completely removes the rebalance directory from the coordinator. Ticket: ADBDEV-6856
This comment was marked as resolved.
This comment was marked as resolved.
|
Is it Ok that after Full log: |
Member
First you need to make a cleanup. |
It exits without suggesting it. It is not user friendly I think. |
Member
Author
You need to create the directory if it does not exist, or provide a valid path.
I'll add that |
In my case log: |
|
When only mirrors are not balanced, I suppose that it should take into account mirrors as well. Config file: |
Member
Author
Discussed this with architect. He agreed that's an oversight, however he allowed to neglect this, and leave current variant (cluster balance = primary balance) in MVP.
Fixed
For now it's out of scope. I'll ask the architect to confirm this in the parent ticket |
Hm, still quitely shutting down: logNot a big issue though for the MVP... |
|
If one of the primary segments is down, |
|
According to functional requirements, item#3, |
Member
typo? |
|
If |
|
I suppose that But that could be left for the final implementation, not for MVP, I guess... |
Member
Author
|
the comments in parent ticket for mvp contain the confirmatioin that current version is enough for MVP. Thanks for review and useful observations, those ones will be included in the final SRS |
|
Ok, approving this PR with the assumption that all not covered issues will be handled in the final implementation. |
whitehawk
approved these changes
Mar 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The proposed code contains main framework for rebalance execution.
Some options are not implemented fully and are expected to be finished in next
tasks.
The code describes the following segment movement approach. Firstly, we creating
a movements plan: simple steps telling which segment to which host to move.
Steps in plan can be different:
For each type of movement we clarify the target dirs and ports at target hosts,
able to contain the size of moved segment. To do that the DiskFree and DiskUsage
commands are used.
The movements, in its turn, are composite and imply extra actions including
segment switching.
firstly moved via gprecoverseg to primary's target host. Then the roles are
switched. Then ex-primary (new mirror) is moved to mirror's target host.
primary dir in its own host. Switch. Ex-primary is moved to mirror dir in its
own host.
The status management is written in general and may contain errors.
The cleanup in finally part of try in gprebalance file is left for test purposes
How to test until behave tests are not written: just create a test multi host
cluster (either in docker or in cloud) and generate imbalanced configurations
via gpinitsystem -I conffile