Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[FLINK-32081][checkpoint] Compatibility between file-merging on and off across job runs #24873

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[FLINK-32081][checkpoint] Compatibility between file-merging on and off across job runs #24873

wants to merge 2 commits into from

Conversation

ljz2051
Copy link
Contributor

@ljz2051 ljz2051 commented May 31, 2024

What is the purpose of the change

This pull request resolves the compatibility between file-merging on and off across job runs.

Brief change log

  • Don't reuse SegmentFileHandles for CheckpointStreamFactory when file-merging is switched from enabling to disabling.
  • Add a compatibility IT case for fileMerging.

Verifying this change

This change added tests and can be verified by SnapshotFileMergingCompatibilityITCase.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes (checkpoint)
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no

@Zakelly Zakelly self-requested a review May 31, 2024 09:24
@flinkbot
Copy link
Collaborator

flinkbot commented May 31, 2024 •
edited
Loading

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Zakelly
Zakelly requested changes May 31, 2024
View reviewed changes
Copy link
Contributor

@Zakelly Zakelly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, I left some comments PTAL

@ljz2051 ljz2051 requested a review from Zakelly May 31, 2024 10:24
Zakelly
Zakelly approved these changes Jun 11, 2024
View reviewed changes
Copy link
Contributor

@Zakelly Zakelly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! LGTM

@Zakelly Zakelly closed this in f33808c Jun 11, 2024
superdiaodiao pushed a commit to superdiaodiao/flink that referenced this pull request Jul 4, 2024
@ljz2051 @superdiaodiao
[FLINK-32081][test] Add Compatibility IT case for fileMerging
a54aae8 
This fixes apache#24873
snuyanzin pushed a commit to snuyanzin/flink that referenced this pull request Jul 21, 2024
@ljz2051 @snuyanzin
[FLINK-32081][test] Add Compatibility IT case for fileMerging
14113f5 
This fixes apache#24873
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Zakelly Zakelly Zakelly approved these changes

Assignees
No one assigned
Labels
component=Runtime/Checkpointing component=Runtime/StateBackends
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ljz2051 @flinkbot @Zakelly