[FLINK-26803][checkpoint] Merging channel state files #20151

1996fanrui · 2022-07-04T09:25:43Z

What is the purpose of the change

Merging channel state files to reduce the pressure on DFS.

Brief change log

Introduce the ChannelStateWriteRequestExecutorFactory to control how many subtasks to share a ChannelStateWriteRequestExecutor.

Verifying this change

This change added tests and can be verified as follows:

Added the TaskExecutorChannelStateExecutorFactoryManagerTest.
Added the ChannelStateWriteRequestExecutorFactoryTest.
Added some tests in the ChannelStateCheckpointWriterTest
ChannelStateWriteRequestExecutorImplTest#testSkipUnreadyDataFuture

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs

flinkbot · 2022-07-04T09:31:38Z

CI report:

0b46716 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

1996fanrui · 2022-10-18T07:26:51Z

Hi @pnowojski , please help take a look in your free time, thanks a lot.

pnowojski

Thanks for the contribution, and again sorry for the delay.

I've started reviewing the code trying to understand the big picture (I've left a couple of a minor side comments in the process), and began to wonder what's the actual threading model here?

Each subtask has its own ChannelStateWriter. Created once at the start of the StreamTask.
ChannelStateWriter is used to start, write input and output data asynchronously. Currently both, from the task thread and netty thread (that's unfortunate, it's more of a technological debt - all writes should be happening from the task thread)
ChannelStateWriter delegates all of the writes, to an instance of ChannelStateWriteRequestExecutor
Currently a new ChannelStateWriteRequestExecutorImpl is created for each ChannelStateWriter.
ChannelStateWriteRequestExecutorImpl has it's own thread doing the actual writes.
ChannelStateWriteRequestExecutorImpl is doing the writes indirectly through the ChannelStateWriteRequestDispatcherImpl. This dispatcher is wholly owned by the request executor.
ChannelStateWriteRequestDispatcherImpl actually owns the ChannelStateCheckpointWriters, that finally encapsulate the actual data stream CheckpointStateOutputStream.

(side note this stack is probably a bit over-engineered)

Now underneath all of that @1996fanrui, you are adding ChannelStateCheckpointStreamManager that's binding a selected number of ChannelStateCheckpointWriter to use a single instance of CheckpointStateOutputStream.

Does this design make sense? Wouldn't it be simpler if instead we would limit the number of ChannelStateWriteRequestExecutorImpl and allow various (sub)tasks to share the same ChannelStateWriteRequestExecutorImpl instance?

In your current proposal, there is already a pre-existing thread synchronisation between ChannelStateWriterImpl and ChannelStateWriteRequestExecutorImpl, and you are adding another synchronisation on the lower layer between different instances of ChannelStateCheckpointWriter. With a shared ChannelStateWriteRequestExecutorImpl there would be no need for the second level of the synchornisation. Making the code more efficient (due to fewer synchronisation and also keeping the one writing thread per file) and also simpler with fewer opportunities for race conditions/deadlocks.

WDYT? Am I missing something?

pnowojski · 2022-10-20T11:32:33Z

.../src/main/java/org/apache/flink/streaming/api/environment/ExecutionCheckpointingOptions.java

+
+    public static final ConfigOption<Integer> CHANNEL_STATE_NUMBER_OF_TASKS_SHARE_FILE =
+            ConfigOptions.key("execution.checkpointing.channel-state.number-of-tasks-share-file")
+                    .intType()
+                    .defaultValue(5)
+                    .withDescription(
+                            "Defines the maximum number of tasks that share the same channel state file. "
+                                    + "It can reduce the number of small files when enable unaligned checkpoint. "
+                                    + "Each task will create a new channel state file when this is configured to 1.");


task -> subtask

I don't see that this has been actually fixed? Note, most likely also the config option should be renamed?

Sorry, I just changed the description, I will change the config key later.

Updated.

BTW, I squashed commits this time due to rebased the master branch and fixed some conflicts. I will use the separate commits when addressing comments in the future.

...time/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequest.java

1996fanrui · 2022-10-20T15:07:00Z

With a shared ChannelStateWriteRequestExecutorImpl there would be no need for the second level of the synchornisation. Making the code more efficient (due to fewer synchronisation and also keeping the one writing thread per file) and also simpler with fewer opportunities for race conditions/deadlocks.

WDYT? Am I missing something?

Hi @pnowojski , thanks for your hard review.

Currently, there is one ChannelStateWriterImpl per subtask and one ChannelStateWriteRequestExecutorImpl per ChannelStateWriterImpl. That is: each subtask has one thread to write the channel state file.

As I understand, you mean that multiple ChannelStateWriterImpl share the same ChannelStateWriteRequestExecutorImpl. When channel-state.number-of-tasks-share-file=5, each thread is responsible for writing the channel state file for 5 subtasks, right? Since the file is written in a single thread, there is no need to consider thread safety issues.

Your proposal should be clearer. I will try to refactor the code according to your proposal. Thanks again~

pnowojski · 2022-10-21T08:41:30Z

As I understand, you mean that multiple ChannelStateWriterImpl share the same
ChannelStateWriteRequestExecutorImpl. When channel-state.number-of-tasks-share-file=5, each thread is responsible
for writing the channel state file for 5 subtasks, right? Since the file is written in a single thread, there is no need to consider thread safety issues.

Yes, that's what I had in mind.

Your proposal should be clearer. I will try to refactor the code according to your proposal. Thanks again~

Great! But please keep in mind that I haven't thought it fully through and I haven't tried to implement it myself, so if you encounter some obstacles, feel free to reach out to me before going too deep!

1996fanrui · 2022-11-28T12:49:25Z

Hi @pnowojski , I have updated the PR according to your great suggestion.

1996fanrui · 2022-11-28T15:39:20Z

...va/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestExecutorImplTest.java

-        try (ChannelStateWriteRequestExecutorImpl worker =
-                new ChannelStateWriteRequestExecutorImpl(TASK_NAME, requestProcessor, deque)) {
+        ChannelStateWriteRequestExecutorImpl worker =
+                new ChannelStateWriteRequestExecutorImpl(requestProcessor, deque, JOB_ID, 5);


Hi @pnowojski , I'm not sure should these channel state classes be moved to the flink-streaming-java module?

All checkpoint-related configurations are defined in the flink-streaming-java module, and these channel state classes are defined in the flink-runtime module.

These classes didn't read any configuration before, so they are well. However, It's hard to read some configurations during unit test due to flink-runtime doesn't depend on the flink-streaming-java module.

pnowojski

Thanks for your patience @1996fanrui , I've looked through most of the production code and I've left some comments. But I think I haven't spotted anything major.

.../src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java

...ime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStatePendingResult.java

.../src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java

...time/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequest.java

...java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestDispatcherImpl.java

...ava/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestExecutorFactory.java

...n/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestExecutorImpl.java

1996fanrui · 2022-12-17T13:30:38Z

Thanks for your patience @1996fanrui , I've looked through most of the production code and I've left some comments. But I think I haven't spotted anything major.

Hi @pnowojski , thanks a lot for your review, I have updated.

.../src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java

pnowojski

Thanks for the updates @1996fanrui. I think I have only a one or two last minor comments (this one #20151 (comment) and the while (true) question). Could you squash the fixup commits before me doing the last pass and merging?

...n/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestExecutorImpl.java

pnowojski · 2023-01-20T08:55:18Z

👍 Thanks, it looks good. Once you squash fixup commits I will do the last pass and hopefully merge.

… 2 times

pnowojski

LGTM % one comment

.../src/main/java/org/apache/flink/streaming/api/environment/ExecutionCheckpointingOptions.java

fixup! Address comments fixup! fixup! Address comments 1. Add some job docs 2. Using the lock to refactor the ChannelStateWriteRequestExecutorImpl fixup! fixup! fixup! Address comments

pnowojski

LGTM

Thanks a lot for your efforts @1996fanrui and your patience :)

1996fanrui force-pushed the 26803/merge-channel-file branch from fb29b1e to 8ac3bf9 Compare July 4, 2022 09:29

flinkbot added component=Runtime/Network component=Runtime/Checkpointing labels Jul 4, 2022

1996fanrui force-pushed the 26803/merge-channel-file branch 3 times, most recently from dfa1eec to 396ab99 Compare July 13, 2022 13:47

1996fanrui force-pushed the 26803/merge-channel-file branch 3 times, most recently from 9e347ea to f560e5f Compare October 14, 2022 07:36

pnowojski self-assigned this Oct 20, 2022

pnowojski reviewed Oct 20, 2022

View reviewed changes

1996fanrui mentioned this pull request Oct 23, 2022

[FLINK-29730][checkpoint] Do not support concurrent unaligned checkpoints in the ChannelStateWriteRequestDispatcherImpl #21131

Merged

1996fanrui force-pushed the 26803/merge-channel-file branch from f560e5f to e00eb2a Compare November 28, 2022 12:48

1996fanrui commented Nov 28, 2022

View reviewed changes

1996fanrui force-pushed the 26803/merge-channel-file branch from e00eb2a to 33f3d53 Compare December 14, 2022 14:34

pnowojski reviewed Dec 14, 2022

View reviewed changes

1996fanrui force-pushed the 26803/merge-channel-file branch from 8d11f65 to 2c33368 Compare December 18, 2022 16:10

pnowojski reviewed Dec 19, 2022

View reviewed changes

.../src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateCheckpointWriter.java Outdated Show resolved Hide resolved

1996fanrui force-pushed the 26803/merge-channel-file branch from 7a673e9 to ff09320 Compare December 20, 2022 12:52

1996fanrui force-pushed the 26803/merge-channel-file branch 2 times, most recently from 325b708 to 19d104f Compare January 5, 2023 03:20

pnowojski reviewed Jan 19, 2023

View reviewed changes

...n/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriteRequestExecutorImpl.java Outdated Show resolved Hide resolved

[hotfix] Remove useless exception in SubtaskCheckpointCoordinatorImpl

584f0ec

[FLINK-26803][checkpoint] Fix the bug of ChannelStateWriterImpl close…

0dfaf91

… 2 times

1996fanrui force-pushed the 26803/merge-channel-file branch from fb1cc52 to cc2613c Compare January 20, 2023 09:23

pnowojski approved these changes Jan 20, 2023

View reviewed changes

.../src/main/java/org/apache/flink/streaming/api/environment/ExecutionCheckpointingOptions.java Outdated Show resolved Hide resolved

[FLINK-26803][checkpoint] Merge the channel state files

0b46716

fixup! Address comments fixup! fixup! Address comments 1. Add some job docs 2. Using the lock to refactor the ChannelStateWriteRequestExecutorImpl fixup! fixup! fixup! Address comments

1996fanrui force-pushed the 26803/merge-channel-file branch from cc2613c to 0b46716 Compare January 20, 2023 17:02

pnowojski approved these changes Jan 23, 2023

View reviewed changes

pnowojski merged commit 8be94e6 into apache:master Jan 23, 2023

1996fanrui deleted the 26803/merge-channel-file branch January 25, 2023 08:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-26803][checkpoint] Merging channel state files #20151

[FLINK-26803][checkpoint] Merging channel state files #20151

1996fanrui commented Jul 4, 2022 •

edited

Loading

flinkbot commented Jul 4, 2022 •

edited

Loading

1996fanrui commented Oct 18, 2022

pnowojski left a comment •

edited

Loading

pnowojski Oct 20, 2022

pnowojski Dec 14, 2022

1996fanrui Dec 14, 2022

1996fanrui Dec 14, 2022

1996fanrui commented Oct 20, 2022

pnowojski commented Oct 21, 2022

1996fanrui commented Nov 28, 2022

1996fanrui Nov 28, 2022

pnowojski left a comment

1996fanrui commented Dec 17, 2022

pnowojski left a comment •

edited

Loading

pnowojski commented Jan 20, 2023

pnowojski left a comment

pnowojski left a comment

[FLINK-26803][checkpoint] Merging channel state files #20151

[FLINK-26803][checkpoint] Merging channel state files #20151

Conversation

1996fanrui commented Jul 4, 2022 • edited Loading

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Jul 4, 2022 • edited Loading

CI report:

1996fanrui commented Oct 18, 2022

pnowojski left a comment • edited Loading

Choose a reason for hiding this comment

pnowojski Oct 20, 2022

Choose a reason for hiding this comment

pnowojski Dec 14, 2022

Choose a reason for hiding this comment

1996fanrui Dec 14, 2022

Choose a reason for hiding this comment

1996fanrui Dec 14, 2022

Choose a reason for hiding this comment

1996fanrui commented Oct 20, 2022

pnowojski commented Oct 21, 2022

1996fanrui commented Nov 28, 2022

1996fanrui Nov 28, 2022

Choose a reason for hiding this comment

pnowojski left a comment

Choose a reason for hiding this comment

1996fanrui commented Dec 17, 2022

pnowojski left a comment • edited Loading

Choose a reason for hiding this comment

pnowojski commented Jan 20, 2023

pnowojski left a comment

Choose a reason for hiding this comment

pnowojski left a comment

Choose a reason for hiding this comment

1996fanrui commented Jul 4, 2022 •

edited

Loading

flinkbot commented Jul 4, 2022 •

edited

Loading

pnowojski left a comment •

edited

Loading

pnowojski left a comment •

edited

Loading