Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 39676a4

Browse files
fmbenhassinemminella
fmbenhassine
authored and
mminella
committed
Add builders to simplify remote partitioning setup
Resolves BATCH-2687 * use hard coded constants in docs * change visibility of method `beanFactory` to public in builders
1 parent 9f2b752 commit 39676a4

File tree

16 files changed

+1351
-185
lines changed

16 files changed

+1351
-185
lines changed

‎spring-batch-docs/asciidoc/images/remote-partitioning-aggregation-config.png‎

362 KB
Loading

‎spring-batch-docs/asciidoc/images/remote-partitioning-polling-config.png‎

377 KB
Loading

‎spring-batch-docs/asciidoc/spring-batch-integration.adoc‎

Lines changed: 90 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -945,7 +945,7 @@ two beans that can be autowired in the application context:
945945
* `RemoteChunkingMasterStepBuilderFactory`: used to configure the master step
946946
* `RemoteChunkingWorkerBuilder`: used to configure the remote worker integration flow
947947

948-
These APIs will take care of configuring a number of components as described in the following diagram:
948+
These APIs take care of configuring a number of components as described in the following diagram:
949949

950950
.Remote Chunking Configuration
951951
image::{batch-asciidoc}images/remote-chunking-config.png[Remote Chunking Configuration, scaledwidth="80%"]
@@ -1261,3 +1261,92 @@ You must also ensure that the partition `handler` attribute maps to the `partiti
12611261

12621262
You can find a complete example of a remote partitioning job
12631263
link:$$https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples#remote-partitioning-sample$$[here].
1264+
1265+
The `@EnableBatchIntegration` annotation that can be used to simplify a remote
1266+
partitioning setup. This annotation provides two beans useful for remote partitioning:
1267+
1268+
* `RemotePartitioningMasterStepBuilderFactory`: used to configure the master step
1269+
* `RemotePartitioningWorkerStepBuilderFactory`: used to configure the worker step
1270+
1271+
These APIs take care of configuring a number of components as described in the following diagram:
1272+
1273+
.Remote Partitioning Configuration (with job repository polling)
1274+
image::{batch-asciidoc}images/remote-partitioning-polling-config.png[Remote Partitioning Configuration (with job repository polling), scaledwidth="80%"]
1275+
1276+
.Remote Partitioning Configuration (with replies aggregation)
1277+
image::{batch-asciidoc}images/remote-partitioning-aggregation-config.png[Remote Partitioning Configuration (with replies aggregation), scaledwidth="80%"]
1278+
1279+
On the master side, the `RemotePartitioningMasterStepBuilderFactory` allows you to
1280+
configure a master step by declaring:
1281+
1282+
* the `Partitioner` used to partition data
1283+
* the output channel ("Outgoing requests") to send requests to workers
1284+
* the input channel ("Incoming replies") to receive replies from workers (when configuring replies aggregation)
1285+
* the poll interval and timeout parameters (when configuring job repository polling)
1286+
1287+
The `MessageChannelPartitionHandler` and the `MessagingTemplate` are not needed to be explicitly configured
1288+
(Those can still be explicitly configured if required).
1289+
1290+
On the worker side, the `RemotePartitioningWorkerStepBuilderFactory` allows you to configure a worker to:
1291+
1292+
* listen to requests sent by the master on the input channel ("Incoming requests")
1293+
* call the `handle` method of `StepExecutionRequestHandler` for each request
1294+
* send replies on the output channel ("Outgoing replies") to the master
1295+
1296+
There is no need to explicitly configure the `StepExecutionRequestHandler` (which can be explicitly configured if required).
1297+
1298+
The following example shows how to use these APIs:
1299+
1300+
[source, java]
1301+
----
1302+
@Configuration
1303+
@EnableBatchProcessing
1304+
@EnableBatchIntegration
1305+
public class RemotePartitioningJobConfiguration {
1306+
1307+
@Configuration
1308+
public static class MasterConfiguration {
1309+
1310+
@Autowired
1311+
private RemotePartitioningMasterStepBuilderFactory masterStepBuilderFactory;
1312+
1313+
@Bean
1314+
public Step masterStep() {
1315+
return this.masterStepBuilderFactory
1316+
.get("masterStep")
1317+
.partitioner("workerStep", partitioner())
1318+
.gridSize(10)
1319+
.outputChannel(outgoingRequestsToWorkers())
1320+
.inputChannel(incomingRepliesFromWorkers())
1321+
.build();
1322+
}
1323+
1324+
// Middleware beans setup omitted
1325+
1326+
}
1327+
1328+
@Configuration
1329+
public static class WorkerConfiguration {
1330+
1331+
@Autowired
1332+
private RemotePartitioningWorkerStepBuilderFactory workerStepBuilderFactory;
1333+
1334+
@Bean
1335+
public Step workerStep() {
1336+
return this.workerStepBuilderFactory
1337+
.get("workerStep")
1338+
.inputChannel(incomingRequestsFromMaster())
1339+
.outputChannel(outgoingRepliesToMaster())
1340+
.chunk(100)
1341+
.reader(itemReader())
1342+
.processor(itemProcessor())
1343+
.writer(itemWriter())
1344+
.build();
1345+
}
1346+
1347+
// Middleware beans setup omitted
1348+
1349+
}
1350+
1351+
}
1352+
----

‎spring-batch-docs/asciidoc/whatsnew.adoc‎

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
The Spring Batch 4.1 release adds the following features:
1010

1111
* A new `@SpringBatchTest` annotation to simplify testing batch components
12-
* A new `@EnableBatchIntegration` annotation to simplify remote chunking configuration
12+
* A new `@EnableBatchIntegration` annotation to simplify remote chunking and partitioning configuration
1313
* A new `JsonItemReader` and `JsonFileItemWriter` to support the JSON format
1414
* Add support for validating items with the Bean Validation API
1515

@@ -125,6 +125,55 @@ that uses these new APIs in the
125125
link:$$https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples#remote-chunking-sample$$[samples module]
126126
as well as more details in the <<spring-batch-integration.adoc#remote-chunking,Spring Batch Integration>> chapter.
127127

128+
Just like the remote chunking configuration simplification, this version also
129+
introduces new APIs to simplify a remote partitioning setup:
130+
`RemotePartitioningMasterStepBuilder` and `RemotePartitioningWorkerStepBuilder`.
131+
Those can be autowired in your configuration class if the
132+
`@EnableBatchIntegration` is present as shown in the following example:
133+
134+
[source, java]
135+
----
136+
@Configuration
137+
@EnableBatchProcessing
138+
@EnableBatchIntegration
139+
public class RemotePartitioningAppConfig {
140+
141+
@Autowired
142+
private RemotePartitioningMasterStepBuilderFactory masterStepBuilderFactory;
143+
144+
@Autowired
145+
private RemotePartitioningWorkerStepBuilderFactory workerStepBuilderFactory;
146+
147+
@Bean
148+
public Step masterStep() {
149+
return this.masterStepBuilderFactory
150+
.get("masterStep")
151+
.partitioner("workerStep", partitioner())
152+
.gridSize(10)
153+
.outputChannel(outgoingRequestsToWorkers())
154+
.inputChannel(incomingRepliesFromWorkers())
155+
.build();
156+
}
157+
158+
@Bean
159+
public Step workerStep() {
160+
return this.workerStepBuilderFactory
161+
.get("workerStep")
162+
.inputChannel(incomingRequestsFromMaster())
163+
.outputChannel(outgoingRepliesToMaster())
164+
.chunk(100)
165+
.reader(itemReader())
166+
.processor(itemProcessor())
167+
.writer(itemWriter())
168+
.build();
169+
}
170+
171+
// Middleware beans setup omitted
172+
}
173+
----
174+
175+
You can find more details about these new APIs in the <<spring-batch-integration.adoc#remote-partitioning,Spring Batch Integration>> chapter.
176+
128177
[[whatsNewJson]]
129178
=== JSON support
130179

‎spring-batch-integration/src/main/java/org/springframework/batch/integration/config/annotation/BatchIntegrationConfiguration.java‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,12 @@
1515
*/
1616
package org.springframework.batch.integration.config.annotation;
1717

18+
import org.springframework.batch.core.explore.JobExplorer;
1819
import org.springframework.batch.core.repository.JobRepository;
1920
import org.springframework.batch.integration.chunk.RemoteChunkingMasterStepBuilderFactory;
2021
import org.springframework.batch.integration.chunk.RemoteChunkingWorkerBuilder;
22+
import org.springframework.batch.integration.partition.RemotePartitioningMasterStepBuilderFactory;
23+
import org.springframework.batch.integration.partition.RemotePartitioningWorkerStepBuilderFactory;
2124
import org.springframework.beans.factory.annotation.Autowired;
2225
import org.springframework.context.annotation.Bean;
2326
import org.springframework.context.annotation.Configuration;
@@ -32,16 +35,20 @@
3235
@Configuration
3336
public class BatchIntegrationConfiguration {
3437

38+
private JobExplorer jobExplorer;
39+
3540
private JobRepository jobRepository;
3641

3742
private PlatformTransactionManager transactionManager;
3843

3944
@Autowired
4045
public BatchIntegrationConfiguration(
4146
JobRepository jobRepository,
47+
JobExplorer jobExplorer,
4248
PlatformTransactionManager transactionManager) {
4349

4450
this.jobRepository = jobRepository;
51+
this.jobExplorer = jobExplorer;
4552
this.transactionManager = transactionManager;
4653
}
4754

@@ -56,4 +63,16 @@ public <I,O> RemoteChunkingWorkerBuilder<I, O> remoteChunkingWorkerBuilder() {
5663
return new RemoteChunkingWorkerBuilder<>();
5764
}
5865

66+
@Bean
67+
public RemotePartitioningMasterStepBuilderFactory remotePartitioningMasterStepBuilderFactory() {
68+
return new RemotePartitioningMasterStepBuilderFactory(this.jobRepository,
69+
this.jobExplorer, this.transactionManager);
70+
}
71+
72+
@Bean
73+
public RemotePartitioningWorkerStepBuilderFactory remotePartitioningWorkerStepBuilderFactory() {
74+
return new RemotePartitioningWorkerStepBuilderFactory(this.jobRepository,
75+
this.jobExplorer, this.transactionManager);
76+
}
77+
5978
}

‎spring-batch-integration/src/main/java/org/springframework/batch/integration/config/annotation/EnableBatchIntegration.java‎

Lines changed: 59 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,24 +23,33 @@
2323

2424
import org.springframework.batch.integration.chunk.RemoteChunkingWorkerBuilder;
2525
import org.springframework.batch.integration.chunk.RemoteChunkingMasterStepBuilderFactory;
26+
import org.springframework.batch.integration.partition.RemotePartitioningMasterStepBuilderFactory;
27+
import org.springframework.batch.integration.partition.RemotePartitioningWorkerStepBuilderFactory;
2628
import org.springframework.context.annotation.Import;
29+
import org.springframework.integration.config.EnableIntegration;
2730

2831
/**
2932
* Enable Spring Batch Integration features and provide a base configuration for
30-
* setting up remote chunking infrastructure beans.
33+
* setting up remote chunking or partitioning infrastructure beans.
3134
*
3235
* By adding this annotation on a {@link org.springframework.context.annotation.Configuration}
3336
* class, it will be possible to autowire the following beans:
3437
*
3538
* <ul>
3639
* <li>{@link RemoteChunkingMasterStepBuilderFactory}:
37-
* used to create a master step by automatically setting up the job repository
38-
* and transaction manager.</li>
40+
* used to create a master step of a remote chunking setup by automatically
41+
* setting the job repository and transaction manager.</li>
3942
* <li>{@link RemoteChunkingWorkerBuilder}: used to create the integration
40-
* flow on the worker side.</li>
43+
* flow on the worker side of a remote chunking setup.</li>
44+
* <li>{@link RemotePartitioningMasterStepBuilderFactory}: used to create
45+
* a master step of a remote partitioning setup by automatically setting
46+
* the job repository, job explorer, bean factory and transaction manager.</li>
47+
* <li>{@link RemotePartitioningWorkerStepBuilderFactory}: used to create
48+
* a worker step of a remote partitioning setup by automatically setting
49+
* the job repository, job explorer, bean factory and transaction manager.</li>
4150
* </ul>
4251
*
43-
* For example:
52+
* For remote chunking, an example of a configuration class would be:
4453
*
4554
* <pre class="code">
4655
* &#064;Configuration
@@ -75,15 +84,60 @@
7584
* .build();
7685
* }
7786
*
87+
* // Middleware beans omitted
88+
*
7889
* }
7990
* </pre>
8091
*
92+
* For remote partitioning, an example of a configuration class would be:
93+
*
94+
* <pre class="code">
95+
* &#064;Configuration
96+
* &#064;EnableBatchIntegration
97+
* &#064;EnableBatchProcessing
98+
* public class RemotePartitioningAppConfig {
99+
*
100+
* &#064;Autowired
101+
* private RemotePartitioningMasterStepBuilderFactory masterStepBuilderFactory;
102+
*
103+
* &#064;Autowired
104+
* private RemotePartitioningWorkerStepBuilderFactory workerStepBuilderFactory;
105+
*
106+
* &#064;Bean
107+
* public Step masterStep() {
108+
* return this.masterStepBuilderFactory
109+
* .get("masterStep")
110+
* .partitioner("workerStep", partitioner())
111+
* .gridSize(10)
112+
* .outputChannel(outgoingRequestsToWorkers())
113+
* .inputChannel(incomingRepliesFromWorkers())
114+
* .build();
115+
* }
116+
*
117+
* &#064;Bean
118+
* public Step workerStep() {
119+
* return this.workerStepBuilderFactory
120+
* .get("workerStep")
121+
* .inputChannel(incomingRequestsFromMaster())
122+
* .outputChannel(outgoingRepliesToMaster())
123+
* .chunk(100)
124+
* .reader(itemReader())
125+
* .processor(itemProcessor())
126+
* .writer(itemWriter())
127+
* .build();
128+
* }
129+
*
130+
* // Middleware beans omitted
131+
*
132+
* }
133+
* </pre>
81134
* @since 4.1
82135
* @author Mahmoud Ben Hassine
83136
*/
84137
@Target(ElementType.TYPE)
85138
@Retention(RetentionPolicy.RUNTIME)
86139
@Documented
140+
@EnableIntegration
87141
@Import(BatchIntegrationConfiguration.class)
88142
public @interface EnableBatchIntegration {
89143
}

0 commit comments

Comments
 (0)