BarunW/Flink-HTTPSourceConnector
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This Flink-HTTPSourceConnector is flink source wrapped around java http client
for rest API endpoints
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
JAVA : VERSION 11
FLINK : 1.20.1
HOW TO USE:
Put the httpSource folder/dir to your project
```
import httpSource.HTTPSource;
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
env.setParallelism(1);
env.enableCheckpointing(1000 * 10) // 10 sec;
HTTPSource<YourPojo> source = HTTPSource.<YourPojo>builder()
.addEndpoint("https://whatever_your_endpoint/")
.setDeserializer(new YourSchemaDeserializer())
.addHeader("", "") // Some headers cannot be set in java httpClient check out for that
.setHTTPMethod("POST") // GET, PUT, PATCH
.setHTTPRequestBodyPublisher("") // Takes String as input for HttpRequest.BodyPublishers.ofString()
.setBoundedness(Boundedness.CONTINUOUS_UNBOUNDED)
.setPollingInterval(5000L * 6) // 30 second
.build();
DataStream<VehicleUpdateDataSchema> stream = env.fromSource(
source,
WatermarkStrategy.noWatermarks(),
"source"
);
```