feat(kafka): refactor kafka topic mgmt and move setup to system-update #14564
Conversation
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2162:info:5:91: read without -r will mangle backslashes [shellcheck]
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2162:info:7:13: read without -r will mangle backslashes [shellcheck]
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:8:10: Double quote to prevent globbing and word splitting [shellcheck]
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
|
||
| @Autowired private OperationContext opContext; | ||
|
|
||
| @Order(1) // This ensures it runs before BuildIndices (@Order(1)) |
There was a problem hiding this comment.
Technically the BuildIndices and Kafka setup can run concurrently. Fine with the strict ordering for now.
There was a problem hiding this comment.
Yes, but since we run them sequentially, have the same ID just means one of them is ahead. Will address this when introducing a DAG to define upgrade step dependencies to get true parallelism.
| private final KafkaConfiguration _kafkaConfiguration; | ||
| private final KafkaProperties _kafkaProperties; | ||
|
|
||
| private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); |
There was a problem hiding this comment.
Prefer the object mapper in opContext
There was a problem hiding this comment.
ObjectMapper in opContext was null. Will revisit this.
datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/KafkaSetup.java
Outdated
Show resolved
Hide resolved
doc updates
b667acb to
711cd08
Compare
Removes kafka-setup and moves the topic setup into system-update.
Topics are listed in the application.yaml config and the KafkaConfiguration now enumerates the list of topics in order to create them. This enables any new topic to be easily created.
The KafkaConfiguration also supports topicDefaults that are merged with per-topic configs when creating the topics.
Includes Doc updates on additional configuration options