Releases: karafka/karafka
Releases · karafka/karafka
v2.5.2
- [EOL] Remove Rails 7.1 support according to EOL while not blocking Rails 7.1 usage.
- [Enhancement] Retry on the KIP-848
stale_member_epocherror. - [Enhancement] Provide
Karafka::Admin.trigger_rebalanceAPI to programmatically trigger consumer group rebalances for operational purposes. - [Enhancement] Nest pause configuration under
config.pause.*namespace (config.pause.timeout,config.pause.max_timeout,config.pause.with_exponential_backoff) while maintaining backwards compatibility with the old flat API (config.pause_timeout, etc.) via delegation methods that will be removed in Karafka 2.6. - [Enhancement] Detect and track involuntary assignment loss during long-running processing that exceeds
max.poll.interval.msviaclient.events_pollevent and automatically updateKarafka::App.assignmentsto reflect reality. - [Enhancement] Extend
Karafka::Admin.read_watermark_offsetsto accept either a single topic with partition or a hash of multiple topics with partitions, using a single consumer instance for improved efficiency when querying multiple partitions. - [Enhancement] Add configurable
Karafka::ActiveJob::Deserializerto support custom serialization formats (Avro, Protobuf, etc.) for ActiveJob payloads. - [Fix] Fix ActiveJob Continuation invalid class reference bug.
v2.5.1
- [Breaking] Remove Ruby 3.1 support according to EOL.
- [Feature] Support Swarm mode on MacOS.
- [Enhancement] Support past
dispatch_attimes withjitter: 0in the OSS Karafka to support ActiveJob continuation. - [Enhancement] Use direct topic dispatches when
dispatch_atis used for past times to bypass Scheduled Messages flow. - [Enhancement] Support immediate error raising with
auto.offset.resetset toerror. - [Enhancement] Don't create not needed dirs in the non-Rails setup template.
- [Enhancement] Improve printing of TTIN to separate threads.
- [Enhancement] Support transactional ID resource in the Admin ACL management.
- [Enhancement] Move post-forceful shutdown termination wait value to a config.
- [Enhancement] Include consumer group, subscription group and other details in error logs for key error locations.
- [Enhancement] Inherit from
ActiveJob::QueueAdapters::AbstractAdapterwhen possible for ActiveJob base class. - [Enhancement] Disable Nagle algorithm by default for improved network performance.
- [Enhancement] Optimize the messages buffer array memory allocation pattern.
- [Maintenance] Add basic direct DD integration spec via DD gem karafka monitoring feature.
- [Maintenance] Add integration specs for WaterDrop connection pool usage from within consumers.
- [Refactoring] Comprehensive Admin module refactoring: Extract topic operations into Admin::Topics class and consumer group operations into Admin::ConsumerGroups class with proper inheritance hierarchy, cross-class method usage optimization, and constants moved to appropriate locations where they are actually used.
- [Refactoring] Move routing-related contracts from
Karafka::Contracts::toKarafka::Routing::Contracts::namespace and reorganize error message structure in YAML files underrouting:scope for better code organization and logical grouping. - [Refactoring] Move config-related contracts from
Karafka::Contracts::ConfigtoKarafka::Setup::Contracts::Confignamespace and reorganize error message structure in YAML files undersetup:scope for better code organization and logical grouping. - [Refactoring] Move CLI server contracts from
Karafka::Contracts::ServerCliOptionstoKarafka::Cli::Contracts::Servernamespace and reorganize error message structure in YAML files undercli:scope for improved naming consistency and logical grouping. - [Refactoring] Replace execution mode symbol-based checks with dedicated
ExecutionModeclass providing cleaner API with query methods (#swarm?,#embedded?) and state change methods (#swarm!,#embedded!) for improved type safety and code clarity. - [Refactoring] Replace connection client mode symbol-based checks with dedicated
Connection::Modeclass providing cleaner API with query methods (#subscribe?,#assign?) and state change methods (#subscribe!,#assign!) for improved code clarity. - [Fix] Improve same timestamp dispatch in scheduled messages on Ruby 3.2.
- [Fix] Fix incorrect (6 seconds vs 60 seconds) reset of connections on non-recoverable errors.
- [Fix] Introduce mutex-safe and thread-safe
#inspectwhere needed. - [Fix] Fix too loose requirement of Ruby
3.0when it was3.1via transitive dependencies. - [Fix] Fix Pro Cleaner Messages compatibility with external libraries that prepend modules to
#eachmethod (e.g., DataDog tracing). - [Fix] SG exclusion in swarm triggers a contract validation error.
- [Change] Require
waterdrop>=2.8.10to support new features. - [Change] Require
karafka-rdkafka>=0.22.0to support new features and require SSL-bug free version. - [Change] Remove no longer needed
cooperative.stickyrebalance patch. - [Change] Normalize how libs and deps are required (no functional change for the end user)
- [Change] Remove Ruby
3.1specs according to the EOL schedule.
v2.5.1.beta1
- [Feature] Support Swarm mode on MacOS.
- [Enhancement] Support past
dispatch_attimes withjitter: 0in the OSS Karafka to support ActiveJob continuation. - [Enhancement] Use direct topic dispatches when
dispatch_atis used for past times to bypass Scheduled Messages flow. - [Enhancement] Support immediate error raising with
auto.offset.resetset toerror. - [Enhancement] Don't create not needed dirs in the non-Rails setup template.
- [Enhancement] Improve printing of TTIN to separate threads.
- [Enhancement] Support transactional ID resource in the Admin ACL management.
- [Enhancement] Move post-forceful shutdown termination wait value to a config.
- [Enhancement] Include consumer group, subscription group and other details in error logs for key error locations.
- [Enhancement] Inherit from
ActiveJob::QueueAdapters::AbstractAdapterwhen possible for ActiveJob base class. - [Enhancement] Disable Nagle algorithm by default for improved network performance.
- [Maintenance] Add basic direct DD integration spec via DD gem karafka monitoring feature.
- [Refactoring] Comprehensive Admin module refactoring: Extract topic operations into Admin::Topics class and consumer group operations into Admin::ConsumerGroups class with proper inheritance hierarchy, cross-class method usage optimization, and constants moved to appropriate locations where they are actually used.
- [Refactoring] Move routing-related contracts from
Karafka::Contracts::toKarafka::Routing::Contracts::namespace and reorganize error message structure in YAML files underrouting:scope for better code organization and logical grouping. - [Refactoring] Move config-related contracts from
Karafka::Contracts::ConfigtoKarafka::Setup::Contracts::Confignamespace and reorganize error message structure in YAML files undersetup:scope for better code organization and logical grouping. - [Refactoring] Move CLI server contracts from
Karafka::Contracts::ServerCliOptionstoKarafka::Cli::Contracts::Servernamespace and reorganize error message structure in YAML files undercli:scope for improved naming consistency and logical grouping. - [Fix] Improve same timestamp dispatch in scheduled messages on Ruby 3.2.
- [Fix] Fix incorrect (6 seconds vs 60 seconds) reset of connections on non-recoverable errors.
- [Fix] Introduce mutex-safe and thread-safe
#inspectwhere needed. - [Fix] Fix too loose requirement of Ruby
3.0when it was3.1via transitive dependencies. - [Fix] Fix Pro Cleaner Messages compatibility with external libraries that prepend modules to
#eachmethod (e.g., DataDog tracing). - [Change] Require
karafka-rdkafka>=0.21.0to support new features. - [Change] Remove no longer needed
cooperative.stickyrebalance patch. - [Change] Normalize how libs and deps are required (no functional change for the end user)
- [Change] Remove Ruby
3.1specs according to the EOL schedule.
v2.5.0
- [Breaking] Change how consistency of DLQ dispatches works in Pro (
partition_keyvs. direct partition id mapping). - [Breaking] Remove the headers
source_keyfrom the Pro DLQ dispatched messages as the original key is now fully preserved. - [Breaking] Use DLQ and Piping prefix
source_instead oforiginal_to align with naming convention of Kafka Streams and Apache Flink for future usage. - [Breaking] Rename scheduled jobs topics names in their config (Pro).
- [Breaking] Change K8s listener response from
204to200and include JSON body with reasons. - [Breaking] Replace admin config
max_attemptswithmax_retries_durationand - [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
- [Enhancement] Normalize topic + partition logs format.
- [Enhancement] Support KIP-82 (header values of arrays).
- [Enhancement] Enhance errors tracker with
#countsthat contains per-error class specific counters for granular flow handling. - [Enhancement] Provide explicit
Karafka::Admin.copy_consumer_groupAPI. - [Enhancement] Return explicit value from
Karafka::Admin.copy_consumer_groupandKarafka::Admin.rename_consumer_groupAPIs. - [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
- [Enhancement] Make the error tracker for advanced DLQ strategies respond to
#topicand#partitionfor context aware dispatches. - [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
- [Enhancement] Enhance low-level
client.pauseevent with timeout value (if provided). - [Enhancement] Introduce
#marking_cursorAPI (defaults to#cursor) in the filtering API (Pro). - [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
- [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
- [Enhancement] Include info level rebalance logger listener data.
- [Enhancement] Include info level subscription start info.
- [Enhancement] Make the generic error handling in the
LoggerListenermore descriptive by logging also the error class. - [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
- [Enhancement] Change optional
#seekreset offset flag default totrueasfalseis almost never used and seek by default should move the internal consumer offset position as well. - [Enhancement] Include Swarm node ID in the swarm process tags.
- [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
- [Enhancement] Improve OSS vs. Pro specs execution isolation.
- [Enhancement] Preload
librdkafkacode prior to forking in the Swarm mode to save memory. - [Enhancement] Extract errors tracker class reference into an internal
errors_tracker_classconfig option (Pro). - [Enhancement] Support rdkafka native kafka polling customization for admin.
- [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
- [Enhancement] Set
topic.metadata.refresh.interval.msfor default producer in dev to 5s to align with consumer setup. - [Enhancement] Alias
-2and-1withlatestandearliestfor seeking. - [Enhancement] Allow for usage of
latestandearliestin theKarafka::Pro::Iterator. - [Enhancement] Failures during
topics migrate(and other subcommands) don't show what topic failed, and why it's invalid. - [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
- [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
- [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
- [Enhancement] Provide
karafka topics helpsub-help due to nesting of Declarative Topics actions. - [Enhancement] Use independent keys for different states of reporting in scheduled messages.
- [Enhancement] Enrich scheduled messages state reporter with debug data.
- [Enhancement] Introduce a new state called
stoppedto the scheduled messages. - [Enhancement] Do not overwrite the
keyin the Pro DLQ dispatched messages for routing reasons. - [Enhancement] Introduce
errors_tracker.trace_idfor distributed error details correlation with the Web UI. - [Enhancement] Improve contracts validations reporting.
- [Enhancement] Optimize topic creation and repartitioning admin operations for topics with hundreds of partitions.
- [Refactor] Introduce a
bin/verify_kafka_warningsscript to clean Kafka from temporary test-suite topics. - [Refactor] Introduce a
bin/verify_topics_namingscript to ensure proper test topics naming convention. - [Refactor] Make sure all temporary topics have a
it-prefix in their name. - [Refactor] Improve CI specs parallelization.
- [Maintenance] Lower the
Karafka::Adminpoll_timeoutto 50 ms to improve responsiveness of admin operations. - [Maintenance] Require
karafka-rdkafka>=0.19.5due to usage of#rd_kafka_global_init, KIP-82, new producer caching engine and improvements to thepartition_keyassignments. - [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
- [Maintenance] Remove Rails
7.0specs due to upcoming EOL. - [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
- [Fix] Fix a case where
unknown_topic_or_parterror could leak out of the consumer on consumer shutdown. - [Fix] Fix missing
virtual_partitions.partitioner.errorcustom error logging in theLoggerListener. - [Fix] Prevent applied system filters
#timeoutfrom potentially interacting with user filters. - [Fix] Use more sane value in
Admin#seek_consumer_groupfor long ago. - [Fix] Prevent multiplexing of 1:1 from routing.
- [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
- [Fix] Fix inconsistency in the logs where
Karafka::Serveroriginating logs would not have server id reference. - [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
- [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
- [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
- [Fix] optparse double parse loses ARGV.
- [Fix]
karafkacannot be required without Bundler. - [Fix] Scheduled Messages re-seek moves to
lateston inheritance of initial offset when0offset is compacted. - [Fix] Seek to
:latestwithouttopic_partition_position(-1) will not seek at all. - [Fix] Extremely high turn over of scheduled messages can cause them not to reach EOF/Loaded state.
- [Fix] Fix incorrectly passed
max_wait_timeto rdkafka (ms instead of seconds) causing too long wait. - [Fix] Remove aggresive requerying of the Kafka cluster on topic creation/removal/altering.
- [Change] Move to trusted-publishers and remove signing since no longer needed.
v2.5.0.rc2
- [Breaking] Change how consistency of DLQ dispatches works in Pro (
partition_keyvs. direct partition id mapping). - [Breaking] Remove the headers
source_keyfrom the Pro DLQ dispatched messages as the original key is now fully preserved. - [Breaking] Use DLQ and Piping prefix
source_instead oforiginal_to align with naming convention of Kafka Streams and Apache Flink for future usage. - [Breaking] Rename scheduled jobs topics names in their config (Pro).
- [Breaking] Change K8s listener response from
204to200and include JSON body with reasons. - [Breaking] Replace admin config
max_attemptswithmax_retries_durationand - [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
- [Enhancement] Normalize topic + partition logs format.
- [Enhancement] Support KIP-82 (header values of arrays).
- [Enhancement] Enhance errors tracker with
#countsthat contains per-error class specific counters for granular flow handling. - [Enhancement] Provide explicit
Karafka::Admin.copy_consumer_groupAPI. - [Enhancement] Return explicit value from
Karafka::Admin.copy_consumer_groupandKarafka::Admin.rename_consumer_groupAPIs. - [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
- [Enhancement] Make the error tracker for advanced DLQ strategies respond to
#topicand#partitionfor context aware dispatches. - [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
- [Enhancement] Enhance low-level
client.pauseevent with timeout value (if provided). - [Enhancement] Introduce
#marking_cursorAPI (defaults to#cursor) in the filtering API (Pro). - [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
- [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
- [Enhancement] Include info level rebalance logger listener data.
- [Enhancement] Include info level subscription start info.
- [Enhancement] Make the generic error handling in the
LoggerListenermore descriptive by logging also the error class. - [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
- [Enhancement] Change optional
#seekreset offset flag default totrueasfalseis almost never used and seek by default should move the internal consumer offset position as well. - [Enhancement] Include Swarm node ID in the swarm process tags.
- [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
- [Enhancement] Improve OSS vs. Pro specs execution isolation.
- [Enhancement] Preload
librdkafkacode prior to forking in the Swarm mode to save memory. - [Enhancement] Extract errors tracker class reference into an internal
errors_tracker_classconfig option (Pro). - [Enhancement] Support rdkafka native kafka polling customization for admin.
- [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
- [Enhancement] Set
topic.metadata.refresh.interval.msfor default producer in dev to 5s to align with consumer setup. - [Enhancement] Alias
-2and-1withlatestandearliestfor seeking. - [Enhancement] Allow for usage of
latestandearliestin theKarafka::Pro::Iterator. - [Enhancement] Failures during
topics migrate(and other subcommands) don't show what topic failed, and why it's invalid. - [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
- [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
- [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
- [Enhancement] Provide
karafka topics helpsub-help due to nesting of Declarative Topics actions. - [Enhancement] Use independent keys for different states of reporting in scheduled messages.
- [Enhancement] Enrich scheduled messages state reporter with debug data.
- [Enhancement] Introduce a new state called
stoppedto the scheduled messages. - [Enhancement] Do not overwrite the
keyin the Pro DLQ dispatched messages for routing reasons. - [Enhancement] Introduce
errors_tracker.trace_idfor distributed error details correlation with the Web UI. - [Enhancement] Improve contracts validations reporting.
- [Enhancement] Optimize topic creation and repartitioning admin operations for topics with hundreds of partitions.
- [Refactor] Introduce a
bin/verify_kafka_warningsscript to clean Kafka from temporary test-suite topics. - [Refactor] Introduce a
bin/verify_topics_namingscript to ensure proper test topics naming convention. - [Refactor] Make sure all temporary topics have a
it-prefix in their name. - [Refactor] Improve CI specs parallelization.
- [Maintenance] Lower the
Karafka::Adminpoll_timeoutto 50 ms to improve responsiveness of admin operations. - [Maintenance] Require
karafka-rdkafka>=0.19.5due to usage of#rd_kafka_global_init, KIP-82, new producer caching engine and improvements to thepartition_keyassignments. - [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
- [Maintenance] Remove Rails
7.0specs due to upcoming EOL. - [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
- [Fix] Fix a case where
unknown_topic_or_parterror could leak out of the consumer on consumer shutdown. - [Fix] Fix missing
virtual_partitions.partitioner.errorcustom error logging in theLoggerListener. - [Fix] Prevent applied system filters
#timeoutfrom potentially interacting with user filters. - [Fix] Use more sane value in
Admin#seek_consumer_groupfor long ago. - [Fix] Prevent multiplexing of 1:1 from routing.
- [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
- [Fix] Fix inconsistency in the logs where
Karafka::Serveroriginating logs would not have server id reference. - [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
- [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
- [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
- [Fix] optparse double parse loses ARGV.
- [Fix]
karafkacannot be required without Bundler. - [Fix] Scheduled Messages re-seek moves to
lateston inheritance of initial offset when0offset is compacted. - [Fix] Seek to
:latestwithouttopic_partition_position(-1) will not seek at all. - [Fix] Extremely high turn over of scheduled messages can cause them not to reach EOF/Loaded state.
- [Fix] Fix incorrectly passed
max_wait_timeto rdkafka (ms instead of seconds) causing too long wait. - [Fix] Remove aggresive requerying of the Kafka cluster on topic creation/removal/altering.
- [Change] Move to trusted-publishers and remove signing since no longer needed.
v2.5.0.rc1
- [Breaking] Change how consistency of DLQ dispatches works in Pro (
partition_keyvs. direct partition id mapping). - [Breaking] Remove the headers
source_keyfrom the Pro DLQ dispatched messages as the original key is now fully preserved. - [Breaking] Use DLQ and Piping prefix
source_instead oforiginal_to align with naming convention of Kafka Streams and Apache Flink for future usage. - [Breaking] Rename scheduled jobs topics names in their config (Pro).
- [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
- [Enhancement] Support KIP-82 (header values of arrays).
- [Enhancement] Enhance errors tracker with
#countsthat contains per-error class specific counters for granular flow handling. - [Enhancement] Provide explicit
Karafka::Admin.copy_consumer_groupAPI. - [Enhancement] Return explicit value from
Karafka::Admin.copy_consumer_groupandKarafka::Admin.rename_consumer_groupAPIs. - [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
- [Enhancement] Make the error tracker for advanced DLQ strategies respond to
#topicand#partitionfor context aware dispatches. - [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
- [Enhancement] Enhance low-level
client.pauseevent with timeout value (if provided). - [Enhancement] Introduce
#marking_cursorAPI (defaults to#cursor) in the filtering API (Pro). - [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
- [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
- [Enhancement] Include info level rebalance logger listener data.
- [Enhancement] Include info level subscription start info.
- [Enhancement] Make the generic error handling in the
LoggerListenermore descriptive by logging also the error class. - [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
- [Enhancement] Change optional
#seekreset offset flag default totrueasfalseis almost never used and seek by default should move the internal consumer offset position as well. - [Enhancement] Include Swarm node ID in the swarm process tags.
- [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
- [Enhancement] Improve OSS vs. Pro specs execution isolation.
- [Enhancement] Preload
librdkafkacode prior to forking in the Swarm mode to save memory. - [Enhancement] Extract errors tracker class reference into an internal
errors_tracker_classconfig option (Pro). - [Enhancement] Support rdkafka native kafka polling customization for admin.
- [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
- [Enhancement] Set
topic.metadata.refresh.interval.msfor default producer in dev to 5s to align with consumer setup. - [Enhancement] Alias
-2and-1withlatestandearliestfor seeking. - [Enhancement] Allow for usage of
latestandearliestin theKarafka::Pro::Iterator. - [Enhancement] Failures during
topics migrate(and other subcommands) don't show what topic failed, and why it's invalid. - [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
- [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
- [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
- [Enhancement] Provide
karafka topics helpsub-help due to nesting of Declarative Topics actions. - [Enhancement] Use independent keys for different states of reporting in scheduled messages.
- [Enhancement] Enrich scheduled messages state reporter with debug data.
- [Enhancement] Introduce a new state called
stoppedto the scheduled messages. - [Enhancement] Do not overwrite the
keyin the Pro DLQ dispatched messages for routing reasons. - [Enhancement] Introduce
errors_tracker.trace_idfor distributed error details correlation with the Web UI. - [Refactor] Introduce a
bin/verify_kafka_warningsscript to clean Kafka from temporary test-suite topics. - [Refactor] Introduce a
bin/verify_topics_namingscript to ensure proper test topics naming convention. - [Refactor] Make sure all temporary topics have a
it-prefix in their name. - [Refactor] Improve CI specs parallelization.
- [Maintenance] Lower the
Karafka::Adminpoll_timeoutto 50 ms to improve responsiveness of admin operations. - [Maintenance] Require
karafka-rdkafka>=0.19.5due to usage of#rd_kafka_global_init, KIP-82, new producer caching engine and improvements to thepartition_keyassignments. - [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
- [Maintenance] Remove Rails
7.0specs due to upcoming EOL. - [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
- [Fix] Fix a case where
unknown_topic_or_parterror could leak out of the consumer on consumer shutdown. - [Fix] Fix missing
virtual_partitions.partitioner.errorcustom error logging in theLoggerListener. - [Fix] Prevent applied system filters
#timeoutfrom potentially interacting with user filters. - [Fix] Use more sane value in
Admin#seek_consumer_groupfor long ago. - [Fix] Prevent multiplexing of 1:1 from routing.
- [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
- [Fix] Fix inconsistency in the logs where
Karafka::Serveroriginating logs would not have server id reference. - [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
- [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
- [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
- [Fix] optparse double parse loses ARGV.
- [Fix]
karafkacannot be required without Bundler. - [Fix] Scheduled Messages re-seek moves to
lateston inheritance of initial offset when0offset is compacted. - [Fix] Seek to
:latestwithouttopic_partition_position(-1) will not seek at all. - [Fix] Extremely high turn over of scheduled messages can cause them not to reach EOF/Loaded state.
- [Change] Move to trusted-publishers and remove signing since no longer needed.
v2.5.0.beta2
- [Breaking] Use DLQ and Piping prefix
source_instead oforiginal_to align with naming convention of Kafka Streams and Apache Flink for future usage. - [Breaking] Rename scheduled jobs topics names in their config (Pro).
- [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
- [Enhancement] Support KIP-82 (header values of arrays).
- [Enhancement] Enhance errors tracker with
#countsthat contains per-error class specific counters for granular flow handling. - [Enhancement] Provide explicit
Karafka::Admin.copy_consumer_groupAPI. - [Enhancement] Return explicit value from
Karafka::Admin.copy_consumer_groupandKarafka::Admin.rename_consumer_groupAPIs. - [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
- [Enhancement] Make the error tracker for advanced DLQ strategies respond to
#topicand#partitionfor context aware dispatches. - [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
- [Enhancement] Enhance low-level
client.pauseevent with timeout value (if provided). - [Enhancement] Introduce
#marking_cursorAPI (defaults to#cursor) in the filtering API (Pro). - [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
- [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
- [Enhancement] Include info level rebalance logger listener data.
- [Enhancement] Include info level subscription start info.
- [Enhancement] Make the generic error handling in the
LoggerListenermore descriptive by logging also the error class. - [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
- [Enhancement] Change optional
#seekreset offset flag default totrueasfalseis almost never used and seek by default should move the internal consumer offset position as well. - [Enhancement] Include Swarm node ID in the swarm process tags.
- [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
- [Enhancement] Improve OSS vs. Pro specs execution isolation.
- [Enhancement] Preload
librdkafkacode prior to forking in the Swarm mode to save memory. - [Enhancement] Extract errors tracker class reference into an internal
errors_tracker_classconfig option (Pro). - [Enhancement] Support rdkafka native kafka polling customization for admin.
- [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
- [Enhancement] Set
topic.metadata.refresh.interval.msfor default producer in dev to 5s to align with consumer setup. - [Enhancement] Alias
-2and-1withlatestandearliestfor seeking. - [Enhancement] Allow for usage of
latestandearliestin theKarafka::Pro::Iterator. - [Enhancement] Failures during
topics migrate(and other subcommands) don't show what topic failed, and why it's invalid. - [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
- [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
- [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
- [Enhancement] Provide
karafka topics helpsub-help due to nesting of Declarative Topics actions. - [Refactor] Introduce a
bin/verify_kafka_warningsscript to clean Kafka from temporary test-suite topics. - [Refactor] Introduce a
bin/verify_topics_namingscript to ensure proper test topics naming convention. - [Refactor] Make sure all temporary topics have a
it-prefix in their name. - [Refactor] Improve CI specs parallelization.
- [Maintenance] Lower the
Karafka::Adminpoll_timeoutto 50 ms to improve responsiveness of admin operations. - [Maintenance] Require
karafka-rdkafka>=0.19.2due to usage of#rd_kafka_global_init, KIP-82 and the new producer caching engine. - [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
- [Maintenance] Remove Rails
7.0specs due to upcoming EOL. - [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
- [Fix] Fix a case where
unknown_topic_or_parterror could leak out of the consumer on consumer shutdown. - [Fix] Fix missing
virtual_partitions.partitioner.errorcustom error logging in theLoggerListener. - [Fix] Prevent applied system filters
#timeoutfrom potentially interacting with user filters. - [Fix] Use more sane value in
Admin#seek_consumer_groupfor long ago. - [Fix] Prevent multiplexing of 1:1 from routing.
- [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
- [Fix] Fix inconsistency in the logs where
Karafka::Serveroriginating logs would not have server id reference. - [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
- [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
- [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
- [Fix] optparse double parse loses ARGV.
- [Fix]
karafkacannot be required without Bundler. - [Fix] Scheduled Messages re-seek moves to
lateston inheritance of initial offset when0offset is compacted. - [Fix] Seek to
:latestwithouttopic_partition_position(-1) will not seek at all. - [Change] Move to trusted-publishers and remove signing since no longer needed.
v2.5.0.beta1
- [Breaking] Use DLQ and Piping prefix
source_instead oforiginal_to align with naming convention of Kafka Streams and Apache Flink for future usage. - [Breaking] Rename scheduled jobs topics names in their config (Pro).
- [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
- [Enhancement] Support KIP-82 (header values of arrays).
- [Enhancement] Enhance errors tracker with
#countsthat contains per-error class specific counters for granular flow handling. - [Enhancement] Provide explicit
Karafka::Admin.copy_consumer_groupAPI. - [Enhancement] Return explicit value from
Karafka::Admin.copy_consumer_groupandKarafka::Admin.rename_consumer_groupAPIs. - [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
- [Enhancement] Make the error tracker for advanced DLQ strategies respond to
#topicand#partitionfor context aware dispatches. - [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
- [Enhancement] Enhance low-level
client.pauseevent with timeout value (if provided). - [Enhancement] Introduce
#marking_cursorAPI (defaults to#cursor) in the filtering API (Pro). - [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
- [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
- [Enhancement] Include info level rebalance logger listener data.
- [Enhancement] Include info level subscription start info.
- [Enhancement] Make the generic error handling in the
LoggerListenermore descriptive by logging also the error class. - [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
- [Enhancement] Change optional
#seekreset offset flag default totrueasfalseis almost never used and seek by default should move the internal consumer offset position as well. - [Enhancement] Include Swarm node ID in the swarm process tags.
- [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
- [Enhancement] Improve OSS vs. Pro specs execution isolation.
- [Enhancement] Preload
librdkafkacode prior to forking in the Swarm mode to save memory. - [Enhancement] Extract errors tracker class reference into an internal
errors_tracker_classconfig option (Pro). - [Enhancement] Support rdkafka native kafka polling customization for admin.
- [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
- [Enhancement] Set
topic.metadata.refresh.interval.msfor default producer in dev to 5s to align with consumer setup. - [Enhancement] Alias
-2and-1withlatestandearliestfor seeking. - [Enhancement] Allow for usage of
latestandearliestin theKarafka::Pro::Iterator. - [Refactor] Introduce a
bin/verify_kafka_warningsscript to clean Kafka from temporary test-suite topics. - [Refactor] Introduce a
bin/verify_topics_namingscript to ensure proper test topics naming convention. - [Refactor] Make sure all temporary topics have a
it-prefix in their name. - [Refactor] Improve CI specs parallelization.
- [Maintenance] Lower the
Karafka::Adminpoll_timeoutto 50 ms to improve responsiveness of admin operations. - [Maintenance] Require
karafka-rdkafka>=0.19.2due to usage of#rd_kafka_global_init, KIP-82 and the new producer caching engine. - [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
- [Maintenance] Remove Rails
7.0specs due to upcoming EOL. - [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
- [Fix] Fix a case where
unknown_topic_or_parterror could leak out of the consumer on consumer shutdown. - [Fix] Fix missing
virtual_partitions.partitioner.errorcustom error logging in theLoggerListener. - [Fix] Prevent applied system filters
#timeoutfrom potentially interacting with user filters. - [Fix] Use more sane value in
Admin#seek_consumer_groupfor long ago. - [Fix] Prevent multiplexing of 1:1 from routing.
- [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
- [Fix] Fix inconsistency in the logs where
Karafka::Serveroriginating logs would not have server id reference. - [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
- [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
- [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
- [Fix] optparse double parse loses ARGV.
- [Fix]
karafkacannot be required without Bundler. - [Fix] Scheduled Messages re-seek moves to
lateston inheritance of initial offset when0offset is compacted.
v2.4.18
v2.4.17
- [Enhancement] Clean message key and headers when cleaning messages via the cleaner API (Pro).
- [Enhancement] Allow for setting
metadata: falsein the cleaner API for granular cleaning control (Pro) - [Enhancement] Instrument successful transaction via
consumer.consuming.transactionevent (Pro).