Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Releases: karafka/karafka

v2.5.2

31 Oct 21:42
395adb7

Choose a tag to compare

  • [EOL] Remove Rails 7.1 support according to EOL while not blocking Rails 7.1 usage.
  • [Enhancement] Retry on the KIP-848 stale_member_epoch error.
  • [Enhancement] Provide Karafka::Admin.trigger_rebalance API to programmatically trigger consumer group rebalances for operational purposes.
  • [Enhancement] Nest pause configuration under config.pause.* namespace (config.pause.timeout, config.pause.max_timeout, config.pause.with_exponential_backoff) while maintaining backwards compatibility with the old flat API (config.pause_timeout, etc.) via delegation methods that will be removed in Karafka 2.6.
  • [Enhancement] Detect and track involuntary assignment loss during long-running processing that exceeds max.poll.interval.ms via client.events_poll event and automatically update Karafka::App.assignments to reflect reality.
  • [Enhancement] Extend Karafka::Admin.read_watermark_offsets to accept either a single topic with partition or a hash of multiple topics with partitions, using a single consumer instance for improved efficiency when querying multiple partitions.
  • [Enhancement] Add configurable Karafka::ActiveJob::Deserializer to support custom serialization formats (Avro, Protobuf, etc.) for ActiveJob payloads.
  • [Fix] Fix ActiveJob Continuation invalid class reference bug.

v2.5.1

29 Sep 08:28
4bfdc3a

Choose a tag to compare

  • [Breaking] Remove Ruby 3.1 support according to EOL.
  • [Feature] Support Swarm mode on MacOS.
  • [Enhancement] Support past dispatch_at times with jitter: 0 in the OSS Karafka to support ActiveJob continuation.
  • [Enhancement] Use direct topic dispatches when dispatch_at is used for past times to bypass Scheduled Messages flow.
  • [Enhancement] Support immediate error raising with auto.offset.reset set to error.
  • [Enhancement] Don't create not needed dirs in the non-Rails setup template.
  • [Enhancement] Improve printing of TTIN to separate threads.
  • [Enhancement] Support transactional ID resource in the Admin ACL management.
  • [Enhancement] Move post-forceful shutdown termination wait value to a config.
  • [Enhancement] Include consumer group, subscription group and other details in error logs for key error locations.
  • [Enhancement] Inherit from ActiveJob::QueueAdapters::AbstractAdapter when possible for ActiveJob base class.
  • [Enhancement] Disable Nagle algorithm by default for improved network performance.
  • [Enhancement] Optimize the messages buffer array memory allocation pattern.
  • [Maintenance] Add basic direct DD integration spec via DD gem karafka monitoring feature.
  • [Maintenance] Add integration specs for WaterDrop connection pool usage from within consumers.
  • [Refactoring] Comprehensive Admin module refactoring: Extract topic operations into Admin::Topics class and consumer group operations into Admin::ConsumerGroups class with proper inheritance hierarchy, cross-class method usage optimization, and constants moved to appropriate locations where they are actually used.
  • [Refactoring] Move routing-related contracts from Karafka::Contracts:: to Karafka::Routing::Contracts:: namespace and reorganize error message structure in YAML files under routing: scope for better code organization and logical grouping.
  • [Refactoring] Move config-related contracts from Karafka::Contracts::Config to Karafka::Setup::Contracts::Config namespace and reorganize error message structure in YAML files under setup: scope for better code organization and logical grouping.
  • [Refactoring] Move CLI server contracts from Karafka::Contracts::ServerCliOptions to Karafka::Cli::Contracts::Server namespace and reorganize error message structure in YAML files under cli: scope for improved naming consistency and logical grouping.
  • [Refactoring] Replace execution mode symbol-based checks with dedicated ExecutionMode class providing cleaner API with query methods (#swarm?, #embedded?) and state change methods (#swarm!, #embedded!) for improved type safety and code clarity.
  • [Refactoring] Replace connection client mode symbol-based checks with dedicated Connection::Mode class providing cleaner API with query methods (#subscribe?, #assign?) and state change methods (#subscribe!, #assign!) for improved code clarity.
  • [Fix] Improve same timestamp dispatch in scheduled messages on Ruby 3.2.
  • [Fix] Fix incorrect (6 seconds vs 60 seconds) reset of connections on non-recoverable errors.
  • [Fix] Introduce mutex-safe and thread-safe #inspect where needed.
  • [Fix] Fix too loose requirement of Ruby 3.0 when it was 3.1 via transitive dependencies.
  • [Fix] Fix Pro Cleaner Messages compatibility with external libraries that prepend modules to #each method (e.g., DataDog tracing).
  • [Fix] SG exclusion in swarm triggers a contract validation error.
  • [Change] Require waterdrop >= 2.8.10 to support new features.
  • [Change] Require karafka-rdkafka >= 0.22.0 to support new features and require SSL-bug free version.
  • [Change] Remove no longer needed cooperative.sticky rebalance patch.
  • [Change] Normalize how libs and deps are required (no functional change for the end user)
  • [Change] Remove Ruby 3.1 specs according to the EOL schedule.

v2.5.1.beta1

09 Sep 14:26
8329614

Choose a tag to compare

v2.5.1.beta1 Pre-release
Pre-release
  • [Feature] Support Swarm mode on MacOS.
  • [Enhancement] Support past dispatch_at times with jitter: 0 in the OSS Karafka to support ActiveJob continuation.
  • [Enhancement] Use direct topic dispatches when dispatch_at is used for past times to bypass Scheduled Messages flow.
  • [Enhancement] Support immediate error raising with auto.offset.reset set to error.
  • [Enhancement] Don't create not needed dirs in the non-Rails setup template.
  • [Enhancement] Improve printing of TTIN to separate threads.
  • [Enhancement] Support transactional ID resource in the Admin ACL management.
  • [Enhancement] Move post-forceful shutdown termination wait value to a config.
  • [Enhancement] Include consumer group, subscription group and other details in error logs for key error locations.
  • [Enhancement] Inherit from ActiveJob::QueueAdapters::AbstractAdapter when possible for ActiveJob base class.
  • [Enhancement] Disable Nagle algorithm by default for improved network performance.
  • [Maintenance] Add basic direct DD integration spec via DD gem karafka monitoring feature.
  • [Refactoring] Comprehensive Admin module refactoring: Extract topic operations into Admin::Topics class and consumer group operations into Admin::ConsumerGroups class with proper inheritance hierarchy, cross-class method usage optimization, and constants moved to appropriate locations where they are actually used.
  • [Refactoring] Move routing-related contracts from Karafka::Contracts:: to Karafka::Routing::Contracts:: namespace and reorganize error message structure in YAML files under routing: scope for better code organization and logical grouping.
  • [Refactoring] Move config-related contracts from Karafka::Contracts::Config to Karafka::Setup::Contracts::Config namespace and reorganize error message structure in YAML files under setup: scope for better code organization and logical grouping.
  • [Refactoring] Move CLI server contracts from Karafka::Contracts::ServerCliOptions to Karafka::Cli::Contracts::Server namespace and reorganize error message structure in YAML files under cli: scope for improved naming consistency and logical grouping.
  • [Fix] Improve same timestamp dispatch in scheduled messages on Ruby 3.2.
  • [Fix] Fix incorrect (6 seconds vs 60 seconds) reset of connections on non-recoverable errors.
  • [Fix] Introduce mutex-safe and thread-safe #inspect where needed.
  • [Fix] Fix too loose requirement of Ruby 3.0 when it was 3.1 via transitive dependencies.
  • [Fix] Fix Pro Cleaner Messages compatibility with external libraries that prepend modules to #each method (e.g., DataDog tracing).
  • [Change] Require karafka-rdkafka >= 0.21.0 to support new features.
  • [Change] Remove no longer needed cooperative.sticky rebalance patch.
  • [Change] Normalize how libs and deps are required (no functional change for the end user)
  • [Change] Remove Ruby 3.1 specs according to the EOL schedule.

v2.5.0

15 Jun 20:00
2b8b09d

Choose a tag to compare

  • [Breaking] Change how consistency of DLQ dispatches works in Pro (partition_key vs. direct partition id mapping).
  • [Breaking] Remove the headers source_key from the Pro DLQ dispatched messages as the original key is now fully preserved.
  • [Breaking] Use DLQ and Piping prefix source_ instead of original_ to align with naming convention of Kafka Streams and Apache Flink for future usage.
  • [Breaking] Rename scheduled jobs topics names in their config (Pro).
  • [Breaking] Change K8s listener response from 204 to 200 and include JSON body with reasons.
  • [Breaking] Replace admin config max_attempts with max_retries_duration and
  • [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
  • [Enhancement] Normalize topic + partition logs format.
  • [Enhancement] Support KIP-82 (header values of arrays).
  • [Enhancement] Enhance errors tracker with #counts that contains per-error class specific counters for granular flow handling.
  • [Enhancement] Provide explicit Karafka::Admin.copy_consumer_group API.
  • [Enhancement] Return explicit value from Karafka::Admin.copy_consumer_group and Karafka::Admin.rename_consumer_group APIs.
  • [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
  • [Enhancement] Make the error tracker for advanced DLQ strategies respond to #topic and #partition for context aware dispatches.
  • [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
  • [Enhancement] Enhance low-level client.pause event with timeout value (if provided).
  • [Enhancement] Introduce #marking_cursor API (defaults to #cursor) in the filtering API (Pro).
  • [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
  • [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
  • [Enhancement] Include info level rebalance logger listener data.
  • [Enhancement] Include info level subscription start info.
  • [Enhancement] Make the generic error handling in the LoggerListener more descriptive by logging also the error class.
  • [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
  • [Enhancement] Change optional #seek reset offset flag default to true as false is almost never used and seek by default should move the internal consumer offset position as well.
  • [Enhancement] Include Swarm node ID in the swarm process tags.
  • [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
  • [Enhancement] Improve OSS vs. Pro specs execution isolation.
  • [Enhancement] Preload librdkafka code prior to forking in the Swarm mode to save memory.
  • [Enhancement] Extract errors tracker class reference into an internal errors_tracker_class config option (Pro).
  • [Enhancement] Support rdkafka native kafka polling customization for admin.
  • [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
  • [Enhancement] Set topic.metadata.refresh.interval.ms for default producer in dev to 5s to align with consumer setup.
  • [Enhancement] Alias -2 and -1 with latest and earliest for seeking.
  • [Enhancement] Allow for usage of latest and earliest in the Karafka::Pro::Iterator.
  • [Enhancement] Failures during topics migrate (and other subcommands) don't show what topic failed, and why it's invalid.
  • [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
  • [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
  • [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
  • [Enhancement] Provide karafka topics help sub-help due to nesting of Declarative Topics actions.
  • [Enhancement] Use independent keys for different states of reporting in scheduled messages.
  • [Enhancement] Enrich scheduled messages state reporter with debug data.
  • [Enhancement] Introduce a new state called stopped to the scheduled messages.
  • [Enhancement] Do not overwrite the key in the Pro DLQ dispatched messages for routing reasons.
  • [Enhancement] Introduce errors_tracker.trace_id for distributed error details correlation with the Web UI.
  • [Enhancement] Improve contracts validations reporting.
  • [Enhancement] Optimize topic creation and repartitioning admin operations for topics with hundreds of partitions.
  • [Refactor] Introduce a bin/verify_kafka_warnings script to clean Kafka from temporary test-suite topics.
  • [Refactor] Introduce a bin/verify_topics_naming script to ensure proper test topics naming convention.
  • [Refactor] Make sure all temporary topics have a it- prefix in their name.
  • [Refactor] Improve CI specs parallelization.
  • [Maintenance] Lower the Karafka::Admin poll_timeout to 50 ms to improve responsiveness of admin operations.
  • [Maintenance] Require karafka-rdkafka >= 0.19.5 due to usage of #rd_kafka_global_init, KIP-82, new producer caching engine and improvements to the partition_key assignments.
  • [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
  • [Maintenance] Remove Rails 7.0 specs due to upcoming EOL.
  • [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
  • [Fix] Fix a case where unknown_topic_or_part error could leak out of the consumer on consumer shutdown.
  • [Fix] Fix missing virtual_partitions.partitioner.error custom error logging in the LoggerListener.
  • [Fix] Prevent applied system filters #timeout from potentially interacting with user filters.
  • [Fix] Use more sane value in Admin#seek_consumer_group for long ago.
  • [Fix] Prevent multiplexing of 1:1 from routing.
  • [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
  • [Fix] Fix inconsistency in the logs where Karafka::Server originating logs would not have server id reference.
  • [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
  • [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
  • [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
  • [Fix] optparse double parse loses ARGV.
  • [Fix] karafka cannot be required without Bundler.
  • [Fix] Scheduled Messages re-seek moves to latest on inheritance of initial offset when 0 offset is compacted.
  • [Fix] Seek to :latest without topic_partition_position (-1) will not seek at all.
  • [Fix] Extremely high turn over of scheduled messages can cause them not to reach EOF/Loaded state.
  • [Fix] Fix incorrectly passed max_wait_time to rdkafka (ms instead of seconds) causing too long wait.
  • [Fix] Remove aggresive requerying of the Kafka cluster on topic creation/removal/altering.
  • [Change] Move to trusted-publishers and remove signing since no longer needed.

v2.5.0.rc2

14 Jun 19:50
aef56ac

Choose a tag to compare

v2.5.0.rc2 Pre-release
Pre-release
  • [Breaking] Change how consistency of DLQ dispatches works in Pro (partition_key vs. direct partition id mapping).
  • [Breaking] Remove the headers source_key from the Pro DLQ dispatched messages as the original key is now fully preserved.
  • [Breaking] Use DLQ and Piping prefix source_ instead of original_ to align with naming convention of Kafka Streams and Apache Flink for future usage.
  • [Breaking] Rename scheduled jobs topics names in their config (Pro).
  • [Breaking] Change K8s listener response from 204 to 200 and include JSON body with reasons.
  • [Breaking] Replace admin config max_attempts with max_retries_duration and
  • [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
  • [Enhancement] Normalize topic + partition logs format.
  • [Enhancement] Support KIP-82 (header values of arrays).
  • [Enhancement] Enhance errors tracker with #counts that contains per-error class specific counters for granular flow handling.
  • [Enhancement] Provide explicit Karafka::Admin.copy_consumer_group API.
  • [Enhancement] Return explicit value from Karafka::Admin.copy_consumer_group and Karafka::Admin.rename_consumer_group APIs.
  • [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
  • [Enhancement] Make the error tracker for advanced DLQ strategies respond to #topic and #partition for context aware dispatches.
  • [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
  • [Enhancement] Enhance low-level client.pause event with timeout value (if provided).
  • [Enhancement] Introduce #marking_cursor API (defaults to #cursor) in the filtering API (Pro).
  • [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
  • [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
  • [Enhancement] Include info level rebalance logger listener data.
  • [Enhancement] Include info level subscription start info.
  • [Enhancement] Make the generic error handling in the LoggerListener more descriptive by logging also the error class.
  • [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
  • [Enhancement] Change optional #seek reset offset flag default to true as false is almost never used and seek by default should move the internal consumer offset position as well.
  • [Enhancement] Include Swarm node ID in the swarm process tags.
  • [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
  • [Enhancement] Improve OSS vs. Pro specs execution isolation.
  • [Enhancement] Preload librdkafka code prior to forking in the Swarm mode to save memory.
  • [Enhancement] Extract errors tracker class reference into an internal errors_tracker_class config option (Pro).
  • [Enhancement] Support rdkafka native kafka polling customization for admin.
  • [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
  • [Enhancement] Set topic.metadata.refresh.interval.ms for default producer in dev to 5s to align with consumer setup.
  • [Enhancement] Alias -2 and -1 with latest and earliest for seeking.
  • [Enhancement] Allow for usage of latest and earliest in the Karafka::Pro::Iterator.
  • [Enhancement] Failures during topics migrate (and other subcommands) don't show what topic failed, and why it's invalid.
  • [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
  • [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
  • [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
  • [Enhancement] Provide karafka topics help sub-help due to nesting of Declarative Topics actions.
  • [Enhancement] Use independent keys for different states of reporting in scheduled messages.
  • [Enhancement] Enrich scheduled messages state reporter with debug data.
  • [Enhancement] Introduce a new state called stopped to the scheduled messages.
  • [Enhancement] Do not overwrite the key in the Pro DLQ dispatched messages for routing reasons.
  • [Enhancement] Introduce errors_tracker.trace_id for distributed error details correlation with the Web UI.
  • [Enhancement] Improve contracts validations reporting.
  • [Enhancement] Optimize topic creation and repartitioning admin operations for topics with hundreds of partitions.
  • [Refactor] Introduce a bin/verify_kafka_warnings script to clean Kafka from temporary test-suite topics.
  • [Refactor] Introduce a bin/verify_topics_naming script to ensure proper test topics naming convention.
  • [Refactor] Make sure all temporary topics have a it- prefix in their name.
  • [Refactor] Improve CI specs parallelization.
  • [Maintenance] Lower the Karafka::Admin poll_timeout to 50 ms to improve responsiveness of admin operations.
  • [Maintenance] Require karafka-rdkafka >= 0.19.5 due to usage of #rd_kafka_global_init, KIP-82, new producer caching engine and improvements to the partition_key assignments.
  • [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
  • [Maintenance] Remove Rails 7.0 specs due to upcoming EOL.
  • [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
  • [Fix] Fix a case where unknown_topic_or_part error could leak out of the consumer on consumer shutdown.
  • [Fix] Fix missing virtual_partitions.partitioner.error custom error logging in the LoggerListener.
  • [Fix] Prevent applied system filters #timeout from potentially interacting with user filters.
  • [Fix] Use more sane value in Admin#seek_consumer_group for long ago.
  • [Fix] Prevent multiplexing of 1:1 from routing.
  • [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
  • [Fix] Fix inconsistency in the logs where Karafka::Server originating logs would not have server id reference.
  • [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
  • [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
  • [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
  • [Fix] optparse double parse loses ARGV.
  • [Fix] karafka cannot be required without Bundler.
  • [Fix] Scheduled Messages re-seek moves to latest on inheritance of initial offset when 0 offset is compacted.
  • [Fix] Seek to :latest without topic_partition_position (-1) will not seek at all.
  • [Fix] Extremely high turn over of scheduled messages can cause them not to reach EOF/Loaded state.
  • [Fix] Fix incorrectly passed max_wait_time to rdkafka (ms instead of seconds) causing too long wait.
  • [Fix] Remove aggresive requerying of the Kafka cluster on topic creation/removal/altering.
  • [Change] Move to trusted-publishers and remove signing since no longer needed.

v2.5.0.rc1

03 Jun 15:13
c543967

Choose a tag to compare

v2.5.0.rc1 Pre-release
Pre-release
  • [Breaking] Change how consistency of DLQ dispatches works in Pro (partition_key vs. direct partition id mapping).
  • [Breaking] Remove the headers source_key from the Pro DLQ dispatched messages as the original key is now fully preserved.
  • [Breaking] Use DLQ and Piping prefix source_ instead of original_ to align with naming convention of Kafka Streams and Apache Flink for future usage.
  • [Breaking] Rename scheduled jobs topics names in their config (Pro).
  • [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
  • [Enhancement] Support KIP-82 (header values of arrays).
  • [Enhancement] Enhance errors tracker with #counts that contains per-error class specific counters for granular flow handling.
  • [Enhancement] Provide explicit Karafka::Admin.copy_consumer_group API.
  • [Enhancement] Return explicit value from Karafka::Admin.copy_consumer_group and Karafka::Admin.rename_consumer_group APIs.
  • [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
  • [Enhancement] Make the error tracker for advanced DLQ strategies respond to #topic and #partition for context aware dispatches.
  • [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
  • [Enhancement] Enhance low-level client.pause event with timeout value (if provided).
  • [Enhancement] Introduce #marking_cursor API (defaults to #cursor) in the filtering API (Pro).
  • [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
  • [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
  • [Enhancement] Include info level rebalance logger listener data.
  • [Enhancement] Include info level subscription start info.
  • [Enhancement] Make the generic error handling in the LoggerListener more descriptive by logging also the error class.
  • [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
  • [Enhancement] Change optional #seek reset offset flag default to true as false is almost never used and seek by default should move the internal consumer offset position as well.
  • [Enhancement] Include Swarm node ID in the swarm process tags.
  • [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
  • [Enhancement] Improve OSS vs. Pro specs execution isolation.
  • [Enhancement] Preload librdkafka code prior to forking in the Swarm mode to save memory.
  • [Enhancement] Extract errors tracker class reference into an internal errors_tracker_class config option (Pro).
  • [Enhancement] Support rdkafka native kafka polling customization for admin.
  • [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
  • [Enhancement] Set topic.metadata.refresh.interval.ms for default producer in dev to 5s to align with consumer setup.
  • [Enhancement] Alias -2 and -1 with latest and earliest for seeking.
  • [Enhancement] Allow for usage of latest and earliest in the Karafka::Pro::Iterator.
  • [Enhancement] Failures during topics migrate (and other subcommands) don't show what topic failed, and why it's invalid.
  • [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
  • [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
  • [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
  • [Enhancement] Provide karafka topics help sub-help due to nesting of Declarative Topics actions.
  • [Enhancement] Use independent keys for different states of reporting in scheduled messages.
  • [Enhancement] Enrich scheduled messages state reporter with debug data.
  • [Enhancement] Introduce a new state called stopped to the scheduled messages.
  • [Enhancement] Do not overwrite the key in the Pro DLQ dispatched messages for routing reasons.
  • [Enhancement] Introduce errors_tracker.trace_id for distributed error details correlation with the Web UI.
  • [Refactor] Introduce a bin/verify_kafka_warnings script to clean Kafka from temporary test-suite topics.
  • [Refactor] Introduce a bin/verify_topics_naming script to ensure proper test topics naming convention.
  • [Refactor] Make sure all temporary topics have a it- prefix in their name.
  • [Refactor] Improve CI specs parallelization.
  • [Maintenance] Lower the Karafka::Admin poll_timeout to 50 ms to improve responsiveness of admin operations.
  • [Maintenance] Require karafka-rdkafka >= 0.19.5 due to usage of #rd_kafka_global_init, KIP-82, new producer caching engine and improvements to the partition_key assignments.
  • [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
  • [Maintenance] Remove Rails 7.0 specs due to upcoming EOL.
  • [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
  • [Fix] Fix a case where unknown_topic_or_part error could leak out of the consumer on consumer shutdown.
  • [Fix] Fix missing virtual_partitions.partitioner.error custom error logging in the LoggerListener.
  • [Fix] Prevent applied system filters #timeout from potentially interacting with user filters.
  • [Fix] Use more sane value in Admin#seek_consumer_group for long ago.
  • [Fix] Prevent multiplexing of 1:1 from routing.
  • [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
  • [Fix] Fix inconsistency in the logs where Karafka::Server originating logs would not have server id reference.
  • [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
  • [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
  • [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
  • [Fix] optparse double parse loses ARGV.
  • [Fix] karafka cannot be required without Bundler.
  • [Fix] Scheduled Messages re-seek moves to latest on inheritance of initial offset when 0 offset is compacted.
  • [Fix] Seek to :latest without topic_partition_position (-1) will not seek at all.
  • [Fix] Extremely high turn over of scheduled messages can cause them not to reach EOF/Loaded state.
  • [Change] Move to trusted-publishers and remove signing since no longer needed.

v2.5.0.beta2

25 May 14:11
6a89127

Choose a tag to compare

v2.5.0.beta2 Pre-release
Pre-release
  • [Breaking] Use DLQ and Piping prefix source_ instead of original_ to align with naming convention of Kafka Streams and Apache Flink for future usage.
  • [Breaking] Rename scheduled jobs topics names in their config (Pro).
  • [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
  • [Enhancement] Support KIP-82 (header values of arrays).
  • [Enhancement] Enhance errors tracker with #counts that contains per-error class specific counters for granular flow handling.
  • [Enhancement] Provide explicit Karafka::Admin.copy_consumer_group API.
  • [Enhancement] Return explicit value from Karafka::Admin.copy_consumer_group and Karafka::Admin.rename_consumer_group APIs.
  • [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
  • [Enhancement] Make the error tracker for advanced DLQ strategies respond to #topic and #partition for context aware dispatches.
  • [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
  • [Enhancement] Enhance low-level client.pause event with timeout value (if provided).
  • [Enhancement] Introduce #marking_cursor API (defaults to #cursor) in the filtering API (Pro).
  • [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
  • [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
  • [Enhancement] Include info level rebalance logger listener data.
  • [Enhancement] Include info level subscription start info.
  • [Enhancement] Make the generic error handling in the LoggerListener more descriptive by logging also the error class.
  • [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
  • [Enhancement] Change optional #seek reset offset flag default to true as false is almost never used and seek by default should move the internal consumer offset position as well.
  • [Enhancement] Include Swarm node ID in the swarm process tags.
  • [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
  • [Enhancement] Improve OSS vs. Pro specs execution isolation.
  • [Enhancement] Preload librdkafka code prior to forking in the Swarm mode to save memory.
  • [Enhancement] Extract errors tracker class reference into an internal errors_tracker_class config option (Pro).
  • [Enhancement] Support rdkafka native kafka polling customization for admin.
  • [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
  • [Enhancement] Set topic.metadata.refresh.interval.ms for default producer in dev to 5s to align with consumer setup.
  • [Enhancement] Alias -2 and -1 with latest and earliest for seeking.
  • [Enhancement] Allow for usage of latest and earliest in the Karafka::Pro::Iterator.
  • [Enhancement] Failures during topics migrate (and other subcommands) don't show what topic failed, and why it's invalid.
  • [Enhancement] Apply changes to topics configuration in atomic independent requests when using Declarative Topics.
  • [Enhancement] Execute the help CLI command when no command provided (similar to Rails) to improve DX.
  • [Enhancement] Remove backtrace from the CLI error for incorrect commands (similar to Rails) to improve DX.
  • [Enhancement] Provide karafka topics help sub-help due to nesting of Declarative Topics actions.
  • [Refactor] Introduce a bin/verify_kafka_warnings script to clean Kafka from temporary test-suite topics.
  • [Refactor] Introduce a bin/verify_topics_naming script to ensure proper test topics naming convention.
  • [Refactor] Make sure all temporary topics have a it- prefix in their name.
  • [Refactor] Improve CI specs parallelization.
  • [Maintenance] Lower the Karafka::Admin poll_timeout to 50 ms to improve responsiveness of admin operations.
  • [Maintenance] Require karafka-rdkafka >= 0.19.2 due to usage of #rd_kafka_global_init, KIP-82 and the new producer caching engine.
  • [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
  • [Maintenance] Remove Rails 7.0 specs due to upcoming EOL.
  • [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
  • [Fix] Fix a case where unknown_topic_or_part error could leak out of the consumer on consumer shutdown.
  • [Fix] Fix missing virtual_partitions.partitioner.error custom error logging in the LoggerListener.
  • [Fix] Prevent applied system filters #timeout from potentially interacting with user filters.
  • [Fix] Use more sane value in Admin#seek_consumer_group for long ago.
  • [Fix] Prevent multiplexing of 1:1 from routing.
  • [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
  • [Fix] Fix inconsistency in the logs where Karafka::Server originating logs would not have server id reference.
  • [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
  • [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
  • [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
  • [Fix] optparse double parse loses ARGV.
  • [Fix] karafka cannot be required without Bundler.
  • [Fix] Scheduled Messages re-seek moves to latest on inheritance of initial offset when 0 offset is compacted.
  • [Fix] Seek to :latest without topic_partition_position (-1) will not seek at all.
  • [Change] Move to trusted-publishers and remove signing since no longer needed.

v2.5.0.beta1

21 May 15:42
f7b5a4d

Choose a tag to compare

v2.5.0.beta1 Pre-release
Pre-release
  • [Breaking] Use DLQ and Piping prefix source_ instead of original_ to align with naming convention of Kafka Streams and Apache Flink for future usage.
  • [Breaking] Rename scheduled jobs topics names in their config (Pro).
  • [Feature] Parallel Segments for concurrent processing of the same partition with more than partition count of processes (Pro).
  • [Enhancement] Support KIP-82 (header values of arrays).
  • [Enhancement] Enhance errors tracker with #counts that contains per-error class specific counters for granular flow handling.
  • [Enhancement] Provide explicit Karafka::Admin.copy_consumer_group API.
  • [Enhancement] Return explicit value from Karafka::Admin.copy_consumer_group and Karafka::Admin.rename_consumer_group APIs.
  • [Enhancement] Introduce balanced non-consistent VP distributor improving the utilization up to 50% (Pro).
  • [Enhancement] Make the error tracker for advanced DLQ strategies respond to #topic and #partition for context aware dispatches.
  • [Enhancement] Allow setting the workers thread priority and set it to -1 (50ms) by default.
  • [Enhancement] Enhance low-level client.pause event with timeout value (if provided).
  • [Enhancement] Introduce #marking_cursor API (defaults to #cursor) in the filtering API (Pro).
  • [Enhancement] Support multiple DLQ target topics via context aware strategies (Pro).
  • [Enhancement] Raise error when post-transactional committing of offset is done outside of the transaction (Pro).
  • [Enhancement] Include info level rebalance logger listener data.
  • [Enhancement] Include info level subscription start info.
  • [Enhancement] Make the generic error handling in the LoggerListener more descriptive by logging also the error class.
  • [Enhancement] Allow marking older offsets to support advanced rewind capabilities.
  • [Enhancement] Change optional #seek reset offset flag default to true as false is almost never used and seek by default should move the internal consumer offset position as well.
  • [Enhancement] Include Swarm node ID in the swarm process tags.
  • [Enhancement] Replace internal usage of MD5 with SHA256 for FIPS.
  • [Enhancement] Improve OSS vs. Pro specs execution isolation.
  • [Enhancement] Preload librdkafka code prior to forking in the Swarm mode to save memory.
  • [Enhancement] Extract errors tracker class reference into an internal errors_tracker_class config option (Pro).
  • [Enhancement] Support rdkafka native kafka polling customization for admin.
  • [Enhancement] Customize the multiplexing scale delay (Pro) per consumer group (Pro).
  • [Enhancement] Set topic.metadata.refresh.interval.ms for default producer in dev to 5s to align with consumer setup.
  • [Enhancement] Alias -2 and -1 with latest and earliest for seeking.
  • [Enhancement] Allow for usage of latest and earliest in the Karafka::Pro::Iterator.
  • [Refactor] Introduce a bin/verify_kafka_warnings script to clean Kafka from temporary test-suite topics.
  • [Refactor] Introduce a bin/verify_topics_naming script to ensure proper test topics naming convention.
  • [Refactor] Make sure all temporary topics have a it- prefix in their name.
  • [Refactor] Improve CI specs parallelization.
  • [Maintenance] Lower the Karafka::Admin poll_timeout to 50 ms to improve responsiveness of admin operations.
  • [Maintenance] Require karafka-rdkafka >= 0.19.2 due to usage of #rd_kafka_global_init, KIP-82 and the new producer caching engine.
  • [Maintenance] Add Deimos routing patch into integration suite not to break it in the future.
  • [Maintenance] Remove Rails 7.0 specs due to upcoming EOL.
  • [Fix] Fix Recurring Tasks and Scheduled Messages not working with Swarm (using closed producer).
  • [Fix] Fix a case where unknown_topic_or_part error could leak out of the consumer on consumer shutdown.
  • [Fix] Fix missing virtual_partitions.partitioner.error custom error logging in the LoggerListener.
  • [Fix] Prevent applied system filters #timeout from potentially interacting with user filters.
  • [Fix] Use more sane value in Admin#seek_consumer_group for long ago.
  • [Fix] Prevent multiplexing of 1:1 from routing.
  • [Fix] WaterDrop level aborting transaction may cause seek offset to move (Pro).
  • [Fix] Fix inconsistency in the logs where Karafka::Server originating logs would not have server id reference.
  • [Fix] Fix inconsistency in the logs where OS signal originating logs would not have server id reference.
  • [Fix] Post-fork WaterDrop instance looses some of the non-kafka settings.
  • [Fix] Max epoch tracking for early cleanup causes messages to be skipped until reload.
  • [Fix] optparse double parse loses ARGV.
  • [Fix] karafka cannot be required without Bundler.
  • [Fix] Scheduled Messages re-seek moves to latest on inheritance of initial offset when 0 offset is compacted.

v2.4.18

09 Apr 20:41
d4dfae4

Choose a tag to compare

  • [Fix] Make sure Bundler.with_unbundled_env is not called multiple times.

v2.4.17

15 Jan 18:25
ee198af

Choose a tag to compare

  • [Enhancement] Clean message key and headers when cleaning messages via the cleaner API (Pro).
  • [Enhancement] Allow for setting metadata: false in the cleaner API for granular cleaning control (Pro)
  • [Enhancement] Instrument successful transaction via consumer.consuming.transaction event (Pro).