Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(lambda): reset volatile DynamoDB Streams ESM checkpoints on restart - #2077

Merged
hectorvent merged 1 commit into
floci-io:mainfrom
aniketshukla1:fix/dynamodb-streams-esm-stale-checkpoints
Aug 1, 2026
Merged

fix(lambda): reset volatile DynamoDB Streams ESM checkpoints on restart#2077
hectorvent merged 1 commit into
floci-io:mainfrom
aniketshukla1:fix/dynamodb-streams-esm-stale-checkpoints

Conversation

@aniketshukla1

@aniketshukla1 aniketshukla1 commented Aug 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

A DynamoDB stream and its sequence numbers live only in memory (the record buffer and the AtomicLong counter in DynamoDbStreamService are never persisted), so a restart recreates the stream empty and its sequence numbers restart from 000000000000000000001. An event source mapping, however, persists its shard checkpoint (shardSequenceNumbers) on graceful shutdown. After a restart the poller resumed from that stale checkpoint with an AFTER_SEQUENCE_NUMBER iterator, which points past every record in the new stream epoch — so every freshly written record was silently skipped (no invoke, no error, no log) until the new sequence numbers climbed back above the stale value.

Discard persisted shard checkpoints for DynamoDB Streams ESMs at startup, so the poller resumes from TRIM_HORIZON and delivers records from the new epoch, matching the volatility of the stream itself. The invalidation is now logged instead of failing silently.

Fixes #2076.

Summary

Type of change

  • Bug fix (fix:)
  • New feature (feat:)
  • Breaking change (feat!: or fix!:)
  • Docs / chore

AWS Compatibility

Checklist

  • ./mvnw test passes locally
  • New or updated integration test added
  • Commit messages follow Conventional Commits

@aniketshukla1
fix(lambda): reset volatile DynamoDB Streams ESM checkpoints on restart
0facfd6 
A DynamoDB stream and its sequence numbers live only in memory (the record
buffer and the AtomicLong counter in DynamoDbStreamService are never
persisted), so a restart recreates the stream empty and its sequence numbers
restart from 000000000000000000001. An event source mapping, however, persists
its shard checkpoint (shardSequenceNumbers) on graceful shutdown. After a
restart the poller resumed from that stale checkpoint with an
AFTER_SEQUENCE_NUMBER iterator, which points past every record in the new
stream epoch — so every freshly written record was silently skipped (no
invoke, no error, no log) until the new sequence numbers climbed back above
the stale value.

Discard persisted shard checkpoints for DynamoDB Streams ESMs at startup, so
the poller resumes from TRIM_HORIZON and delivers records from the new epoch,
matching the volatility of the stream itself. The invalidation is now logged
instead of failing silently.

Fixes floci-io#2076.
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

🎉 Thanks for your first pull request to Floci!

Your CI checks need a maintainer to approve them before they run. That is GitHub's standard gate on first-time contributors, not a problem with your PR — so if the checks look like they are doing nothing, that is why. Once a maintainer approves, CI and the compatibility suite start automatically. Nothing is needed from you in the meantime.

While you wait, a couple of things that make review faster:

  • Link the issue this fixes with Closes #N in the description
  • Commits follow Conventional Commits (feat(s3): ..., fix(dynamodb): ...)
  • Behaviour changes come with a test — see CONTRIBUTING.md

Come join us in Slack — it is the fastest way to reach maintainers if you get stuck, or want feedback on an approach before investing more time in it.

@github-actions github-actions Bot added the over-pr-limit Author has more than 5 open PRs — advisory only label Aug 1, 2026
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

👋 Thanks for the pull request!

It looks like you now have 12 open pull requests in this repository, which is above our advisory limit of 5. To keep the review backlog focused and make sure every contribution gets proper attention, we ask contributors to shepherd their existing PRs to the finish line before opening new ones — responding to review feedback, rebasing, or closing any that are no longer needed.

Here is your list of open PRs. Nothing is blocked and this PR stays open — this is just a friendly nudge. The over-pr-limit label helps maintainers see at a glance which PRs to prioritize.

If some of these PRs are stacked or intentionally related, just say so here and a maintainer will take that into account.

@greptile-apps

greptile-apps Bot commented Aug 1, 2026

Copy link
Copy Markdown

Greptile Summary

The PR resets persisted DynamoDB Streams event-source mapping checkpoints during application startup so volatile, newly recreated streams resume from TRIM_HORIZON rather than skipping records behind stale sequence numbers.

  • Clears and persists stale shard checkpoints before starting enabled DynamoDB Streams pollers
  • Logs checkpoint invalidation and exposes pollAndInvoke at package scope for deterministic testing
  • Adds regression coverage for checkpoint removal and post-restart record delivery

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness or security issues identified.

Checkpoint invalidation runs only during the application startup lifecycle, persists through the established ESM storage pattern, and is covered by regression tests for both state reset and event delivery.

Important Files Changed
Filename Overview
src/main/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePoller.java Correctly aligns persisted ESM checkpoints with the in-memory lifecycle of DynamoDB stream records and sequence numbers.
src/test/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePollerTest.java Adds focused regression tests proving stale checkpoints are removed and records from the new stream epoch are delivered.

Reviews (1): Last reviewed commit: "fix(lambda): reset volatile DynamoDB Str..." | Re-trigger Greptile

@hectorvent hectorvent added bug Something isn't working lambda AWS Lambda dynamodb Amazon DynamoDB labels Aug 1, 2026
hectorvent
hectorvent approved these changes Aug 1, 2026
View reviewed changes

@hectorvent hectorvent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @aniketshukla1

@hectorvent
hectorvent merged commit 847c30a into floci-io:main Aug 1, 2026
18 checks passed
@hectorvent

hectorvent commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

🎉 This PR is included in version 1.6.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

abanna pushed a commit to abanna/floci that referenced this pull request Aug 6, 2026
@semantic-release-bot
chore(release): 1.6.0
e0aab2e 
# [1.6.0](floci-io/floci@1.5.34...1.6.0) (2026-08-06)

### Bug Fixes

* **1790:** better populate describe alarm responses ([floci-io#1792](floci-io#1792)) ([fcb22be](floci-io@fcb22be))
* **apigateway:** fall through to less specific resources on method mi… ([floci-io#1630](floci-io#1630)) ([96c7aa6](floci-io@96c7aa6)), closes [floci-io#1547](floci-io#1547)
* **apigateway:** import authorizers and security requirements from OpenAPI ([floci-io#1798](floci-io#1798)) ([91e06eb](floci-io@91e06eb))
* **apigateway:** use last duplicate header value ([floci-io#1806](floci-io#1806)) ([6a214e8](floci-io@6a214e8))
* **athena:** report the result CSV object key as OutputLocation ([floci-io#1895](floci-io#1895)) ([418ec1b](floci-io@418ec1b)), closes [floci-io#1752](floci-io#1752)
* **cloudformation:** AWS::IAM::ManagedPolicy exposes PolicyArn for Fn::GetAtt ([floci-io#2056](floci-io#2056)) ([f719033](floci-io@f719033))
* **cloudformation:** keep uniqueness suffix when truncating generated resource names ([floci-io#1802](floci-io#1802)) ([48b2139](floci-io@48b2139)), closes [floci-io#1825](floci-io#1825)
* **cloudformation:** preserve subnet public IP setting ([floci-io#2031](floci-io#2031)) ([b62d1e7](floci-io@b62d1e7))
* **cloudformation:** provision AWS::ApiGatewayV2::Authorizer and wire Route.AuthorizerId ([floci-io#1760](floci-io#1760)) ([7f51c74](floci-io@7f51c74)), closes [floci-io#1773](floci-io#1773)
* **cloudformation:** resolve Fn::GetAtt [Vpc, DefaultSecurityGroup] ([floci-io#1977](floci-io#1977)) ([bdad066](floci-io@bdad066)), closes [floci-io#1976](floci-io#1976)
* **cloudformation:** treat delete of an already-removed DynamoDB table or Lambda function as idempotent ([floci-io#1803](floci-io#1803)) ([5cb26ff](floci-io@5cb26ff))
* **codebuild:** stabilize start and retry responses ([floci-io#2052](floci-io#2052)) ([fa96b79](floci-io@fa96b79))
* **cognito:** reject unconfirmed users in SRP auth ([floci-io#2027](floci-io#2027)) ([733c72d](floci-io@733c72d))
* **dockerfile:** ensure docker-java compatibility ([floci-io#2096](floci-io#2096)) ([7ade290](floci-io@7ade290))
* **dynamodb:** resolve nested document paths in ADD and DELETE update actions ([floci-io#1908](floci-io#1908)) ([c052259](floci-io@c052259))
* **ec2:** DescribeKeyPairs returns InvalidKeyPair.NotFound for missing names/ids ([floci-io#1932](floci-io#1932)) ([7d96106](floci-io@7d96106)), closes [floci-io#1911](floci-io#1911)
* **ec2:** return AWS-parity MissingParameter for CreateSubnet without VpcId ([floci-io#2094](floci-io#2094)) ([88408d8](floci-io@88408d8)), closes [floci-io#2089](floci-io#2089)
* **ecs:** resolve Secrets Manager JSON-key selectors in task definition secrets ([floci-io#2133](floci-io#2133)) ([e3c180b](floci-io@e3c180b)), closes [floci-io#1912](floci-io#1912)
* **lambda:** carry the owning account into published version snapshots ([floci-io#2041](floci-io#2041)) ([f4e8bd8](floci-io@f4e8bd8)), closes [floci-io#2040](floci-io#2040)
* **lambda:** report resolved executed version ([floci-io#2026](floci-io#2026)) ([2889fbd](floci-io@2889fbd))
* **lambda:** reset volatile DynamoDB Streams ESM checkpoints on restart ([floci-io#2077](floci-io#2077)) ([847c30a](floci-io@847c30a))
* **pipes:** register kafka-clients reflection metadata for native image ([floci-io#2068](floci-io#2068)) ([ab73379](floci-io@ab73379))
* **s3:** accept a percent-encoded bucket/key separator in copy sources ([floci-io#2060](floci-io#2060)) ([a8ed218](floci-io@a8ed218)), closes [floci-io#2038](floci-io#2038)
* **s3:** apply CreateBucketConfiguration tags on bucket creation ([floci-io#2115](floci-io#2115)) ([79031a2](floci-io@79031a2))
* **s3:** include EventBridgeConfiguration in GetBucketNotification response ([floci-io#2072](floci-io#2072)) ([b8edd37](floci-io@b8edd37))
* **s3:** recognize virtual-hosted-style requests over HTTP/2 ([floci-io#1954](floci-io#1954)) ([5f7d4d2](floci-io@5f7d4d2)), closes [floci-io#1866](floci-io#1866)
* **ssm:** report invalid batch parameter names ([floci-io#2075](floci-io#2075)) ([04873bf](floci-io@04873bf))
* **tls:** reuse the persisted self-signed certificate across restarts ([floci-io#1799](floci-io#1799)) ([a4356f2](floci-io@a4356f2))

### Features

* Add AWSCloudFormationReadOnlyAccess as a managed policy. ([floci-io#2057](floci-io#2057)) ([0142dc4](floci-io@0142dc4))
* **apigateway:** support floci:override-id for v1 and v2 ([floci-io#2045](floci-io#2045)) ([02385da](floci-io@02385da)), closes [floci-io#1593](floci-io#1593)
* **bedrock:** add proxy backend for real LLM responses via OpenAI-compatible API ([floci-io#1789](floci-io#1789)) ([875e0f2](floci-io@875e0f2))
* **cloudcontrol:** include EC2 resource tags ([floci-io#1933](floci-io#1933)) ([205a8f1](floci-io@205a8f1))
* **cloudformation:** provision AWS::EC2::LaunchTemplate ([floci-io#1973](floci-io#1973)) ([4e0a815](floci-io@4e0a815)), closes [floci-io#1971](floci-io#1971)
* **cloudformation:** provision AWS::EC2::VPCGatewayAttachment ([floci-io#1972](floci-io#1972)) ([1a99721](floci-io@1a99721)), closes [floci-io#1970](floci-io#1970)
* **cloudformation:** provision AWS::SecretsManager::SecretTargetAttachment ([floci-io#1804](floci-io#1804)) ([62a576c](floci-io@62a576c))
* **cloudformation:** support AWS::Events::EventBus and EventBusPolicy ([floci-io#1794](floci-io#1794)) ([76f602f](floci-io@76f602f))
* **cognito:** implement ResendConfirmationCode ([floci-io#2071](floci-io#2071)) ([bb7c45c](floci-io@bb7c45c)), closes [floci-io#2067](floci-io#2067)
* **ec2:** added attach and detach volume support ([floci-io#1787](floci-io#1787)) ([2c74329](floci-io@2c74329))
* **ec2:** return an empty set for DescribeVpnGateways ([floci-io#1975](floci-io#1975)) ([3baf4f4](floci-io@3baf4f4)), closes [floci-io#1974](floci-io#1974)
* **iam:** list entities attached to managed policies ([floci-io#1808](floci-io#1808)) ([78ba5b3](floci-io@78ba5b3))
* **iam:** seed Amazon Bedrock managed policies ([floci-io#2034](floci-io#2034)) ([5874348](floci-io@5874348)), closes [floci-io#1559](floci-io#1559) [floci-io#1216](floci-io#1216)
* **iam:** seed the managed policies CDK bootstrap and the scenario stacks attach ([floci-io#2064](floci-io#2064)) ([43aea09](floci-io@43aea09)), closes [floci-io#2059](floci-io#2059) [floci-io#2057](floci-io#2057) [floci-io#2057](floci-io#2057)
* **kms:** implement ListKeyPolicies ([floci-io#2046](floci-io#2046)) ([760115d](floci-io@760115d)), closes [floci-io#1528](floci-io#1528)
* **lambda:** implement the Lambda Extensions API ([floci-io#1773](floci-io#1773)) ([9f0dec9](floci-io@9f0dec9))
* **lambda:** support extra /etc/hosts entries on Lambda launch ([floci-io#2073](floci-io#2073)) ([b543aa4](floci-io@b543aa4))
* **mwaa:** add Amazon MWAA emulation backed by real Airflow (LocalExecutor) ([floci-io#2086](floci-io#2086)) ([7e4e3e8](floci-io@7e4e3e8))
* **release:** one-button release cut from main and ECR Public versioned publishing ([floci-io#2127](floci-io#2127)) ([61011c0](floci-io@61011c0))
* **rum:** add CloudWatch RUM app-monitor service ([floci-io#1797](floci-io#1797)) ([aadc93e](floci-io@aadc93e))
* **sqs:** retain MessageGroupId on standard queue messages ([floci-io#1891](floci-io#1891)) ([a01b707](floci-io@a01b707)), closes [floci-io#1496](floci-io#1496)

### Performance Improvements

* **docker:** stop duplicating the native binary into a second image layer ([floci-io#2069](floci-io#2069)) ([9e90e5c](floci-io@9e90e5c))
@chenrui333 chenrui333 mentioned this pull request Aug 6, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hectorvent hectorvent hectorvent approved these changes

Assignees

No one assigned

Labels

bug Something isn't working dynamodb Amazon DynamoDB lambda AWS Lambda over-pr-limit Author has more than 5 open PRs — advisory only released

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] Persisted DynamoDB Streams ESM checkpoints outlive volatile stream sequence numbers, silently dropping records after restart

2 participants

@aniketshukla1 @hectorvent