Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(ingestion): make REST emitter batch max payload size configurable #14123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
pedro93 merged 2 commits into from
Jul 21, 2025
Merged

feat(ingestion): make REST emitter batch max payload size configurable #14123

pedro93 merged 2 commits into from
Jul 21, 2025

Conversation

@pedro93
Copy link
Collaborator

@pedro93 pedro93 commented Jul 17, 2025 •
edited
Loading

Summary

  • Make the REST emitter's maximum payload size configurable via environment variable
  • Add DATAHUB_REST_EMITTER_BATCH_MAX_PAYLOAD_BYTES environment variable to control batch size limits
  • Maintains backward compatibility with default value of 15MB

Changes

  • Modified INGEST_MAX_PAYLOAD_BYTES in rest_emitter.py to read from environment variable
  • Allows users to customize payload size limits based on their infrastructure constraints
  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Jul 17, 2025
@codecov
Copy link

codecov bot commented Jul 17, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Jul 17, 2025
@pedro93 pedro93 changed the title Update rest_emitter.py feat(ingestion): make REST emitter batch max payload size configurable Jul 17, 2025
sgomezvillamor
sgomezvillamor approved these changes Jul 17, 2025
View reviewed changes
Copy link
Contributor

@sgomezvillamor sgomezvillamor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Wondering if new env var should also be listed here https://docs.datahub.com/docs/deploy/environment-vars

@pedro93
Copy link
Collaborator Author

pedro93 commented Jul 17, 2025

LGTM Wondering if new env var should also be listed here https://docs.datahub.com/docs/deploy/environment-vars

Makes sense to add it there. I don't however see any rest emitter related env vars in there. This is not a component that we seem to be registering. Do we have any other place for those?

@datahub-cyborg datahub-cyborg bot added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Jul 17, 2025
@sgomezvillamor
Copy link
Contributor

sgomezvillamor commented Jul 17, 2025

Apparently we are not documenting any of those

https://github.com/search?q=repo%3Adatahub-project%2Fdatahub+os.getenv+path%3A%2F%5Emetadata-ingestion%5C%2Fsrc%5C%2Fdatahub%5C%2Femitter%5C%2F%2F&type=code

@pedro93 pedro93 merged commit 12db9aa into master Jul 21, 2025
69 of 70 checks passed
@pedro93 pedro93 deleted the feature/cus-5309 branch July 21, 2025 17:42
kartikey-visa pushed a commit to kartikey-visa/datahub that referenced this pull request Jul 23, 2025
@pedro93 @kartikey-visa
feat(ingestion): make REST emitter batch max payload size configurable (
2ca5cff 
datahub-project#14123)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sgomezvillamor sgomezvillamor sgomezvillamor approved these changes

@hsheth2 hsheth2 Awaiting requested review from hsheth2

Assignees

No one assigned

Labels

ingestion PR or Issue related to the ingestion of metadata pending-submitter-merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pedro93 @sgomezvillamor