Codestin Search App

xiangfu0 · 2026-03-03T06:46:23Z

Summary

Add global runtime switch pinot.md5.disabled (default false) via CommonConstants and new PinotMd5Mode.
Initialize PinotMd5Mode from node startup config in broker/controller/server/minion.
Block upsert/dedup MD5 hash function when MD5 is disabled, and add runtime fail-safe in HashUtils.hashPrimaryKey(...).
Keep SQL md5() behavior unchanged.
In ADLS FS, if enableChecksum=true while MD5 is globally disabled, fail fast with IllegalStateException.
In S3 config, reject useLegacyMd5Plugin=true when MD5 is globally disabled.
Add/extend tests for SPI, segment-local, S3, and ADLS coverage.

Testing

./mvnw -pl pinot-spi -Dtest=PinotMd5ModeTest test
./mvnw -pl pinot-segment-local -am -Dtest=TableConfigUtilsTest,HashUtilsTest -Dsurefire.failIfNoSpecifiedTests=false -Dcheckstyle.skip=true test
./mvnw -pl pinot-plugins/pinot-file-system/pinot-s3 -am -Dtest=S3ConfigTest -Dsurefire.failIfNoSpecifiedTests=false -Dcheckstyle.skip=true test
./mvnw -pl pinot-plugins/pinot-file-system/pinot-adls -am -Dtest=ADLSGen2PinotFSTest -Dsurefire.failIfNoSpecifiedTests=false -Dcheckstyle.skip=true test
./mvnw -pl pinot-broker,pinot-controller,pinot-server,pinot-minion -am -DskipTests -Dcheckstyle.skip=true compile

codecov-commenter · 2026-03-03T07:35:24Z

Codecov Report

❌ Patch coverage is 73.07692% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.23%. Comparing base (8076f80) to head (d4423a7).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
...he/pinot/segment/local/utils/TableConfigUtils.java	57.14%	0 Missing and 3 partials ⚠️
...ava/org/apache/pinot/minion/BaseMinionStarter.java	0.00%	2 Missing ⚠️
.../pinot/server/starter/helix/BaseServerStarter.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #17800   +/-   ##
=========================================
  Coverage     63.23%   63.23%           
- Complexity     1454     1456    +2     
=========================================
  Files          3185     3186    +1     
  Lines        191549   191551    +2     
  Branches      29301    29306    +5     
=========================================
+ Hits         121117   121127   +10     
+ Misses        60967    60951   -16     
- Partials       9465     9473    +8

Flag	Coverage Δ
custom-integration1	`100.00% <ø> (ø)`
integration	`100.00% <ø> (ø)`
integration1	`100.00% <ø> (ø)`
integration2	`0.00% <ø> (ø)`
java-11	`63.21% <73.07%> (+<0.01%)`	⬆️
java-21	`63.20% <73.07%> (+<0.01%)`	⬆️
temurin	`63.23% <73.07%> (+<0.01%)`	⬆️
unittests	`63.23% <73.07%> (+<0.01%)`	⬆️
unittests1	`55.60% <43.75%> (+<0.01%)`	⬆️
unittests2	`34.14% <65.38%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

Adds a global runtime switch to disable MD5-dependent code paths across Pinot, wiring it into node startup and enforcing the restriction in upsert/dedup hashing and filesystem plugins.

Changes:

Introduces pinot.md5.disabled (CommonConstants) and new global switch PinotMd5Mode, initialized from broker/controller/server/minion configs.
Enforces MD5 disabling for upsert/dedup table validation and in HashUtils.hashPrimaryKey(...) (runtime fail-safe).
Updates ADLS/S3 filesystem behavior to respect the global MD5 disable switch and adds targeted unit tests.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`pinot-spi/src/main/java/org/apache/pinot/spi/utils/PinotMd5Mode.java`	Adds global MD5 disable mode holder.
`pinot-spi/src/test/java/org/apache/pinot/spi/utils/PinotMd5ModeTest.java`	Tests setter/getter and system-property initialization behavior.
`pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java`	Adds `pinot.md5.disabled` config key constant.
`pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java`	Initializes `PinotMd5Mode` from server config at startup.
`pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/BaseBrokerStarter.java`	Initializes `PinotMd5Mode` from broker config at startup.
`pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java`	Initializes `PinotMd5Mode` from controller config at startup.
`pinot-minion/src/main/java/org/apache/pinot/minion/BaseMinionStarter.java`	Initializes `PinotMd5Mode` from minion config at startup.
`pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java`	Rejects MD5 hash function for upsert/dedup when MD5 is globally disabled.
`pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java`	Adds validation tests for upsert/dedup MD5 rejection when disabled.
`pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java`	Adds runtime guard to block MD5 hashing when disabled.
`pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/HashUtilsTest.java`	Adds test ensuring MD5 hashing throws when disabled.
`pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3Config.java`	Rejects `useLegacyMd5Plugin=true` when MD5 is globally disabled.
`pinot-plugins/pinot-file-system/pinot-s3/src/test/java/org/apache/pinot/plugin/filesystem/S3ConfigTest.java`	Adds test for legacy MD5 plugin rejection when disabled.
`pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/ADLSGen2PinotFS.java`	Forces checksum behavior off when MD5 is globally disabled; avoids computing file MD5 when disabled.
`pinot-plugins/pinot-file-system/pinot-adls/src/test/java/org/apache/pinot/plugin/filesystem/test/ADLSGen2PinotFSTest.java`	Adds test coverage for checksum being disabled when MD5 is disabled.

...file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/ADLSGen2PinotFS.java

...em/pinot-adls/src/test/java/org/apache/pinot/plugin/filesystem/test/ADLSGen2PinotFSTest.java

pinot-spi/src/main/java/org/apache/pinot/spi/utils/PinotMd5Mode.java

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

pinot-spi/src/main/java/org/apache/pinot/spi/utils/PinotMd5Mode.java

...file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/ADLSGen2PinotFS.java

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.

...inot-file-system/pinot-s3/src/test/java/org/apache/pinot/plugin/filesystem/S3ConfigTest.java

...em/pinot-adls/src/test/java/org/apache/pinot/plugin/filesystem/test/ADLSGen2PinotFSTest.java

...t-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java

pinot-spi/src/test/java/org/apache/pinot/spi/utils/PinotMd5ModeTest.java

...t-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java

pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/HashUtilsTest.java

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

...t-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java

...em/pinot-adls/src/test/java/org/apache/pinot/plugin/filesystem/test/ADLSGen2PinotFSTest.java

xiangfu0 requested review from Jackie-Jiang and Copilot March 3, 2026 07:38

xiangfu0 added Configuration Config changes (addition/deletion/change in behavior) security labels Mar 3, 2026

Copilot started reviewing on behalf of xiangfu0 March 3, 2026 07:39 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

xiangfu0 force-pushed the security/add-disable-md5-flag branch 2 times, most recently from 6ee97e6 to a8d6623 Compare March 3, 2026 08:08

xiangfu0 requested a review from Copilot March 3, 2026 17:49

Copilot started reviewing on behalf of xiangfu0 March 3, 2026 17:49 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

pinot-spi/src/main/java/org/apache/pinot/spi/utils/PinotMd5Mode.java Outdated Show resolved Hide resolved

...file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/ADLSGen2PinotFS.java Show resolved Hide resolved

Add global pinot.md5.disabled switch and enforce MD5 guards

d4423a7

xiangfu0 force-pushed the security/add-disable-md5-flag branch from a8d6623 to d4423a7 Compare March 3, 2026 17:59

xiangfu0 requested a review from Copilot March 3, 2026 18:00

Copilot started reviewing on behalf of xiangfu0 March 3, 2026 18:03 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

xiangfu0 requested a review from Copilot March 3, 2026 20:06

Copilot started reviewing on behalf of xiangfu0 March 3, 2026 20:07 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

...t-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java Show resolved Hide resolved

...em/pinot-adls/src/test/java/org/apache/pinot/plugin/filesystem/test/ADLSGen2PinotFSTest.java Show resolved Hide resolved

Jackie-Jiang approved these changes Mar 3, 2026

View reviewed changes

xiangfu0 merged commit 5df2ffb into apache:master Mar 3, 2026
24 checks passed

xiangfu0 deleted the security/add-disable-md5-flag branch March 3, 2026 23:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add global pinot.md5.disabled switch and enforce MD5 guards#17800

Add global pinot.md5.disabled switch and enforce MD5 guards#17800
xiangfu0 merged 1 commit intoapache:masterfrom
xiangfu0:security/add-disable-md5-flag

xiangfu0 commented Mar 3, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Mar 3, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

xiangfu0 commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

codecov-commenter commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xiangfu0 commented Mar 3, 2026 •

edited

Loading

codecov-commenter commented Mar 3, 2026 •

edited

Loading