Add BRIGHT SPLADEv3 cached regressions and fix ONNX #2928
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lintool
approved these changes
Aug 19, 2025
Member
lintool
left a comment
There was a problem hiding this comment.
Minor nits - please fix averages, per section + overall.
| | Robotics | 0.1091 | 0.1578 | | ||
| | Stack Overflow | 0.1626 | 0.1290 | | ||
| | Sustainable Living | 0.1613 | 0.1497 | | ||
| | **StackExchange** average | **0.1705** | **0.1704** | |
Member
There was a problem hiding this comment.
The averages change also, right?
| queryTokens.add(CLS); | ||
| queryTokens.addAll(tokenizer.tokenize(query)); | ||
| if (queryTokens.size() > MAX_SEQ_LEN - 2) { | ||
| queryTokens = queryTokens.subList(0, MAX_SEQ_LEN - 2); |
Member
There was a problem hiding this comment.
in terms of code alignment, imo a ternary operator, i.e., x = y ? a : b; would look prettier, but don't feel too strongly.
Co-authored-by: Jimmy Lin <[email protected]>
…nto bright-splade
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2928 +/- ##
============================================
+ Coverage 71.82% 71.83% +0.01%
Complexity 1321 1321
============================================
Files 186 186
Lines 12280 12294 +14
Branches 1413 1414 +1
============================================
+ Hits 8820 8832 +12
- Misses 2932 2933 +1
- Partials 528 529 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Member
|
For the record, I reran all SPLADE-v3 conditions for BEIR, did not appear to have any score changes. Finding all the conditions: $ grep splade-v3 src/main/python/regressions-batch04.txt | grep beirRunning: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bug in Anserini's SPLADE encoder logic that showed up from BRIGHT's really long queries: queries longer than the max sequence length of 512 have the CLS ending token truncated, resulting in the model going haywire. This is fixed in this PR, but we should probably rerun everything SPLADE ONNX that might have queries that fell in this e.g. ArguAna.
Fixing this resulted in some updates to BRIGHT SPLADEv3 ONNX numbers, updated here. Also fixed a typo in the regression docs.
Added the entire regression pipeline for BRIGHT SPLADEv3 cached, including topic bindings.