fix problems with repackmerge #4023
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a race condition between the check for available
files for repack merge, the check for completely repacked
lumis without data for the merge and the completion check
for split repacked lumis.
If a lumi has data, is split repacked and the split repack
lumi is marked complete between the check for available
files and the check for completely repacked lumis without
data, a merge job can be created that merges across this
lumi without its data. When the data is then found in the
next cycle, the system is broken because it cannot merge
data in order anymore.
I can't fix this by time ordering these calls as the
completion check for split repacked lumis runs in a
different component as the other two calls.
What I am going to do is to combine the check for empty
lumis (no streamers) and complete lumis that have
finished repacking and do it in a single query before
checking for available files. That's save because at
worst I will have incomplete information about the
holes in the available lumis we can merge across.
And that at worst leads to a repack merge job being
issued one polling cycle later than it could be.
Also relax the in order merge requierement to not
apply when the input fileset is closed, then just
merge everything we get.