fix some problems with handling of empty lumis in repackmerge, fixes #4014 #4018
Conversation
The repackmerge job splitting algorithm enforces lumi order and only allows holes in the lumi order when there is no streamer files for that lumi. This is not the only coniditon for which we don't get RAW data though, it can also be that for that lumi no events were selected for the dataset in question. Add a check for that as well. Add a unittest to test this case. Also simplify the lumi hole detection algorithm a bit and apply the same simplification to the repack job splitting.
|
@dballesteros7 or @samircury , can one of you test this in a replay ? Unit tests all work and they now contain a test case for the issue in #4014, I just want to make sure nothing else gets broken by this. |
|
Sure, only problem is we don't have a machine for this. Is it okay if we deploy an agent in vocms229? |
|
I think that's ok. We should ask for more vm voboxes though. |
|
Yes, we'll make sure to raise the point in Monday's meeting. |
|
Replay worked. the setup was:
The only remark is that here we tested a slightly different case as the one described above. In this case we didn't have unmerged files in the first lumis not because of zero events lumis but because they mostly went straight to merge. Anyway I think this should be enough proof for the other case. Details on the elog: |
fix some problems with handling of empty lumis in repackmerge, fixes #4014
The repackmerge job splitting algorithm enforces lumi order and
only allows holes in the lumi order when there is no streamer
files for that lumi. This is not the only coniditon for which
we don't get RAW data though, it can also be that for that lumi
no events were selected for the dataset in question. Add a
check for that as well. Add a unittest to test this case.
Also simplify the lumi hole detection algorithm a bit and
apply the same simplification to the repack job splitting.