Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@hufnagel
Copy link
Member

@hufnagel hufnagel commented Jan 9, 2013

There is a race condition between the check for available
files for repack merge, the check for completely repacked
lumis without data for the merge and the completion check
for split repacked lumis.

If a lumi has data, is split repacked and the split repack
lumi is marked complete between the check for available
files and the check for completely repacked lumis without
data, a merge job can be created that merges across this
lumi without its data. When the data is then found in the
next cycle, the system is broken because it cannot merge
data in order anymore.

I can't fix this by time ordering these calls as the
completion check for split repacked lumis runs in a
different component as the other two calls.

What I am going to do is to combine the check for empty
lumis (no streamers) and complete lumis that have
finished repacking and do it in a single query before
checking for available files. That's save because at
worst I will have incomplete information about the
holes in the available lumis we can merge across.
And that at worst leads to a repack merge job being
issued one polling cycle later than it could be.

Also relax the in order merge requierement to not
apply when the input fileset is closed, then just
merge everything we get.

There is a race condition between the check for available
files for repack merge, the check for completely repacked
lumis without data for the merge and the completion check
for split repacked lumis.

If a lumi has data, is split repacked and the split repack
lumi is marked complete between the check for available
files and the check for completely repacked lumis without
data, a merge job can be created that merges across this
lumi without its data. When the data is then found in the
next cycle, the system is broken because it cannot merge
data in order anymore.

I can't fix this by time ordering these calls as the
completion check for split repacked lumis runs in a
different component as the other two calls.

What I am going to do is to combine the check for empty
lumis (no streamers) and complete lumis that have
finished repacking and do it in a single query before
checking for available files. That's save because at
worst I will have incomplete information about the
holes in the available lumis we can merge across.
And that at worst leads to a repack merge job being
issued one polling cycle later than it could be.

Also relax the in order merge requierement to not
apply when the input fileset is closed, then just
merge everything we get.
hufnagel added a commit that referenced this pull request Jan 9, 2013
@hufnagel hufnagel merged commit b06e0be into dmwm:master Jan 9, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant