This comes with yet another merge failure, @ethomson I'm going to leave this in your capable hands.

I know I've long since given up much right to comment on such things, but pulling the entire typedef of the fs and workdir iterators into iterator.h and then accessing the internal structure just to pull out a single field doesn't seem like the right design to me. There is precedent for accessor APIs that only work with a particular type of iterator (such as git_iterator_current_workdir_path) that would be less invasive, I think.

It did seem ugly, but at the time I was trying to get it to work. Since we do have precedent, I'll look into having git_iterator_index() which should also make the diff quite a bit more elegant.

The history needs some love, but this should behave the same way as git. The check for racily clean entries does a diff for each candidate. It may be faster to collect them all and do a single diff, but having a large amount of racily-clean entries is rather an edge case.

I'll take a look at that merge test shortly. I'm surprised it's so brittle.

In the meantime, we'll probably need a p_sleep type of creation on Windows, since it only offers Sleep(millis).

I thought we already had it, but it's the status examples which has this define. I think I'd rather have us set the timestamps manually long-term, rather than relying on a second of sleep, though.

Tracing a bit what differently_filtered is doing, it seems to be running afoul of the fixed racy protections done here. When we write out the index, we check for racy timestamps, which we in fact set the index to have. In this case, we check whether there is a difference between the contents of what we have in the entry and what we have in the workdir. We do have a difference here, so the entry gets smudged in order to avoid the next status/diff to believe that the file is unchanged.

I've changed that test such that we mess with the timestamps so much that we don't detect a change, but I don't believe that this represents a realistic situation anymore. Even if the first change is sensible in order to simulate an implementation which performs a different form of filtering, the second one is stupid, because it's timing-dependent. If we were to see the situation, we would detect it as a difference, because the entry/file/index timestamps would all match, and we need to protect ourselves from someone doing

git add foo
echo lolchange >foo
git status || git merge somebranch || ...

which makes us have to look at the change.

Using nanosecs would remediate this a bit on systems where we can trust it, but that's asking too much, I think, as git on Windows does not use this, and we can't use this on OS X at all IIRC.

Which is to say that by hacking the test so much that we pass, it feels like we're mocking the situation so much that we're not really testing anything anymore.

I've added extra checks so we avoid doing a diff and smudge of files for which the stat data is different, though I wonder if we have something in diff which would tell us already whether we can detect that a file is different just by looking at the stat.

If we were to see the situation, we would detect it as a difference,

Well, we should detect it as a difference if the file was written at the same time as the index. And we shouldn't detect is as a different if the file was not written at the same time. You can certainly envision a case where checkout takes > 1 second, and some files are not eligible for being rescanned due to raciness, and some (those written in the same second as the index) are.

The goal of this test is really to ensure that merge treats the files as clean/dirty identically to status, to avoid regressions there.

The scenario is that one user has core.autocrlf=false, makes some change on Windows, another user with core.autocrlf=true checks that file out. Now - if that file was not written at the same time as the index, the racy protections don't kick in and the file is clean. (Prior to the raciness fixes, this was always clean.)

At this point, our status and git status will both show this file as clean, but the merge prior to c0b10c2 would show the file as dirty, as it did not correctly take the index into account when looking at the workdir contents.

This test was really to ensure that we never differed in calculating cleanness in merge vs status.

It's possible that given the additional complexities that raciness throws into the mix that it is no longer viable to maintain this test and - hey - we probably have it right now! But this is bug was exceptionally costly for us, both in terms of support and in terms of mindshare. (See, eg http://stackoverflow.com/questions/27458236/visual-studio-2013-does-not-offer-to-do-merge-on-git-pull though there are hundreds of questions addressing this topic.)

So if it is viable, I would certainly like to keep this test.

What the test is proving right now is that if the index' timestamp is ahead of the files, we won't look at the contents before deciding that there's no difference. This should be the same as status does, but we still have the problem that this is timing-dependent. If git happens to update the index in some operation after the checkout, then we'd do this, but even if an operation takes over a second, some of the files will have the same timestamp as the index and if we have a different autocrlf setting from the installation which wrote the files, then we'd still need to figure out if it changed just before or after the index, so we'd have to look at the contents, so we might still fail to merge.

I'm beyond tired of arguing about this. Just delete the test and I'll put some unit tests in VS instead.

This is looking pretty good to me; how are you feeling about it?

I think that "index: don't smudge entries with different stat" is completely redundant, as diff does already take this into account, and I'm just re-implementing the logic. That needs to go.

It turns out that that commit wasn't quite redundant, but it's an optimisation for an edge-case which I believe we can solve much better than by having another copy of the comparison functions. I'll open a PR with a test to show a situation in which we don't need to smudge the entry but currently do (as it would be a failing test and we don't really have a way to express that's expected in clar).

I just remembered another edge case (smuding a file and then quickly making that file empty, I don't think we handle that) which I'll push up after breakfast.

I'm feeling about it atm, that edge case is handled, and there's a test for it.