Reproduction steps

Create a repo on an NTFS or Fat32 drive and try to checkout different branches from either MacOS or Linux.

Expected behavior

Checkout works fine

Actual behavior

Checkout fails due to potential filemode issues

Version of libgit2 (release number or SHA1)

Any version after this PR was merged

Operating system(s) tested

MacOSx
Ubuntu 16.04

This works fine on Windows since Windows does not care about filemode, but after said PR was merged there seem to be issues when the target drive does not care about filemodes on an OS that does.

I tracked it down finally and it turns out we get in a weird state where we have an added file in our current branch that is not in the branch I want to check out, and when the checkout code tries to remove this file it finds that the mtime and mode on the workdir version of the file do not match that of the index. This causes the code to check the mode against the baseitem->mode which always fails. Since we are doing a safe checkout the checkout says there is a conflict that needs to be resolved and fails.

Here is where it fails

Not sure what exactly causes the state or how to properly rectify it yet. Attached is a repo that can reproduce the issue. Just unzip it and checkout any of the other branches with libgit2. I tried to diagnose this myself but I ran out of time and haven't had another chance since I originally found the issue. Ed told me to open an issue here so maybe another contributor could have a crack at it if they have time.

dummyrepo.zip