I think with this new commit we'll behave correctly everywhere but on Windows (USE_NSEC is simply unavailable on Windows). For Windows, I believe @carlosmn is correct that we'll need some sort of compat layer which lets us get at Windows' nanosecond-precision m/ctime fields cleanly.

@ethomson Glancing at the MSDN docs, it seems that struct __stat64 contains a 64-bit time_t, but that value is still only in seconds, not nanoseconds. My Google skills are failing me, and I can't seem to find any MSDN documentation on how to get a file's m/ctime in nanoseconds. If you point me in the right direction, I'll happy add a compat layer so this Just Works on Windows as well.

Yes, I must have been making that up. More importantly, we don't even call the win32 stat, we call GetFileAttributes and convert all the data ourselves. I think I commented on that before sufficient coffee.

I had been thinking that this was basically a matter of not ifdefing Windows out. But obviously I was mistaken. Given that, I definitely don't have any objections to waiting to deal with Windows until such time as Git for Windows starts doing this also (or until somebody actually cares about libgit2 doing this on Windows).

So anyway, you're welcome to dig in (see git_win32__file_attribute_to_stat in w32_util.h) if you want, but dealing with struct stat on Windows is not an enviable task.

@CmdrMoozy it seems that NTFS doesn't record nano-seconds.

A file time is a 64-bit value that represents the number of 100-nanosecond intervals that have elapsed since 12:00 A.M. January 1, 1601 Coordinated Universal Time (UTC). The system records file times when applications create, access, and write to files.

Mike Kelly has a rather old blog post about it on MSDN for reference.
http://blogs.msdn.com/b/mikekelly/archive/2009/01/17/unix-time-and-windows-time.aspx

Thanks, @ethomson and @whoisj!

I'm thinking that the cleanest thing to do to support sub-second time resolution (looks like 100ns is the best we'll get) on Windows is to define our own struct timespec and a struct p_stat structure in win32/posix.h (and typedef it to say GIT_P_STAT), which includes the st_{a,m,c}tim.tv_nsec fields we expect to find on UNIX systems with a semi-modern glibc. For UNIX, we merely typedef struct stat GIT_P_STAT, since that structure already contains the fields we want (or USE_NSEC is unavailable).

This is a bit hacky, but it does keep the hackiness and OS-conditionals confined to posix.h and CMake. Unfortunately, this would mean replacing instances if struct stat with GIT_P_STAT - a quick grep shows something like 100 +/- 50 instances of this.

Does this sound like a reasonable approach?

I think this is reasonable and this is roughly how I would approach it.

However, I wonder if we might be able to take advantage of the fact that struct stat isn't defined by default on Win32 (see https://github.com/libgit2/libgit2/blob/master/src/win32/mingw-compat.h#L14 and https://github.com/libgit2/libgit2/blob/master/src/win32/msvc-compat.h#L13) and just declare our own structure to keep struct stat and avoid messing around with typedefs.

just declare our own structure to keep struct stat and avoid messing around with typedefs.

Thats exactly what I did for Git for Windows (see kblees/git@1fd19e8). The only problem is that you need to re-implement stat, lstat and fstat from scratch, but you probably have your own implementations for these anyway.

Yep, we already have our own stat-like implementations for win32, we'd just have to add the nanoseconds to our struct stat and fill them in for Windows.

One thing I'd be worried about implementing this on Windows is that Git for Windows does not fill these fields so if we do, we might never detect racy entries.

@carlosmn If msysgit can fill in the fields, then I think we're fine at detecting racily clean files. Anyway, I don't think it's bad to support it, with the caveat that if you're recompiling to change this option you ought to have some idea of what you're doing.

I think these commits more or less do the job - although I don't have a Windows machine on hand to build, so I'll have to wait for AppVeyor.

Rebased onto current master, to resolve merge conflicts.

Is any additional work needed on this fix to get it merged? I believe this pull incorporates everything discussed so far, and is ready to go.

The index also needs to know whether to consider the nanos in order to consider some entries racily-clean or not. Otherwise you're going to end up with many entries with zero nanos and then use that to compare against file info with filled nanos.

Sorry it took me so long to get this updated. I switched jobs, so I had to get my new employer's "ok" to continue working on this. I believe this pull is now ready to go, including the changes to account for racily clean file detection in the index code. The Travis failure seems to be due to some sort of networking problem on their OS X build servers. Let me know if more work is needed on this. Thanks!

Sorry for the delay in reviewing this. Two questions:

1. Why is this a compile-time option and not a run-time option? (I realize that some systems may lack nsecs and thus we must switch at compile time.)
2. Why is it compiled in by default, if the platform supports it? Does this match a git setting, or is it optimistic?

These questions may be due to a misunderstanding in how git is computing differences for nsecs. My git (2.5.3, Windows) is not storing nsecs in the index. Would this not mean that my libgit2 would always see an index that git had written as dirty (mismatched nsecs)?

Why is it compiled in by default, if the platform supports it?

As I understand these patches, the new USE_NSEC compile time option is disabled by default (as in upstream git).

My git (2.5.3, Windows) is not storing nsecs in the index.

Git for Windows 2.6.0 does store nsecs, but does not use them for comparisons (yet) [1].

Would this not mean that my libgit2 would always see an index that git had written as dirty (mismatched nsecs)?

Yes. I'm currently working on solutions for the same problem in upstream git [2].

[1] git-for-windows/git#443
[2] http://thread.gmane.org/gmane.comp.version-control.git/278683

No worries about the delay, thanks for taking the time to look at this. I'm pushed up some extra commits which resolve a merge conflict, and fix one spot where I was (incorrectly) not checking GIT_USE_NSEC I noticed when looking at this again myself.

Why is this a compile-time option and not a run-time option? (I realize that some systems may lack nsecs and thus we must switch at compile time.)

I think the main argument for making it compile-time only is that switching the option back-and-forth at runtime makes it much more likely users will run into the various weird situations @kblees mentioned. Is there a real use-case where we would want to be able to toggle the option at runtime? I think ideally, in the future, all code interacting with Git repositories (Git itself, libgit2, and so on) will all deal with nanoseconds in some common way on all platforms, and we can get rid of this compile-time check.

Why is it compiled in by default, if the platform supports it? Does this match a git setting, or is it optimistic?

The option is only available at all on platforms which support nanosecond-resolution file times, and on those platforms it is off by default. I think what we want to do (and this is what this patch does, currently) in libgit2, if the option is disabled, is:

• Never use nanosecond values for comparisons.
• Never write any nanosecond values to the disk (e.g. in the index).
• Leave nanosecond values set to 0 for any in-memory data structures (e.g. when we stat a file).

Hopefully this addresses your questions. Let me know if I've missed anything.