Strongly in favor of (2). Symlinks are a really useful concept in typeshed.

How urgently do we need a fix/workaround for this?

Well, 0.501 was released March 2nd, so I think it's about time for the next mypy release. Then again, this isn't a regression. So if we don't have a solution we can live with it.

There's also the rather hacky option 3,

3. Teach mypy and pytype to treat files that only contain a path (and nothing else) as a link to the real file.

That should be easy enough for mypy. I would do this only on Windows to limit the risk of misfires.

@JukkaL, @ddfisher, thoughts?

That sounds acceptable if we only use it on Windows. The only issues I'm a little worried about are contributor confusion, since the link files probably can't have comments in them, and the ability to create symlinks on Windows. Code duplication is probably worse though. I think that having a Windows-specific section in the typeshed readme where we discuss this would be sufficient.

I'm opposed to 3. It's hacky, and I'd expect it to cause problems in the long run. 2 feels like the correct solution to me -- this is a build-time issue and should be fixed with a build-time fix.

Ok, it seems that we at least all agree that we want 2 or 3, and not 1. Do we want to do 3 (pathline hack) until one of us gets around to implementing 2 (git post-checkout hook)? Or leave this problem alone, for the time being?

I think if we implement 3 as a temporary measure it will live forever. Let's do the right thing and implement 2 -- it doesn't seem particularly hard.

it doesn't seem particularly hard.

Famous last words :-) My experience suggests that Windows-specific things tend to be harder than you'd expect.

After some more thought, I no longer like 3, but I'm also strongly opposed to 2 -- I don't like the idea of having Windows-specific workflows and Windows-specific scripts that potentially nobody in the core teams uses regularly, and that may be hard to get working on everybody's current and future Windows environments. I also generally don't like post-checkout scripts and other git magic, since when things go wrong, most people don't know how they work and don't expect them.

I have two additional proposals that wouldn't require any Windows-specific code and would avoid the problems with 1:

4. Use duplicate files but have a test that verifies that certain files always have the same contents. These files can be described in a simple text file so adding new "implicitly linked" files would be easy. The test would be trivial to implement and no changes are needed to type checkers.
5. Implement custom symlinks (e.g. foo.pyi.link) and update tools to look for them in addition to .pyi files. The files itself would just contain the link target path and perhaps an optional comment. Basically, we'd use 3 on every platform for consistency. Though fairly simple, this would cause extra work for everybody who wants to use typeshed and there are some corner cases that could cause incompatibility between tools.

I like (4). I agree with all Jukka's arguments.

I suggested (4) in #1173 before. I can implement it, hopefully this weekend.

Great!

Hmm. Supposedly recent versions of git support symlinks on Windows if you clone the repository with core.symlinks=true. Is that an option here?

If that works it would be brilliant. Does anyone know how to set that flag to true in AppVeyor? Though from the docs at https://git-scm.com/docs/git-config I cannot tell whether setting it to True on Windows would have the desired effect. Searching http://stackoverflow.com/questions/5917249/git-symlinks-in-windows for "core.symlinks" suggests that it works if you can run a recent git version in Administrator mode like this: git clone -c core.symlinks=true <URL>

I'll try it out.

I just submitted python/mypy#3213 to make Appveyor use symlinks correctly (using tips from appveyor/ci#650). Once that's merged I'll restore the *path.pyi symlinks in typeshed.

Reopening. We may have to roll back the fix and try one of the other solutions; the Windows development experience is pretty bad currently (you have to be Administrator for many git commands). See e.g. python/mypy#3222 (comment)

I like solution #4, since it will work without any git-specific tooling for anyone who isn't developing the few files affected by this.

Enum as it stands is not the same in 3.4/ as it is in the third_party/2/. But there are several symlinks in the 2/3 folders, that unless Im mistaken we can forward with from *path import *, which seems cleaner to me. SocketServer.pyi is a symlink to the Python 3 version, so that should probably go into 2and3. I think that covers all the symlinks, so all that needs to be checked is enum and enum34, if we want to copy enum to the third_party stubs. Does this plan make sense?

Note though that one of the symlinks is for the builtins in Python 2—I wouldn't be surprised if the import * solution doesn't work there.

A simple test indicates that the import * works for builtins, but I will of course test more thoroughly. I think the only stub that will be of issue is the enum, which is right now out of sync in the two folders.

I'm not sure I like the idea of making PY2 builtins.pyi textually different than __builtins__.pyi.

The out-of-sync-ness of enum.pyi in 2 and 3.4 is intentional AFAIK (there are comments IIRC), but enum in thirdparty/3 and stdlib/3.4 should match.

I'm not sure I like the idea of making PY2 builtins.pyi textually different than __builtins__.pyi

So from __builtins__ import * isn't sufficient? Could you explain what issue this would cause?

As for enum, I don't see any comments in the sources. It seems the third party stub (Is this supposed to be enum34's implementation?) has a unique file for thirdparty/2/ and a symlink to the stdlib/3.4/ file for the third_party/3/ file. If these are supposed to be enum34's backport, shouldn't they all be the same?

I can't find the comment about keeping enum.pyi for PY2 and PY3 separate either, it was probably in one of the PR comments. But it doesn't really matter; I just think it's not worth making them the same given the difference in metaclass syntax.

(And yes, those are implementations for enum34 -- that package actually installs a module named enum.)

Regarding builtins vs. __builtin__.py, I'm mostly just very concerned that mypy doesn't consistently use one over the other -- all I recall is that the PY2 solution is a bit hacky and the symlink was needed to make the hacks work. Note that at runtime there is no builtins.py in PY2. I'm not sure how one would prove that your approach doesn't break anything.

This issue means mypy cannot be installed locally with pip install . on Windows, since pip follow symlinks when copying files, and thus fails on permissions 😕

@JelleZijlstra would you perchance have an implementation of #4?

I never actually implemented it because we ended up tinkering with Windows settings to allow symlinks instead. I'm open to switching to solution (4) (use separate files with a test to confirm they stay identical) instead.

Yes I think 4 is the best option, while not ideal.