Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add support for Git LFS#110

Merged
dscho merged 1 commit into
git-for-windows:masterfrom
github:add-support-for-git-lfs
Apr 24, 2016
Merged

Add support for Git LFS#110
dscho merged 1 commit into
git-for-windows:masterfrom
github:add-support-for-git-lfs

Conversation

@shiftkey

@shiftkey shiftkey commented Apr 14, 2016

Copy link
Copy Markdown
Contributor

There's likely some improvements to do here, but opening this early for feedback:

  • package and test i686 package locally
  • package and test x86_64 package locally
  • confirm compressed file size is accurate
  • wireup equivalent git lfs install and git lfs uninstall steps to update config
  • understand the root cause of the issue with this file not being executable after install
brendanforster@SHIFTKEYDE8B MINGW32 /usr/src/build-extra/git-lfs (add-support-for-git-lfs)
$ /mingw32/bin/git-lfs.exe
bash: /mingw32/bin/git-lfs.exe: cannot execute binary file: Exec format error

brendanforster@SHIFTKEYDE8B MINGW32 /usr/src/build-extra/git-lfs (add-support-for-git-lfs)
$ file /mingw32/bin/git-lfs.exe
/mingw32/bin/git-lfs.exe: PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows
  • simplify packaging steps
  • update version to v1.2
  • correct installation to use new rules

cc @dscho as we discussed last week
cc @technoweenie just cause

@strich

strich commented Apr 14, 2016

Copy link
Copy Markdown

Hey @shiftkey. I'm one of the primary contributors for the LFS Windows installer.

I'm not sure how much has been discussed with @technoweenie, but just a quick history: When LFS first came out the Win installer made attempts at finding the Git-for-windows install and placing the git-lfs.exe exec there. It was a bit hit and miss and as part of upgrading the installer framework itself we decided to install LFS into its own folder under Program Files (By default - The user can choose), and the global PATH env var gets updated with the install location. There are other feature now such as being dual-arch and automatically initializing Git LFS for the first time.

Integrating Git LFS with Git-for-windows will help close the gap for non-technical users, so this is really exciting! Might I suggest that instead of directly packaging Git LFS execs in your installer, you instead package the Git LFS installer itself (Or better yet download the latest version at install-time) and run it silently. That way you gain the benefit of the features already in the LFS installer as well as allowing users to independently update/remove LFS outside of git-for-windows.

@shiftkey

Copy link
Copy Markdown
Contributor Author

@strich thanks for the feedback! I did mention this in passing to @technoweenie, and while I want to avoid duplicating the work of the Git LFS team there are some packaging constraints that make things interesting on the Git for Windows side.

I went with this approach as this makes packaging simpler for the Git for Windows team:

  • no need to run a secondary installer
  • this supports both PortableGit (bin-deployed) and machine-wide (requires elevation) installs
  • packaging infrastructure supports 32/64-bit packages

I've not thought about scenarios where you might have multiple versions of git-lfs.exe on your PATH but I'm not sure what I can do here.

There are other feature now such as being dual-arch and automatically initializing Git LFS for the first time.

Ah yes, I knew there was something that I'd missed. I'd need to ensure that Git LFS is initialized after installation - by invoking git lfs init or equivalent - and I think I need to do that as part of the publishing step - rather than here inside the package.

Might I suggest that instead of directly packaging Git LFS execs in your installer, you instead package the Git LFS installer itself (Or better yet download the latest version at install-time) and run it silently.

I'm not sure how this fits with the existing Git for Windows philosophy. I'll let @dscho speak to this but I think users will want this to be as standalone as possible.

That way you gain the benefit of the features already in the LFS installer as well as allowing users to independently update/remove LFS outside of git-for-windows.

A couple of questions I have:

  • Is managing Git LFS explicitly a common request? If we're publishing frequent updates for Git for Windows (there's been three in the last four weeks) and we're updating this Git LFS package whenever it has new releases, what other sorts of scenarios are we not covering here?
  • Are there uninstall steps that we'd need to consider if we're mimicking the Git LFS installer here? I know the ~/.gitconfig changes might need to be removed, but are there others?
[filter "lfs"]
    smudge = git-lfs smudge %f
    required = true
    clean = git-lfs clean %f

@strich

strich commented Apr 14, 2016

Copy link
Copy Markdown

Is managing Git LFS explicitly a common request?

Unlikely. Its more that since its currently early in its life-cycle there may be quite a few patch releases people might be required to often update for. However if your team release quite often yourselves then maybe its not an issue.

Are there uninstall steps that we'd need to consider if we're mimicking the Git LFS installer here?

As you guessed - Only the .gitconfig entries need to be removed, which you can do with git lfs uninstall prior to removing it.

Ah yes, I knew there was something that I'd missed. I'd need to ensure that Git LFS is initialized after installation...

Yep just run git lfs install.

I've not thought about scenarios where you might have multiple versions of git-lfs.exe on your PATH but I'm not sure what I can do here.

If you have git-lfs.exe next to git.exe, as you propose you'll do, then I believe Windows will first check there before accessing PATH. So in the event someone installs git-for-windows as well as Git LFS using our installer, the git-for-windows version of LFS should win.

You guys have a few options here as I see it:

  1. You can completely ignore the potential presence of an existing Git LFS installation and given the above your package should win and should work as the user expects.
  2. You can package LFS as you propose and also automatically detect and uninstall any existing LFS installation.
  3. You can package the LFS installer and run it silently, passing in your directory, and any other options you might wish (I could add any custom functionality you need).

I think I would prefer option 3 if only because it should come with the least amount of edge case issues the user might find themselves in. For example I'd worry that the user would install git-for-windows and then later Git LFS using our installer, either because they weren't aware it was in git-for-windows or because they wanted to update LFS. Such an edge case would not break LFS, but they might find it didn't update as expected.

I do understand you guys have your own ecosystem and requirements however. Happy to continue answering questions, etc.

@dscho

dscho commented Apr 14, 2016

Copy link
Copy Markdown
Member

Might I suggest that instead of directly packaging Git LFS execs in your installer, you instead package the Git LFS installer itself (Or better yet download the latest version at install-time) and run it silently.

I'm not sure how this fits with the existing Git for Windows philosophy. I'll let @dscho speak to this but I think users will want this to be as standalone as possible.

Yeah, unfortunately this is a no-go. The idea of Git for Windows (since 2.x) is to have the installer reflect a partial MSYS2 installation, with a MSYS2 setup being the primary development environment. Therefore, everything that gets packages should be bundled in MSYS2 packages.

Yep just run git lfs install.

What does this do? Just editing the etc/gitconfig file?

I am asking because I do not want to run git lfs install due to concerns about the portable Git.

If you have git-lfs.exe next to git.exe, as you propose you'll do, then I believe Windows will first check there before accessing PATH.

That is only correct for .dll files, not .exe files. Whatever comes first in your PATH will win.

I think I would prefer option 3

I do not think this would fly well with portable Git (which is what @shiftkey is mostly concerned, as he packages a modified version of portable Git as the command-line tools of GitHub for Windows). Even having to run a command to install something we would want to avoid: it is too easy to put absolute paths into the gitconfig that are then invalid when moving the portable Git to a new computer.

So I would prefer option 1.

As to releasing new Git for Windows versions when new Git LFS versions become available: we already release new Git for Windows versions when either Git or the Git Credential Manager releases new versions. I could easily add Git LFS to that list.

@strich

strich commented Apr 14, 2016

Copy link
Copy Markdown

Ah okay. Fair enough then. So the only edge case I can see here is if a user installed LFS before installing Git-for-windows, and thereby making LFS appear first in PATH. I could add a check in the LFS installer to tell the user to install git first if it cannot find it anywhere in that case.

What does this do? Just editing the etc/gitconfig file?
I am asking because I do not want to run git lfs install due to concerns about the portable Git.

I believe so yes. @technoweenie would better be able to tell you exactly what changes you need to replicate manually in place of git lfs install. Though I do not think it adds any kind of paths, absolute or otherwise, so it would be better to use git lfs install so you don't have to worry about any future changes to the way that works.

Comment thread git-lfs/PKGBUILD Outdated
pkgver=1.1.2
pkgrel=1
pkgdesc="An open source Git extension for versioning large files"
arch=('i686' 'x86_64')

This comment was marked as off-topic.

This comment was marked as off-topic.

@dscho

dscho commented Apr 14, 2016

Copy link
Copy Markdown
Member

wireup git lfs init steps as part of post-install work

This could be done in a post-install step. See https://github.com/git-for-windows/MINGW-packages/blob/master/mingw-w64-qt5-git/qt5.install and https://github.com/git-for-windows/MINGW-packages/blob/master/mingw-w64-qt5-git/PKGBUILD#L11 for an example.

@dscho

dscho commented Apr 14, 2016

Copy link
Copy Markdown
Member

understand the root cause of the issue with this file not being executable after install

I believe that the options=('!strip') might help with that.

@shiftkey

Copy link
Copy Markdown
Contributor Author

I'll look at the post-install stuff next and putting that in /etc/gitconfig, but that feedback helped out a lot ✨

@shiftkey

shiftkey commented Apr 14, 2016

Copy link
Copy Markdown
Contributor Author

One minor issue:

$ git-lfs install --help
git lfs install [options]

Perform the following actions to ensure that Git LFS is setup properly:

  • Set up the clean and smudge filters under the name "lfs" in the global Git
    config.
  • Install a pre-push hook to run git lfs pre-push for the current repository,
    if run from inside one.

...

@strich would it be possible for us to run this against the system config? Running this at package time means it won't be available to users when they install...

@strich

strich commented Apr 14, 2016

Copy link
Copy Markdown

That was recently discussed here: git-lfs/git-lfs#799 (comment).

Basically - There is currently no option to write the smudge config to the system Git config via git lfs install. Though most would agree it should by default.

However as @technoweenie points out in the link above, the config above doesn't affect any repos that aren't managed with LFS so it shouldn't be an issue beyond the principle of it.

@technoweenie

Copy link
Copy Markdown

I'm cool with changing it, but I imagine we'd have to worry about system privs. I'm going to add --system to the roadmap: git-lfs/git-lfs#1145

@dscho

dscho commented Apr 15, 2016

Copy link
Copy Markdown
Member

Total Installed Size: 11.73 MiB

Is that accurate? Is the git-lfs.exe really that large?

@technoweenie

Copy link
Copy Markdown

Yup. Go binaries are statically linked with the entire Go runtime.

@shiftkey shiftkey changed the title [WIP] Add support for Git LFS Add support for Git LFS Apr 16, 2016
@shiftkey

Copy link
Copy Markdown
Contributor Author

Added the --system version of git lfs install/uninstall to the package.

Ready for review 🤘

@shiftkey shiftkey force-pushed the add-support-for-git-lfs branch from 03c4b0d to 964e48b Compare April 16, 2016 07:18
Comment thread git-lfs/git-lfs.install Outdated
if [ -f "${arch}/bin/git-lfs.exe" ]; then
git config --system filter.lfs.clean "git-lfs clean %f"
git config --system filter.lfs.smudge "git-lfs smudge %f"
git config --system filter.lfs.required true

This comment was marked as off-topic.

This comment was marked as off-topic.

@shiftkey

Copy link
Copy Markdown
Contributor Author

🍎

@dscho

dscho commented Apr 18, 2016

Copy link
Copy Markdown
Member

Yup. Go binaries are statically linked with the entire Go runtime.

Urgh. So we would increase the size of the installer by a whopping 35% just to include Git LFS? Hrm. Isn't it possible to strip this a bit? Or is at least the compressed size not that horrible?

@technoweenie

technoweenie commented Apr 18, 2016

Copy link
Copy Markdown

Funny you should ask! Shrink your Go binaries with this one weird trick was posted yesterday.

TLDR:

I don't want to do anything that increases startup time (such as upx compression), since the default configuration using git clean/smudge filters can call the git-lfs binary many times on large repositories. But, we can probably live without debugging.

Good news is that it does compress really well. Check out the zip file releases. The Windows Installer is only 4.78MB, while including both the 32bit and 64bit versions in an Inno Setup installer.

@dscho

dscho commented Apr 18, 2016

Copy link
Copy Markdown
Member

Okay. I still think it is quite concerning, though, if the Git for Windows installer would increase by more than 10% just for LFS support (which not many people need, quite honestly).

Comment thread git-lfs/git-lfs.install Outdated
post_install() {
for arch in mingw32 mingw64; do
if [ -f "${arch}/bin/git-lfs.exe" ]; then
git config --system filter.lfs.clean "git-lfs clean -- %f"

This comment was marked as off-topic.

@shiftkey

Copy link
Copy Markdown
Contributor Author

I'll leave the packaging size discussion for everyone else to debate vigorously. I'm curious why we're not seeing those same compression benefits here @dscho - the compressed packages (which are then installed into the Git SDK environment) are only slightly larger than the upstream Git LFS dual installer:

@dscho

dscho commented Apr 22, 2016

Copy link
Copy Markdown
Member

Hey @shiftkey, I'd like to merge this Pull Request (even if I would prefer to build git lfs ourselves so we can squeeze the binaries, but MSYS2's go package does not work correctly yet). To this end, may I ask you to rebase it interactively, add your Sign-off's to all of the commits, and squash as appropriate?

Signed-off-by: Brendan Forster <[email protected]>
@shiftkey shiftkey force-pushed the add-support-for-git-lfs branch from c4d0730 to 2b85cda Compare April 22, 2016 23:20
@shiftkey

Copy link
Copy Markdown
Contributor Author

@dscho squashed, rebased and signed off

@dscho dscho merged commit 379253c into git-for-windows:master Apr 24, 2016
@dscho

dscho commented Apr 24, 2016

Copy link
Copy Markdown
Member

Perfect, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants