Describe the bug
Git lfs version 3.6.1 does not pull all the files that are part of the lfs management. When testing with version 3.5.1 and 3.3.0 all the files are correctly created.
To Reproduce
Steps to reproduce the behavior:
- git lfs install
- git clone https://huggingface.co/nreimers/mmarco-mMiniLMv2-L12-H384-v1
- cd mmarco-mMiniLMv2-L12-H384-v1
- git lfs pull
- see that tokenizer.json is not pulled - only reference to file is in the repository
I was unable to convince git lfs version 3.6.1 to checkout tokenizer.json file.
Expected behavior
All the files including tokenizer.json should be downloaded.
System environment
Bug was reproduced on OpenSuse Tumbleweed and Fedora 40 with git lfs version 3.6.1
No proxy or firewall.
Output of git lfs env
git-lfs/3.6.1 (GitHub; linux amd64; go 1.22.10)
git version 2.49.0
Endpoint=https://huggingface.co/nreimers/mmarco-mMiniLMv2-L12-H384-v1.git/info/lfs (auth=none)
LocalWorkingDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1
LocalGitDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git
LocalGitStorageDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git
LocalMediaDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneVerifyUnreachableAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"
Additional context
$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git lfs pull
19:43:28.968733 git.c:772 trace: exec: git-lfs pull
19:43:28.968788 run-command.c:667 trace: run_command: git-lfs pull
19:43:28.968810 run-command.c:759 trace: start_command: /usr/bin/git-lfs pull
19:43:28.983809 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
19:43:28.986397 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
19:43:28.988627 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
19:43:28.990931 trace git-lfs: exec: git 'config' '--includes' '-l'
19:43:28.992261 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
19:43:28.993126 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
19:43:28.993972 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
19:43:28.996417 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
19:43:28.998382 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
19:43:29.000302 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
19:43:29.001901 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
19:43:29.003989 trace git-lfs: tq: running as batched queue, batch size of 100
19:43:29.004160 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-files' '--cached' '--exclude-standard' '--full-name' '--sparse' '-z' '--format=%(objectmode) %(objecttype) %(objectname) %(objectsize) %(path)' ':(top,attr:filter=lfs)'
19:43:29.004562 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
19:43:29.006405 trace git-lfs: filepathfilter: accepting "pytorch_model.bin"
19:43:29.006458 trace git-lfs: filepathfilter: accepting "sentencepiece.bpe.model"
19:43:29.007164 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
19:43:29.007185 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
19:43:29.007238 trace git-lfs: filepathfilter: accepting "tmp"
19:43:29.007309 trace git-lfs: filepathfilter: accepting "1795256050"
19:43:29.007323 trace git-lfs: filepathfilter: accepting "3318263198"
19:43:29.007331 trace git-lfs: filepathfilter: accepting "3873456486"
19:43:29.007341 trace git-lfs: filepathfilter: accepting "1514504523"
19:43:29.007350 trace git-lfs: filepathfilter: accepting "1432722858"
Describe the bug
Git lfs version 3.6.1 does not pull all the files that are part of the lfs management. When testing with version 3.5.1 and 3.3.0 all the files are correctly created.
To Reproduce
Steps to reproduce the behavior:
I was unable to convince git lfs version 3.6.1 to checkout tokenizer.json file.
Expected behavior
All the files including tokenizer.json should be downloaded.
System environment
Bug was reproduced on OpenSuse Tumbleweed and Fedora 40 with git lfs version 3.6.1
No proxy or firewall.
Output of
git lfs envgit-lfs/3.6.1 (GitHub; linux amd64; go 1.22.10)
git version 2.49.0
Endpoint=https://huggingface.co/nreimers/mmarco-mMiniLMv2-L12-H384-v1.git/info/lfs (auth=none)
LocalWorkingDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1
LocalGitDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git
LocalGitStorageDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git
LocalMediaDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneVerifyUnreachableAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/martin/mmarco-mMiniLMv2-L12-H384-v1/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"
Additional context
$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git lfs pull
19:43:28.968733 git.c:772 trace: exec: git-lfs pull
19:43:28.968788 run-command.c:667 trace: run_command: git-lfs pull
19:43:28.968810 run-command.c:759 trace: start_command: /usr/bin/git-lfs pull
19:43:28.983809 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
19:43:28.986397 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
19:43:28.988627 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
19:43:28.990931 trace git-lfs: exec: git 'config' '--includes' '-l'
19:43:28.992261 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
19:43:28.993126 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
19:43:28.993972 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
19:43:28.996417 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
19:43:28.998382 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
19:43:29.000302 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
19:43:29.001901 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
19:43:29.003989 trace git-lfs: tq: running as batched queue, batch size of 100
19:43:29.004160 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-files' '--cached' '--exclude-standard' '--full-name' '--sparse' '-z' '--format=%(objectmode) %(objecttype) %(objectname) %(objectsize) %(path)' ':(top,attr:filter=lfs)'
19:43:29.004562 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
19:43:29.006405 trace git-lfs: filepathfilter: accepting "pytorch_model.bin"
19:43:29.006458 trace git-lfs: filepathfilter: accepting "sentencepiece.bpe.model"
19:43:29.007164 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
19:43:29.007185 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
19:43:29.007238 trace git-lfs: filepathfilter: accepting "tmp"
19:43:29.007309 trace git-lfs: filepathfilter: accepting "1795256050"
19:43:29.007323 trace git-lfs: filepathfilter: accepting "3318263198"
19:43:29.007331 trace git-lfs: filepathfilter: accepting "3873456486"
19:43:29.007341 trace git-lfs: filepathfilter: accepting "1514504523"
19:43:29.007350 trace git-lfs: filepathfilter: accepting "1432722858"