Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix tensor groups for encoder-decoder models in gguf-dump.py #8090

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 24, 2024
Merged

Fix tensor groups for encoder-decoder models in gguf-dump.py #8090

merged 2 commits into from
Jun 24, 2024

Conversation

fairydreaming
Copy link
Collaborator

@fairydreaming fairydreaming commented Jun 24, 2024 •
edited
Loading

t5-small-dump.txt
This PR corrects tensor groups for encoder-decoder models like T5 and FLAN-T5 family. Separate tensor groups are created for each enc.blk.[bid], additional tensor groups are created for remaining non-blk enc and dec tensors. Example:

## Tensors Overview ~61M Elements

Total number of elements in all tensors: 60506880 Elements

- [Decoder Block 0 Tensor Group - ~4M Elements](#dec_blk_0)
- [Decoder Block 1 Tensor Group - ~4M Elements](#dec_blk_1)
- [Decoder Block 2 Tensor Group - ~4M Elements](#dec_blk_2)
- [Decoder Block 3 Tensor Group - ~4M Elements](#dec_blk_3)
- [Decoder Block 4 Tensor Group - ~4M Elements](#dec_blk_4)
- [Decoder Block 5 Tensor Group - ~4M Elements](#dec_blk_5)
- [Decoder Tensor Group - 512 Elements](#dec)
- [Encoder Block 0 Tensor Group - ~3M Elements](#enc_blk_0)
- [Encoder Block 1 Tensor Group - ~3M Elements](#enc_blk_1)
- [Encoder Block 2 Tensor Group - ~3M Elements](#enc_blk_2)
- [Encoder Block 3 Tensor Group - ~3M Elements](#enc_blk_3)
- [Encoder Block 4 Tensor Group - ~3M Elements](#enc_blk_4)
- [Encoder Block 5 Tensor Group - ~3M Elements](#enc_blk_5)
- [Encoder Tensor Group - 512 Elements](#enc)
- [Base Tensor Group - ~16M Elements](#base)

In attached file there is a complete output from python3 gguf-py/scripts/gguf-dump.py --markdown /mnt/md0/models/t5-small.gguf command.

@mofosyne mofosyne added the Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix label Jun 24, 2024
@mofosyne mofosyne added the merge ready indicates that this may be ready to merge soon and is just holding out in case of objections label Jun 24, 2024
@github-actions github-actions bot added the python python script changes label Jun 24, 2024
@fairydreaming fairydreaming merged commit d62e4aa into ggml-org:master Jun 24, 2024
17 checks passed
@fairydreaming fairydreaming mentioned this pull request Jun 24, 2024
Merged
4 tasks
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Jun 30, 2024
@fairydreaming @sszymczy
@mofosyne @NeoZhangJianyu
gguf-py : fix tensor groups for encoder-decoder models in gguf-dump.py (
e6f3969 
ggml-org#8090)

Co-authored-by: Stanisław Szymczyk <[email protected]>
Co-authored-by: Brian <[email protected]>
MagnusS0 pushed a commit to MagnusS0/llama.cpp-normistral-tokenizer that referenced this pull request Jul 1, 2024
@fairydreaming @sszymczy
@mofosyne @MagnusS0
gguf-py : fix tensor groups for encoder-decoder models in gguf-dump.py (
68a59ed 
ggml-org#8090)

Co-authored-by: Stanisław Szymczyk <[email protected]>
Co-authored-by: Brian <[email protected]>
@fairydreaming fairydreaming deleted the gguf-dump-grouping-fix branch March 22, 2025 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mofosyne mofosyne mofosyne approved these changes

@compilade compilade Awaiting requested review from compilade

Assignees
No one assigned
Labels
merge ready indicates that this may be ready to merge soon and is just holding out in case of objections python python script changes Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@fairydreaming @mofosyne @sszymczy