Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add support for BERT embedding models #5423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Feb 11, 2024
+616 −52
Merged

Add support for BERT embedding models #5423

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7286b83
BERT WIP
cebtenzzre Feb 6, 2024
ef10d78
merge from master
iamlemec Feb 7, 2024
0051c82
it runs; tokenization is messed up; pooling is wrong for multi batches
iamlemec Feb 8, 2024
59c1829
add in wordpiece tokenizer
iamlemec Feb 8, 2024
5f1c21d
put causal_attn flag in gguf
iamlemec Feb 8, 2024
e0e14e3
Merge remote-tracking branch 'origin/master' into bert
iamlemec Feb 8, 2024
7218c7b
Merge remote-tracking branch 'upstream/master' into bert
iamlemec Feb 8, 2024
e3efcf1
Update convert-hf-to-gguf.py
iamlemec Feb 8, 2024
96d37f8
add causal attention gguf key
iamlemec Feb 8, 2024
e78388d
use ctx_output for tok_norm of BERT and BLOOM
cebtenzzre Feb 9, 2024
b14c457
bert : add some missing graph callbacks
cebtenzzre Feb 9, 2024
6875808
fix up model sizing and result acquisition
iamlemec Feb 9, 2024
d080beb
hard-code token_type = 0
iamlemec Feb 9, 2024
3a1895d
Merge branch 'bert' of github.com:iamlemec/llama.cpp into bert
iamlemec Feb 9, 2024
961e98f
style fixes
iamlemec Feb 9, 2024
56afb2f
undo attempted type_embd simplify
iamlemec Feb 9, 2024
ab49e9e
bert : simplify token type embedding access
cebtenzzre Feb 9, 2024
6972e7e
flake8 : add W503 to ignore list
cebtenzzre Feb 9, 2024
8fbefed
minor : code style normalization
ggerganov Feb 11, 2024
e379e8c
avoid use of ggml_graph_get_tensor
iamlemec Feb 11, 2024
61bab47
Merge branch 'bert' of github.com:iamlemec/llama.cpp into bert
iamlemec Feb 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
avoid use of ggml_graph_get_tensor
  • Loading branch information
@iamlemec
iamlemec committed Feb 11, 2024
commit e379e8c10b11f20ca6a2fdeb19b45a05a9247128
22 changes: 14 additions & 8 deletions llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7484,15 +7484,21 @@ static int llama_decode_internal(

ggml_cgraph * gf = llama_build_graph(lctx, batch);

// get logits and embeddings
struct ggml_tensor * res = ggml_graph_get_tensor(gf, "result_output");
struct ggml_tensor * embeddings = ggml_graph_get_tensor(gf, "result_norm");

// if logits are none we must be doing embeddings
if (res == nullptr) {
embeddings = ggml_graph_get_tensor(gf, "result_embed");
// the output is always the last tensor in the graph
struct ggml_tensor * res = gf->nodes[gf->n_nodes - 1];
struct ggml_tensor * embeddings = gf->nodes[gf->n_nodes - 2];
if (strcmp(res->name, "result_output") == 0) {
// the embeddings could be the second to last tensor, or the third to last tensor
if (strcmp(embeddings->name, "result_norm") != 0) {
embeddings = gf->nodes[gf->n_nodes - 3];
GGML_ASSERT(strcmp(embeddings->name, "result_norm") == 0);
}
} else if (strcmp(res->name, "result_embed") == 0) {
embeddings = res;
res = nullptr;
} else {
GGML_ASSERT(false);
}
GGML_ASSERT(res || embeddings);

// LLAMA_LOG_INFO("graph build time: %.3f ms (%d nodes, %d leafs)\n", (ggml_time_us() - t_start_us)/1000.0, gf->n_nodes, gf->n_leafs);

Expand Down