This repository was archived by the owner on Jun 3, 2025. It is now read-only.
[Text Generation] Support for causal masks, internal KV cache, and initial testing framework#1172
Merged
bfineran merged 16 commits intoAug 10, 2023
Merged
Conversation
causal mask support in the pipeline.
dbogunowicz
commented
Aug 8, 2023
bfineran
reviewed
Aug 8, 2023
Co-authored-by: Benjamin Fineran <[email protected]>
Co-authored-by: Benjamin Fineran <[email protected]>
…ttps://github.com/neuralmagic/deepsparse into feature/damian/backward_comp_no_causal_mask_models
bfineran
reviewed
Aug 9, 2023
bfineran
reviewed
Aug 9, 2023
bfineran
reviewed
Aug 9, 2023
bfineran
reviewed
Aug 9, 2023
bfineran
previously approved these changes
Aug 9, 2023
Satrat
previously approved these changes
Aug 10, 2023
Satrat
previously approved these changes
Aug 10, 2023
bfineran
previously approved these changes
Aug 10, 2023
…rk (#1163) * Create test_nl_decoder_engine.py * [Text Generation][Tests] DecoderKVCache (#1154) * [Text Generation][Tests] NLDecoderEngine (#1155) * initial commit * initial commit * [Text Generation][Tests] Text Generation Pipeline (#1162) * initial implementation * problems with multitoken prefill * almost there... * finally all tests pass * just need to change to stub * fix bad merge * Make tests work with stub (as much as possible), cleanup test names, disable heavy tests, include patch for running without causal mask * use patch from unittest library - remove additional dependency * Update tests/deepsparse/transformers/pipelines/test_text_generation.py * clarify todo comment * [Text Generation] KV Cache internal Deepsparse support (#1135) * fix kv cache * refactor * add validation pathway * avx2 support * initial commit * initial commit * initial implementation * problems with multitoken prefill * its working * almost there... * finally all tests pass * just need to change to stub * fix bad merge * added some tests * ready for review * full support --------- Co-authored-by: dbogunowicz <[email protected]> Co-authored-by: Damian <[email protected]> * incomplete string in parametrize * few nits before the merge --------- Co-authored-by: Benjamin Fineran <[email protected]> Co-authored-by: Sage Moore <[email protected]>
causal mask support in the pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
INCLUDED PRs:
Includes PR #1163 which adds testing and internal KV cache support, made up of: