Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

[Text Generation][KVCacheStorage] ChatPipeline implementation#1266

Merged
bfineran merged 22 commits into
mainfrom
feature/damian/chat_pipeline_new
Sep 22, 2023
Merged

[Text Generation][KVCacheStorage] ChatPipeline implementation#1266
bfineran merged 22 commits into
mainfrom
feature/damian/chat_pipeline_new

Conversation

@dbogunowicz

@dbogunowicz dbogunowicz commented Sep 22, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Implementation of ChatPipeline that builds on top of TextGenerationPipeline, but uses session_ids and StorageKVCache to enable continuous recollection of past kv_cache information.

Example use:

from deepsparse import Pipeline

pipeline = Pipeline.create(
    task="chat",
    model_path="/home/ubuntu/damian/sparseml/deployment_opt",
)
session_id = "session_id"
while True:
    # get input from user
    input_text = input("User: ")
    response = pipeline(sequences=[input_text], session_ids=session_id, max_tokens=32)
    print("Bot: ", response.generations[0].text)
User: Hi my name is Damian and I am from Italy.
Bot:   I am a professional photographer and I have been working in the industry for over 10 years. I have been working in the industry for the last 5 years and I
User: am from   
Bot:   Italy. I have been working in the industry for the last 5 years and I am from  Italy. I have been working in the industry for the last 5
User: Did I mention that my name is
Bot:   Damian?

I am a professional photographer and I have been working in the industry for the last 5 years and

Testing:

  • successfully ran the full LLM testing suite

dbogunowicz and others added 17 commits September 14, 2023 15:20
@dbogunowicz
initial commit
837905a
@dbogunowicz
Merge branch 'main' of https://github.com/neuralmagic/deepsparse into…
22534d7 
… main
@dbogunowicz
Merge branch 'main' of https://github.com/neuralmagic/deepsparse into…
d9b35e9 
… main
@dbogunowicz
upload draft for review
803e536
@dbogunowicz
Merge branch 'main' of https://github.com/neuralmagic/deepsparse into…
4707d02 
… main
@dbogunowicz
initial implementation. testing now
76ae856
@dbogunowicz
Merge branch 'main' into feature/damian/chat_pipeline
34a9164
@dbogunowicz
in this form tests pass
44cb44a
@dbogunowicz
Merge branch 'feature/damian/chat_pipeline' of https://github.com/neu…
2468e77 
…ralmagic/deepsparse into feature/damian/chat_pipeline
@dbogunowicz
cleanup
0b0e75c
@dbogunowicz
ready for review
412cfc7
@dbogunowicz
initial commit
fc965f5
@dbogunowicz
Update src/deepsparse/transformers/pipelines/text_generation.py
10fe119
@dbogunowicz
PR review changes
86cacef
@dbogunowicz
Merge branch 'feature/damian/chat_pipeline' into feature/damian/actua…
5986977 
…l_chat_pipeline
@dbogunowicz
ready for review
660e45a
@dbogunowicz
Merge remote-tracking branch 'origin/main' into feature/damian/chat_p…
587e651 
…ipeline_new
@dbogunowicz dbogunowicz changed the title Feature/damian/chat pipeline new [Text Generation][KVCacheStorage] ChatPipeline implementation Sep 22, 2023
@bfineran

bfineran commented Sep 22, 2023

Copy link
Copy Markdown
Contributor

error looks unrelated right now - merging

@bfineran bfineran merged commit 098d1db into main Sep 22, 2023
@bfineran bfineran deleted the feature/damian/chat_pipeline_new branch September 22, 2023 20:49
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@bfineran bfineran bfineran approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dbogunowicz @bfineran @dsikka