Codestin Search App

mgoin · 2023-08-07T21:34:54Z

Goal: Make a text-generation server that is compatible with the OpenAI API Reference so it can plug-in readily with applications that use the interface.

Install requirements

pip install -r requirements.txt

Example usage

Set up the server:

python examples/openai-server/server.py --model zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none --prompt-processing-sequence-length 1
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2023-08-07 17:18:32 __main__     INFO     args: Namespace(model='zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none', max_model_len=512, prompt_processing_sequence_length=1, use_deepsparse_cache=False, host='localhost', port=8000, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], served_model_name=None)
2023-08-07 17:18:32 deepsparse.transformers WARNING  The neuralmagic fork of transformers may not be installed. It can be installed via `pip install nm_transformers`
Using pad_token, but it is not set yet.
2023-08-07 17:18:34 deepsparse.transformers.engines.nl_decoder_engine INFO     Overwriting in-place the input shapes of the transformer model at /home/mgoin/.cache/sparsezoo/neuralmagic/codegen_mono-350m-bigpython_bigquery_thepile-base/model.onnx
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.6.0 COMMUNITY | (98238edf) (optimized) (system=avx512_vnni, binary=avx512)
2023-08-07 17:18:48 deepsparse.transformers.engines.nl_decoder_engine INFO     Overwriting in-place the input shapes of the transformer model at /home/mgoin/.cache/sparsezoo/neuralmagic/codegen_mono-350m-bigpython_bigquery_thepile-base/model.onnx
INFO:     Started server process [314509]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)

Query the server for what models are available using the Models API:

curl http://localhost:8000/v1/models     
{"object":"list","data":[{"id":"zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none","object":"model","created":1691444523,"owned_by":"neuralmagic","root":"zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none","parent":null,"permission":[{"id":"modelperm-d0d9f0bb6a5c48458848e6b9a8cb8aca","object":"model_permission","created":1691444523,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}

Then you can hit the Completions API with a curl command and see the streaming output:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none",
        "prompt": "def fib():",
        "max_tokens": 16,
        "stream": true
    }'
data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "def fib():\n", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "a, ", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "0, ", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "  ", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "while ", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "True:\n", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "  ", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-473d4978ecc64a61a5eb6c442505aeba", "object": "text_completion", "created": 1691444444, "model": "zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": null}]}

data: [DONE]

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1205218032618017

bfineran

LGTM overall for examples @dsikka will be taking a look at what we can do to get something in for src

mgoin force-pushed the openai-server-api branch from 7c57939 to 2d8a4cd Compare August 7, 2023 21:50

mgoin added 3 commits August 9, 2023 11:47

Implement OpenAI-compatible server

dc86821

Add client example and README

d77d316

Style

e89ff48

mgoin force-pushed the openai-server-api branch from a2d97ee to e89ff48 Compare August 9, 2023 15:47

mgoin added 5 commits August 9, 2023 17:17

Merge branch 'main' into openai-server-api

f8a8633

Merge branch 'main' into openai-server-api

aad71d7

Update and rebase

70c4bb7

Format

f3f2c60

Format

f34898d

mgoin marked this pull request as ready for review August 14, 2023 20:45

mgoin requested review from a team, DaltheCow, Satrat, bfineran and eldarkurtic and removed request for a team August 14, 2023 21:28

mgoin added 2 commits August 22, 2023 16:47

Merge branch 'main' into openai-server-api

41c786a

Merge branch 'main' into openai-server-api

1dc6f1d

bfineran approved these changes Aug 23, 2023

View reviewed changes

mgoin merged commit ea853cb into main Aug 26, 2023

mgoin deleted the openai-server-api branch August 26, 2023 19:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement OpenAI-compatible server#1171

Implement OpenAI-compatible server#1171
mgoin merged 10 commits into
mainfrom
openai-server-api

mgoin commented Aug 7, 2023 •

edited

Loading

Uh oh!

bfineran left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mgoin commented Aug 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Install requirements

Example usage

Uh oh!

bfineran left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mgoin commented Aug 7, 2023 •

edited

Loading

bfineran left a comment •

edited

Loading