Adding new API for Counting Number of Token from Input Text #13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ramitsurana
commented
Apr 16, 2021
- Updated git ignore
- Remove unused import statement
- Add count token from text api
|
@ramitsurana Thanks for the contribution. Dividing the number of characters by 4 only offers an estimate for the number of tokens, and for certain kinds of text (code, symbols, non-English languages, etc) it may not be very accurate at all. If we were to add a token counter to this project, We'd likely want to provide the real count of tokens instead, which is non-trivial – we'd either need to add in something similar to this (but more lightweight) or call a backend endpoint that offers this functionality. |
|
@schnerd Sure, Thanks for your feedback and sharing the useful insights. I think we could do it by creating a endpoint at the backend and calling it for this command,so in case in future we want to use this functionality for other commands ( like for file create command for comparison between the tokens available in the account vs the tokens that would be consumed by the file that we want to process) we could use it more easily. |
pull bot
referenced
this pull request
in mcx/openai-python
Oct 10, 2021
* Add support for modifying files and file_set support (#4) * Add support for modifying files and file_set support * Bump version * Updated file sets to use name everywhere * Bump to 0.4.0 * Added cli support for file and fileset (#5) * Added cli support for file and fileset * Back to .4.0 * Typo * Add some basic tests to confirm that array stuff works * Added a test for multiple prompts. * refactor retriever endpoint (#6) * Make higherlevel have class methods so you can call with openai.HigherLevel.answer (#7) * refactor retriever endpoint * Actually just make everything a classmethod so you can call it like openai.HigherLevel * Rename file_sets to collections everywhere (#8) * Rename file_sets to collections everywhere * Remove collections (#10) * Higherlevel endpoints now point to /v1 (#11) * Higherlevel endpoints now point to v1 * new line * Move answer and classification to top level attributes, rename higherlevel (#12) * Move answer and classification to top level attributes * New namespaces for answers and classifications * Meant to make the method create * Go up the class stack since we don't need all the things that engineapiresource gives us * Add file support to search (#13) * Add file support to search * Add support for max_rerank * Added return_metadata support * Fixed some cherry pick issues
|
I'm going to close this PR for now since we don't currently support a tokenizer in our API. Feel free to open an issue and we can track progress there |
cgayapr
pushed a commit
to cgayapr/openai-python
that referenced
this pull request
Dec 14, 2024
* Add support for modifying files and file_set support (openai#4) * Add support for modifying files and file_set support * Bump version * Updated file sets to use name everywhere * Bump to 0.4.0 * Added cli support for file and fileset (openai#5) * Added cli support for file and fileset * Back to .4.0 * Typo * Add some basic tests to confirm that array stuff works * Added a test for multiple prompts. * refactor retriever endpoint (openai#6) * Make higherlevel have class methods so you can call with openai.HigherLevel.answer (openai#7) * refactor retriever endpoint * Actually just make everything a classmethod so you can call it like openai.HigherLevel * Rename file_sets to collections everywhere (openai#8) * Rename file_sets to collections everywhere * Remove collections (openai#10) * Higherlevel endpoints now point to /v1 (openai#11) * Higherlevel endpoints now point to v1 * new line * Move answer and classification to top level attributes, rename higherlevel (openai#12) * Move answer and classification to top level attributes * New namespaces for answers and classifications * Meant to make the method create * Go up the class stack since we don't need all the things that engineapiresource gives us * Add file support to search (openai#13) * Add file support to search * Add support for max_rerank * Added return_metadata support * Fixed some cherry pick issues
safa0
pushed a commit
to safa0/openai-agents-python
that referenced
this pull request
Apr 27, 2025
fix: correct typo in README regarding tool responses
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.