Configure assistant model's generation_config with user parameters#40740
Configure assistant model's generation_config with user parameters#40740geoffrey-young wants to merge 2 commits into
Conversation
|
cc @gante |
f134dd4 to
30c2ed5
Compare
0f07796 to
9c2579d
Compare
zucchini-nlp
left a comment
There was a problem hiding this comment.
There is so much legacy around the generation config currently, with layers of prioritization 🥲
At first glance looks okay to me, though I also wonder if we could move this block to CandidateGenerator. I think there is at least one edge case which is causing the CI to fails with Gemma3 models currently. So I think we need to write out in code comment what is prioritized for generation config values for our future selves and fix the failing tests @geoffrey-young
Ideally for long term, it's time for us to break the BC taking advantage of v5 and clean up the code. It will definitely be a different PR and take longer time though
|
thanks for taking a look zucchini-nlp. I wouldn't say this is a high priority. however, I did at least want to be sure to point out the inheritance issues with the assistant_model configuration API, since the effects really aren't obvious downstream. I only noticed the issue because I was doing a deep trace to understand how speculative decoding worked :) generating a PR to at least offer a potential path to a fix is the least I could do. my local dev test env is small and the larger suite is blowing up even without the changes, so it's hard for me to trace what the breaking aspects are in order to assist much further. I do appreciate you taking a look, though - many thanks. |
|
Yeah, agreed 100%. Thanks for pointing out to the issue, I will start cleaning up generation config little by little. Hopefully for the v5 release, it will be solved |
What does this PR do?
Fixes issue where num_assistant_tokens and other parameters intended for the assistant model are not properly passed down, leaving the assistant model with default values.
Ultimately, we want to configure the assistant model properly, so that it:
Tests added to illustrate the issue, and detailed comments to illustrate behaviors.
Fixes # 40739
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.