feat: add service tier to TokenUsage recording - #28709
Conversation
10f23a2 to
b441839
Compare
b441839 to
0c14405
Compare
mtojek
left a comment
There was a problem hiding this comment.
plumbing and testing 👍 left minor question about the structure.
| serviceTier := string(response.ServiceTier) | ||
| var metadata recorder.Metadata | ||
| if serviceTier != "" { | ||
| metadata = recorder.Metadata{recorder.MetadataKeyServiceTier: serviceTier} |
There was a problem hiding this comment.
just confirmation: we want an empty structure rather rather nil-property?
There was a problem hiding this comment.
I think it is the same in case of recorder.Metadata which is a map[string]any since empty map == nil.
Struct with Metadata not set should be equal to one with Metadata set to empty map / nil, tested here: https://github.com/coder/coder/pull/28709/changes#diff-132fbd5ab5a0ddb045cc7066cb363bd86cbf6ea92151b10d7995be40dfdb1647R451-R470
| InterceptionID: i.ID().String(), | ||
| MsgID: message.ID, | ||
| Output: delta.Usage.OutputTokens, | ||
| i.recordTokenUsage(streamCtx, message.ID, anthropic.Usage{ |
There was a problem hiding this comment.
This is a behaviour change: before we were only storing Output, but now we store all the fields from recordTokenUsage. Are all those fields reported by message_delta events?
There was a problem hiding this comment.
Good catch, I believe other then ExtraTokenTypes would default to 0 before so there is no behaviour change for those. For ExtraTokenTypes there is as there was empty map before and now there is map with entries that has 0 value.
Will change so ExtraTokenTypes are only added when non 0 which is also a behaviour change but I think it is an improvement, no need to store 0 values in this map.
| i.mcpProxy = mcpProxy | ||
| } | ||
|
|
||
| func (i *interceptionBase) recordTokenUsage(ctx context.Context, msgID string, usage anthropic.Usage) { |
There was a problem hiding this comment.
nit: Could we add a small unit test for Anthropic usage without service_tier? That would verify we do not create a misleading empty tier, matching the coverage in the OpenAI paths.
Adds
service_tierinformation to token usage recordings made by OpenAI and Anthropic interceptors.