Move optimization credentials to configuration #5410

GabrielBianconi · 2025-12-25T17:52:14Z

Move credential-like fields from optimization interface to configuration.
Refactor mock inference server handling: unify under TENSORZERO_INTERNAL_MOCK_PROVIDER_API instead of configuration, and rename to mock-provider-api (since it covers more than inference now).

Note

Centralizes optimizer credential management and standardizes the mock provider across the stack.

Config/optimizers: credentials/api_base removed from optimizer job configs; new provider_types.*.sft blocks (Fireworks, GCP Vertex Gemini, Together); optimizer loading made sync; internal loaders updated to use Box::pin.
Mock provider: rename mock-inference-provider → mock-provider-api; adopt TENSORZERO_INTERNAL_MOCK_PROVIDER_API; all compose, env, and scripts updated; gateway uses mock API base for OpenAI when set.
CI/workflows: all jobs/images/artifacts/needs switched to mock-provider-api; added mock-only UI E2E job; adjusted build/push steps.
Clients/bindings: strip credential/api_base fields from TS bindings and Python stubs; simplify job handles; add pytest mock marker; minor test script quoting fix.
Docs/recipes: add sft config reference details; update Fireworks SFT recipe to use config account_id; minor benchmark command change.
Misc: provider-types restructuring; evaluation/client/config loading refactors; HTTP no-proxy list updated for new mock service name.

^{Written by Cursor Bugbot for commit 687a61d. This will update automatically on new commits. Configure here.}

Important

Centralizes optimizer credentials in configuration and standardizes mock provider API for testing.

Behavior:
- Centralizes optimizer credentials in configuration files, moving them from optimization interfaces.
- Refactors mock inference server handling, using TENSORZERO_INTERNAL_MOCK_PROVIDER_API for standardization.
Configuration:
- Adds provider_types for fireworks.sft, gcp_vertex_gemini.sft, and together.sft in configuration files.
- Splits GCP Vertex configs into Gemini/Anthropic.
Testing:
- Replaces mock-inference-provider with mock-provider-api across tests and CI configurations.
- Updates Python/TS SDK bindings and tests to reflect changes.
Documentation:
- Updates SFT config docs for Fireworks/GCP Vertex/Together.
- Updates Fireworks recipe to read account ID from gateway config.

^{This description was created by}^{for 687a61d. You can customize this summary. It will automatically update as commits are pushed.}

GabrielBianconi · 2025-12-25T17:52:23Z

/merge-queue

github-actions · 2025-12-25T17:53:14Z

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20508839409

Copilot

Pull request overview

This PR refactors optimization configuration by moving credential-like fields (API keys, account IDs, API base URLs) from individual optimization job configs to provider-level configuration. It also unifies mock server handling under a single environment variable TENSORZERO_INTERNAL_MOCK_PROVIDER_API (renamed from multiple provider-specific environment variables) to support testing of both batch inference and optimization workflows.

Key Changes:

Credentials and provider-specific settings (e.g., Fireworks account_id, Together wandb/HuggingFace tokens) moved from per-job optimizer configs to [provider_types.<provider>.sft] sections in gateway config
Mock API handling unified under TENSORZERO_INTERNAL_MOCK_PROVIDER_API environment variable with new helper functions in utils/mock.rs
All optimizer configs (load() methods) simplified from async to sync since credential resolution is now handled at runtime via provider defaults

Reviewed changes

Copilot reviewed 97 out of 111 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`tensorzero-core/src/optimization/{openai_sft,openai_rft,fireworks_sft,together_sft,dicl,gepa}/mod.rs`	Removed credential/api_base fields from configs, simplified `load()` to sync, credentials now fetched at job launch/poll time
`tensorzero-core/src/config/provider_types.rs`	Added `FireworksSFTConfig` and `TogetherSFTConfig` for provider-level SFT settings
`tensorzero-core/src/utils/mock.rs`	New module with `is_mock_mode()` and `get_mock_provider_api_base()` helpers
`tensorzero-optimizers/src/{openai_sft,openai_rft,fireworks_sft,together_sft}.rs`	Updated to fetch credentials and API base from config at runtime instead of storing in job handle
`ui/fixtures/config/tensorzero.toml`	Added Fireworks SFT account_id configuration
`ui/fixtures/config/tensorzero.mock_optimization.toml`	Deleted (no longer needed with new mock approach)
Docker compose files	Renamed `mock-inference-provider` to `mock-provider-api`
CI configurations	Updated environment variables and container names
Python client and TypeScript bindings	Removed credential parameters from optimizer constructors
Recipes and documentation	Updated to reflect new configuration approach

cursor

This PR is being reviewed by Cursor Bugbot

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

clients/python/tests/test_optimization.py

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tensorzero-core/src/config/provider_types.rs

Copilot

Pull request overview

Copilot reviewed 100 out of 114 changed files in this pull request and generated no new comments.

GabrielBianconi · 2025-12-25T18:13:37Z

/merge-queue

GabrielBianconi · 2025-12-25T18:14:31Z

@codex review

github-actions · 2025-12-25T18:14:32Z

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20509067148

ci/buildkite/ui-e2e-tests.sh

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ui/fixtures/regenerate-model-inference-cache.sh

GabrielBianconi · 2025-12-25T18:42:36Z

/merge-queue

GabrielBianconi · 2025-12-25T18:42:45Z

@codex review

github-actions · 2025-12-25T18:43:28Z

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20509388384

Copilot

Pull request overview

Copilot reviewed 101 out of 115 changed files in this pull request and generated no new comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tensorzero-core/src/model.rs

GabrielBianconi · 2025-12-25T18:54:22Z

@greptile review

greptile-apps · 2025-12-25T19:02:38Z

Greptile Summary

This PR successfully centralizes optimizer credentials in configuration files and refactors mock provider handling. The changes move credential-like fields (API keys, account IDs, base URLs) from per-job optimizer configurations to provider_types configuration sections, addressing issues #5384 and #5181.

Major Changes

Configuration centralization: Optimizer credentials now live in provider_types.{provider}.sft sections instead of being passed per-job. This includes Fireworks (account_id), Together (wandb/HuggingFace tokens), and GCP Vertex Gemini (project/region/bucket settings).
Simplified job handles: Removed credential fields from job handle structs (FireworksSFTJobHandle, OpenAISFTJobHandle, etc.), storing only minimal state needed for polling.
Mock provider unification: Renamed mock-inference-provider to mock-provider-api and standardized access via TENSORZERO_INTERNAL_MOCK_PROVIDER_API environment variable.
Credential resolution: All optimizers now resolve credentials at runtime from provider defaults using get_defaulted_credential() pattern.
GCP provider split: Separated GCP Vertex into gcp_vertex_gemini and gcp_vertex_anthropic provider types for clarity.

Implementation Quality

Consistent patterns across all optimizer types (DICL, OpenAI SFT/RFT, Fireworks SFT, Together SFT, GCP SFT)
Comprehensive updates to TypeScript bindings, Python stubs, and documentation
All CI/CD workflows updated with new mock provider image names
Test configurations properly updated with provider-level settings
Box::pin() pattern correctly applied for async config loading

Confidence Score: 5/5

This PR is safe to merge with minimal risk
The refactoring is thorough and well-executed across the entire codebase. All optimizer types follow consistent patterns for credential resolution. The changes are backwards-compatible in terms of behavior (credentials still work, just configured differently). Comprehensive updates to tests, CI/CD, documentation, and bindings demonstrate careful attention to all integration points. The mock provider refactoring is clean and isolated.
No files require special attention

Important Files Changed

Filename	Overview
tensorzero-core/src/config/provider_types.rs	Added SFT configuration structs for Fireworks, GCP Vertex Gemini, and Together, centralizing optimizer credentials in provider config
tensorzero-core/src/optimization/fireworks_sft/mod.rs	Removed credential fields from config and job handle, delegating credential resolution to provider-level configuration
tensorzero-optimizers/src/fireworks_sft.rs	Updated to fetch account_id from provider_types.fireworks.sft configuration and resolve credentials via provider defaults
tensorzero-core/src/utils/mock.rs	New utility module for mock API base URL resolution using TENSORZERO_INTERNAL_MOCK_PROVIDER_API env var
.github/workflows/build-mock-provider-api-container.yml	Renamed workflow from mock-inference-provider to mock-provider-api, updating image tags and artifact names
docs/gateway/configuration-reference.mdx	Added comprehensive documentation for provider_types SFT configuration (Fireworks, GCP Vertex Gemini, Together)

Sequence Diagram

sequenceDiagram
    participant UI as UI/Client
    participant Gateway as Gateway
    participant Config as Configuration
    participant Optimizer as Optimizer
    participant ProviderAPI as Provider API
    
    Note over UI,ProviderAPI: Optimization Job Launch Flow
    
    UI->>Gateway: POST /optimize (job config)
    Gateway->>Config: Load provider_types config
    Config-->>Gateway: Provider SFT settings<br/>(account_id, credentials, etc)
    Gateway->>Optimizer: launch(config, credentials)
    Optimizer->>Config: Get provider defaults
    Config-->>Optimizer: Credential locations<br/>(from provider_types)
    Optimizer->>Optimizer: Resolve credentials<br/>(API keys, etc)
    Optimizer->>ProviderAPI: Upload training data
    ProviderAPI-->>Optimizer: File/dataset ID
    Optimizer->>ProviderAPI: Create fine-tuning job
    ProviderAPI-->>Optimizer: Job ID
    Optimizer-->>Gateway: JobHandle (minimal state)
    Gateway-->>UI: Job created response
    
    Note over UI,ProviderAPI: Optimization Job Polling Flow
    
    UI->>Gateway: GET /optimize/status (job_handle)
    Gateway->>Config: Load provider_types config
    Config-->>Gateway: Provider SFT settings
    Gateway->>Optimizer: poll(job_handle)
    Optimizer->>Config: Get provider defaults
    Config-->>Optimizer: Credential locations
    Optimizer->>Optimizer: Resolve credentials
    Optimizer->>ProviderAPI: Check job status
    ProviderAPI-->>Optimizer: Job status/model info
    Optimizer-->>Gateway: OptimizationJobInfo
    Gateway-->>UI: Status response
    
    Note over Config,ProviderAPI: Key Changes in This PR
    Note over Config: Credentials moved from<br/>per-job config to<br/>provider_types config
    Note over Optimizer: JobHandle stores<br/>minimal state<br/>(no credentials)
    Note over ProviderAPI: Mock provider unified<br/>under single API

greptile-apps · 2025-12-25T19:02:39Z

Greptile found no issues!

From now on, if a review finishes and we haven't found any issues, we will not post anything, but you can confirm that we reviewed your changes in the status check section.

_{This feature can be toggled off in your Code Review Settings by deselecting "Create a status check for each PR".}

GabrielBianconi · 2025-12-25T20:29:58Z

/merge-queue

GabrielBianconi · 2025-12-25T20:30:04Z

@codex review

github-actions · 2025-12-25T20:30:53Z

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20510538059

Copilot

Pull request overview

Copilot reviewed 104 out of 118 changed files in this pull request and generated no new comments.

chatgpt-codex-connector · 2025-12-25T20:43:02Z

Codex Review: Didn't find any major issues. Bravo.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

GabrielBianconi · 2025-12-25T21:06:00Z

/merge-queue

github-actions · 2025-12-25T21:06:53Z

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20510909893

GabrielBianconi · 2025-12-25T21:38:43Z

/merge-queue

github-actions · 2025-12-25T21:39:33Z

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20511270037

Copilot

Pull request overview

Copilot reviewed 104 out of 118 changed files in this pull request and generated no new comments.

Move optimization credentials to configuration

b4e9d13

Copilot AI review requested due to automatic review settings December 25, 2025 17:52

Copilot started reviewing on behalf of GabrielBianconi December 25, 2025 17:52 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

cursor bot reviewed Dec 25, 2025

View reviewed changes

clients/python/tests/test_optimization.py Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Dec 25, 2025

View reviewed changes

tensorzero-core/src/config/provider_types.rs Show resolved Hide resolved

GabrielBianconi added 2 commits December 25, 2025 13:08

Move optimization credentials to configuration

36cba87

Move optimization credentials to configuration

b62d355

Copilot AI review requested due to automatic review settings December 25, 2025 18:11

Copilot started reviewing on behalf of GabrielBianconi December 25, 2025 18:11 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

cursor bot reviewed Dec 25, 2025

View reviewed changes

ci/buildkite/ui-e2e-tests.sh Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Dec 25, 2025

View reviewed changes

ui/fixtures/regenerate-model-inference-cache.sh Show resolved Hide resolved

Move optimization credentials to configuration

76391b8

Move optimization credentials to configuration

4587b68

Copilot AI review requested due to automatic review settings December 25, 2025 18:51

Copilot started reviewing on behalf of GabrielBianconi December 25, 2025 18:52 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

chatgpt-codex-connector bot reviewed Dec 25, 2025

View reviewed changes

tensorzero-core/src/model.rs Show resolved Hide resolved

Copilot AI review requested due to automatic review settings December 25, 2025 20:29

Copilot started reviewing on behalf of GabrielBianconi December 25, 2025 20:30 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

Move optimization credentials to configuration

32bc911

Move optimization credentials to configuration

73f4c57

Copilot AI review requested due to automatic review settings December 25, 2025 21:38

Copilot started reviewing on behalf of GabrielBianconi December 25, 2025 21:39 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

GabrielBianconi assigned virajmehta Dec 25, 2025

virajmehta previously approved these changes Dec 26, 2025

View reviewed changes

virajmehta added this pull request to the merge queue Dec 26, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Dec 26, 2025

virajmehta assigned GabrielBianconi and unassigned virajmehta Dec 26, 2025

Fix

687a61d

GabrielBianconi dismissed virajmehta’s stale review via 687a61d December 26, 2025 15:54

virajmehta approved these changes Dec 26, 2025

View reviewed changes

virajmehta enabled auto-merge December 26, 2025 15:59

virajmehta added this pull request to the merge queue Dec 26, 2025

Merged via the queue into main with commit 033935e Dec 26, 2025
61 checks passed

virajmehta deleted the gb/fix-5384 branch December 26, 2025 16:46

Move optimization credentials to configuration #5410

Move optimization credentials to configuration #5410

Uh oh!

Conversation

GabrielBianconi commented Dec 25, 2025 • edited by ellipsis-dev bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

github-actions bot commented Dec 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

This PR is being reviewed by Cursor Bugbot

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

github-actions bot commented Dec 25, 2025

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

github-actions bot commented Dec 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

greptile-apps bot commented Dec 25, 2025

Greptile Summary

Major Changes

Implementation Quality

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot commented Dec 25, 2025

Greptile found no issues!

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

GabrielBianconi commented Dec 25, 2025

Uh oh!

github-actions bot commented Dec 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

GabrielBianconi commented Dec 25, 2025 •

edited by ellipsis-dev bot

Loading