© 2025 Muvon Un Limited (Hong Kong) | Website | Product Page
Octolib is a comprehensive, self-sufficient AI provider library that provides a unified, type-safe interface for interacting with multiple AI services. It offers intelligent model selection, robust error handling, and advanced features like cross-provider tool calling and vision support.
- 🔌 Multi-Provider Support: OpenAI, Anthropic, OpenRouter, Google, Amazon, Cloudflare, DeepSeek
- 🛡️ Unified Interface: Consistent API across different providers
- 🔍 Intelligent Model Validation: Strict
provider:modelformat parsing - 📋 Structured Output: JSON and JSON Schema support for OpenAI, OpenRouter, and DeepSeek
- 💰 Cost Tracking: Automatic token usage and cost calculation
- 🖼️ Vision Support: Image attachment handling for compatible models
- 🧰 Tool Calling: Cross-provider tool call standardization
- ⏱️ Retry Management: Configurable exponential backoff
- 🔒 Secure Design: Environment-based API key management
- 🎯 Embedding Support: Multi-provider embedding generation with Jina, Voyage, Google, OpenAI, FastEmbed, and HuggingFace
# Add to Cargo.toml
octolib = { git = "https://github.com/muvon/octolib" }use octolib::{ProviderFactory, ChatCompletionParams, Message};
async fn example() -> anyhow::Result<()> {
// Parse model and get provider
let (provider, model) = ProviderFactory::get_provider_for_model("openai:gpt-4o")?;
// Create messages
let messages = vec![
Message::user("Hello, how are you?"),
];
// Create completion parameters
let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000);
// Get completion (requires OPENAI_API_KEY environment variable)
let response = provider.chat_completion(params).await?;
println!("Response: {}", response.content);
Ok(())
}Get structured JSON responses with schema validation:
use octolib::{ProviderFactory, ChatCompletionParams, Message, StructuredOutputRequest};
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize, Debug)]
struct PersonInfo {
name: String,
age: u32,
skills: Vec<String>,
}
async fn structured_example() -> anyhow::Result<()> {
let (provider, model) = ProviderFactory::get_provider_for_model("openai:gpt-4o")?;
// Check if provider supports structured output
if !provider.supports_structured_output(&model) {
return Err(anyhow::anyhow!("Provider does not support structured output"));
}
let messages = vec![
Message::user("Tell me about a software engineer in JSON format"),
];
// Request structured JSON output
let structured_request = StructuredOutputRequest::json();
let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000)
.with_structured_output(structured_request);
let response = provider.chat_completion(params).await?;
if let Some(structured) = response.structured_output {
let person: PersonInfo = serde_json::from_value(structured)?;
println!("Person: {:?}", person);
}
Ok(())
}Use AI models to call functions with automatic parameter extraction:
use octolib::{ProviderFactory, ChatCompletionParams, Message, FunctionDefinition, ToolCall};
use serde_json::json;
async fn tool_calling_example() -> anyhow::Result<()> {
let (provider, model) = ProviderFactory::get_provider_for_model("openai:gpt-4o")?;
// Define available tools/functions
let tools = vec![
FunctionDefinition {
name: "get_weather".to_string(),
description: "Get the current weather for a location".to_string(),
parameters: json!({
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}),
cache_control: None,
},
FunctionDefinition {
name: "calculate".to_string(),
description: "Perform a mathematical calculation".to_string(),
parameters: json!({
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "Mathematical expression to evaluate"
}
},
"required": ["expression"]
}),
cache_control: None,
},
];
let mut messages = vec![
Message::user("What's the weather in Tokyo and calculate 15 * 23?"),
];
// Initial request with tools
let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000)
.with_tools(tools.clone());
let response = provider.chat_completion(params).await?;
// Check if model wants to call tools
if let Some(tool_calls) = response.tool_calls {
println!("Model requested {} tool calls", tool_calls.len());
// Add assistant's response with tool calls to conversation
let mut assistant_msg = Message::assistant(&response.content);
assistant_msg.tool_calls = Some(serde_json::to_value(&tool_calls)?);
messages.push(assistant_msg);
// Execute each tool call and add results
for tool_call in tool_calls {
println!("Calling tool: {} with args: {}", tool_call.name, tool_call.arguments);
// Execute the tool (your implementation)
let result = match tool_call.name.as_str() {
"get_weather" => {
let location = tool_call.arguments["location"].as_str().unwrap_or("Unknown");
json!({
"location": location,
"temperature": 22,
"unit": "celsius",
"condition": "sunny"
})
}
"calculate" => {
let expr = tool_call.arguments["expression"].as_str().unwrap_or("0");
// Simple calculation (in real app, use proper eval)
json!({
"expression": expr,
"result": 345 // 15 * 23
})
}
_ => json!({"error": "Unknown tool"}),
};
// Add tool result to conversation
messages.push(Message::tool(
&serde_json::to_string(&result)?,
&tool_call.id,
&tool_call.name,
));
}
// Get final response with tool results
let params = ChatCompletionParams::new(&messages, &model, 0.7, 1.0, 50, 1000)
.with_tools(tools);
let final_response = provider.chat_completion(params).await?;
println!("Final response: {}", final_response.content);
} else {
println!("Direct response: {}", response.content);
}
Ok(())
}Tool Calling Features:
- ✅ Cross-provider support (OpenAI, Anthropic, Google, Amazon, OpenRouter)
- ✅ Automatic parameter validation via JSON Schema
- ✅ Multi-turn conversations with tool results
- ✅ Parallel tool execution support
- ✅ Standardized
ToolCallandGenericToolCallformats across all providers - ✅ Provider-specific metadata preservation (e.g., Gemini thought signatures)
- ✅ Clean conversion API with
to_generic_tool_calls()method
Generate embeddings using multiple providers:
use octolib::embedding::{generate_embeddings, generate_embeddings_batch, InputType};
async fn embedding_example() -> anyhow::Result<()> {
// Single embedding generation
let embedding = generate_embeddings(
"Hello, world!",
"voyage", // provider
"voyage-3.5-lite" // model
).await?;
println!("Embedding dimension: {}", embedding.len());
// Batch embedding generation
let texts = vec![
"First document".to_string(),
"Second document".to_string(),
];
let embeddings = generate_embeddings_batch(
texts,
"jina", // provider
"jina-embeddings-v4", // model
InputType::Document, // input type for better embeddings
16, // batch size
100_000, // max tokens per batch
).await?;
println!("Generated {} embeddings", embeddings.len());
Ok(())
}
// Supported embedding providers:
// - Jina: jina-embeddings-v4, jina-clip-v2, etc.
// - Voyage: voyage-3.5, voyage-code-2, etc.
// - Google: gemini-embedding-001, text-embedding-005
// - OpenAI: text-embedding-3-small, text-embedding-3-large
// - FastEmbed: Local models (feature-gated)
// - HuggingFace: sentence-transformers models| Provider | Structured Output | Vision | Tool Calls | Caching |
|---|---|---|---|---|
| OpenAI | ✅ JSON + Schema | ✅ Yes | ✅ Yes | ✅ Yes |
| OpenRouter | ✅ JSON + Schema | ✅ Yes | ✅ Yes | ✅ Yes |
| DeepSeek | ✅ JSON Mode | ❌ No | ❌ No | ✅ Yes |
| Anthropic | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes |
| Google Vertex | ❌ No | ✅ Yes | ✅ Yes | ❌ No |
| Amazon Bedrock | ❌ No | ✅ Yes | ✅ Yes | ❌ No |
| Cloudflare | ❌ No | ❌ No | ❌ No | ❌ No |
- JSON Mode: Basic JSON object output
- JSON Schema: Full schema validation with strict mode
- Provider Detection: Use
provider.supports_structured_output(&model)to check capability
📖 Quick Navigation
- Overview - Library introduction and core concepts
- Installation Guide - Setup and configuration
- Advanced Usage - Advanced features and customization
- Advanced Guide - Comprehensive usage patterns
- Embedding Guide - Embedding generation with multiple providers
- Tool Calling Guide - Tool calling architecture and metadata handling
| Provider | Status | Capabilities |
|---|---|---|
| OpenAI | ✅ Full Support | Chat, Vision, Tools, Structured Output |
| Anthropic | ✅ Full Support | Claude Models, Vision, Tools, Caching |
| OpenRouter | ✅ Full Support | Multi-Provider Proxy, Vision, Caching, Structured Output |
| DeepSeek | ✅ Full Support | Open-Source AI Models, Structured Output |
| Google Vertex AI | ✅ Supported | Enterprise AI Integration |
| Amazon Bedrock | ✅ Supported | Cloud AI Services |
| Cloudflare Workers AI | ✅ Supported | Edge AI Compute |
- 🏠 Local-first design
- 🔑 Secure API key management
- 📁 Respects .gitignore
- 🛡️ Comprehensive error handling
- 🐛 Issues: GitHub Issues
- 📧 Email: [email protected]
- 🏢 Company: Muvon Un Limited (Hong Kong)
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Built with ❤️ by the Muvon team in Hong Kong