Prompt Caching

​

Overview

​

Cost Savings

• Cache read tokens: 10% of regular price (90% discount)
• Cache write tokens: 25% more than regular price
• Net effect: Savings increase with conversation length, reaching 75%+ for 10+ turn conversations

​

How It Works

1. System Prompt — your agent’s instructions (static)
2. Tool Definitions — all tool schemas (rarely change)
3. Conversation History — the growing message history (last content block)

​

When Caching Helps Most

• Long conversations: More turns = more savings
• Many tools: Large tool definitions cached once
• Batch processing: Multiple similar requests in same session
• Active sessions: Continuous conversation within 5 minutes

​

Usage

let config = AgentConfig::new("You are helpful")
    .with_tools(tools);  // Caching automatically enabled

let agent = StandardAgent::new(config, llm);

let config = AgentConfig::new("You are helpful")
    .with_prompt_caching(false);

​

Monitoring

let config = AgentConfig::new("You are helpful")
    .with_debug(true);

Cache creation tokens: 5000
Cache read tokens: 12000
Input tokens: 500

let response = llm.send_message(...).await?;

println!("Cache creation: {}", response.usage.cache_creation_input_tokens);
println!("Cache read: {}", response.usage.cache_read_input_tokens);
println!("Regular input: {}", response.usage.input_tokens);

​

Limitations

• Minimum cacheable prefix: 1024 tokens
• Cache expiry: 5 minutes of inactivity
• Dynamic prompts: Changing the system prompt or tools between calls invalidates the cache
• Short conversations: Overhead may exceed savings for single-turn interactions

​

Next Steps