Skip to content

Price OpenAI and Anthropic effective cost with llm-prices cache rates - #344

Open
Mattbusel wants to merge 1 commit into
ServiceNow:mainfrom
Mattbusel:llm-prices-cost-tracking
Open

Mattbusel wants to merge 1 commit into
ServiceNow:mainfrom
Mattbusel:llm-prices-cost-tracking

Conversation

@Mattbusel

Copy link
Copy Markdown

The effective cost that study summaries report as cum_effective_cost uses fixed cache factors: OpenAI cache reads at 0.5x input, but the gpt-5 family (including the GPT_5_mini and GPT_5_nano agents in tool_use_agent.py) bills them at 0.1x, so a gpt-5 call with 100k prompt tokens, 80k of them cached, and 2k output is recorded as $0.095 instead of $0.055. On the Anthropic side the code reads usage.cache_input_tokens, which the SDK never sets (the field is cache_read_input_tokens), so cache reads are priced at $0, and the fixed 0.1x read factor would be 2x too high for claude-opus-5-5 and claude-sonnet-5-5 anyway (their reads are 0.05x). This PR prices the openai and anthropic paths with llm-prices, a dated, source-cited price table with each model's cache read and 5m/1h cache write prices and long-context tiers, and passes the real cached and cache-write counts through Price.cost_usage; models without a row (claude-3.x, fine-tunes, self-hosted base_url models) keep the previous pricing path and factors, and the openrouter and litellm paths are untouched. It is a regular runtime dependency (pure Python, no dependencies of its own, no network access) because prices change and a copied table goes stale; I maintain it (MIT). tests/llm passes with 5 new cases checked against the providers' published prices, a two-step ToolUseAgent run (gpt-5-mini, mocked client) and a claude-opus-5-5 step record effective costs that match litellm's cost_per_token and Anthropic's price list exactly, and the rest of the suite gives the same results as main.

The effective cost used fixed cache factors (OpenAI cache reads at 0.5x
input, Anthropic reads at 0.1x and writes at 1.25x). gpt-5 family cache
reads are 0.1x and claude-opus-5-5 / claude-sonnet-5-5 reads are 0.05x,
so cached agent steps were mispriced. The Anthropic path also read
usage.cache_input_tokens, which the SDK does not set, so cache reads
were priced at $0.

OpenAI and Anthropic models are now priced from the llm-prices table
(each model's own input, output, cache read and 5m/1h cache write
prices, long-context tiers included). Models without a row keep the
previous pricing path and the fixed factors.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant