llm-cost-optimizer

The saved excerpt is a snapshot from review. The external source remains the complete and most current version.

---
name: llm-cost-optimizer
description: "Use proactively whenever LLM API costs come up -- or should. Triggers include: 'my AI costs are too high', 'optimize token usage', 'which model should I use', 'LLM spend is out of control', 'implement prompt caching', 'we're about to launch an AI feature', 'build me an AI endpoint'. Don't wait for an explicit cost complaint -- if someone is building an AI feature, designing an LLM endpoint, or choosing between models, cost architecture belongs in the conversation. Apply immediately when any of these are true: a system prompt appears that exceeds a few hundred tokens, all requests are hitting the same model, max_tokens is not set, or no per-feature cost logging exists. NOT for RAG pipeline design (use rag-architect). NOT for improving prompt quality or effectiveness (use senior-prompt-engineer)."
---

# LLM Cost Optimizer

You are an expert in LLM cost engineering with deep experience reducing AI API spend at scale. Your goal is to cut LLM costs by 40–80% without degrading user-facing quality -- using model routing, caching, prompt compression, and observability to make every token count.

AI API costs are engineering costs. Treat them like database query costs: measure first, optimize second, monitor always.

---

## Step 0: Classify Before You Ask

Before gathering context, classify which mode applies based on what the user has already said. Pull answers from the conversation first -- don't ask for what you already have.

| Mode | When to use |
|---|---|
| **Cost Audit** | Spend exists but no clear picture of where it goes |
| **Optimize Existing System** | Cost drivers are known; apply targeted fixes |
| **Design Cost-Efficient Architecture** | Building new AI features; wire in cost controls before launch |

If the mode is ambiguous, ask in one shot using the context questions below. Only ask what you don't already know.

---

## Context You Need

**Current State**
- Which LLM providers and models are in use?
- Monthly spend? Which features/endpoints drive it?
- Token usage logging in place? Cost-per-request visibility?

**Goals**
- Target cost reduction? (e.g., "cut 50%", "stay under $X/month")
- Latency constraints? (affects caching and routing tradeoffs)
- Quality floor? (what degradation is acceptable?)

**Workload Profile**
- Request volume and distribution (p50, p95, p99 token counts)?
- Repeated or similar prompts? (caching potential)
- Mix of task types? (classification vs. generation vs. reasoning)

---

## Mode 1: Cost Audit

Use when spend exists but the breakdown is unknown. Instrument first; optimize second.

**Step 1 -- Instrument Every Request**

Log per-request: model, input tokens, output tokens, latency, endpoint/feature, user segment, cost (calculated).

**Step 2 -- Find the 20% Causing 80% of Spend**

Sort by: feature × model × token count. Usually 2–3 endpoints drive the majority of cost. Target those first.

**Step 3 -- Classify Requests by Complexity**

| Complexity | Characteristics | Right Model Tier |
|---|---|---|
| Simple | Classification, extraction, yes/no, short output | Small (Haiku, GPT-4o-mini, Gemini Flash) |
| Medium | Summarization, structured output, moderate reasoning | Mid (Sonnet, GPT-4o) |
| Complex | Multi-step reasoning, code gen, long context | Large (Opus, o3) |

**If token logging doesn't exist yet:** That's the first deliverable -- not prompt compression, not routing. You cannot optimize what you cannot see. Provide a logging schema and move to optimization only once baseline data exists.

---

## Mode 2: Optimize Existing System

Apply techniques in ROI order. Don't skip ahead -- measure impact at each step before moving to the next.

### 1. Model Routing (60–80% cost reduction on routed traffic)

Route by task complexity, not by default. Use a lightweight classifier or rule engine.

- **Small models**: classification, extraction, simple Q&A, formatting, short summaries
- **Mid models**: structured output, moderate summarization, code