AI chat cost control

Control cost with model sizing, prompt caching and batched evaluation. Track price per resolved conversation, not price per token.

Price per resolved conversation

Token prices are the wrong unit. The metric that matters is cost per conversation that ended without escalation.

• Size the model to the bucket. Easy intents on a small model can cost a tenth of the big one.
• Cache the system prompt. Static instructions should not be re-billed every message.
• Batch evaluation. Golden-set runs happen off-peak and in bulk.
• Kill the loops. A retry storm on an outage is pure spend with zero value.

Watch the ratio of tokens in versus answers out. If it's climbing, the prompts are growing - trim them.