Skip to main content

Model Comparison

Model Comparison sits beside Evaluations and answers a single question: for the workloads currently running in the fleet, how do the models in use compare? Once traces from multiple models have been ingested, the view renders a side-by-side breakdown so you can switch models with data instead of a guess.

The window is fixed at last 24 hours. If only one model has produced traces in that window, the page shows an empty state — comparison needs at least two model cohorts.

Dimensions​

Every model cell on the page exposes three categories of signal:

  • Quality — Aggregated scores from any Evaluation pipeline that ran against the model's traces. Rendered per-metric (faithfulness, answer relevance, custom scores).
  • Cost — Total USD, mean per trace, and per-1k-token rates. Pulled from the Gateway's per-model pricing.
  • Performance — P50 / P95 latency, total tokens, and error rate.

Because the three categories come from the same underlying traces, Model Comparison lets you make apples-to-apples trade-off calls — for example, "Model A is 4% worse on faithfulness but 30% cheaper and 2× faster; the trade-off holds for the RAG corpus."

Fair Comparison​

Aggregates are only fair when the inputs match. Two patterns work:

  • Same dataset across models — Run an Evaluation against a saved Dataset with each candidate model swapped in. Model Comparison then reads the evaluator scores alongside the live metrics.
  • Same agent, time-sliced models — Run Model A for a period, switch the agent's model to Model B, and compare the two windows. Lower-friction than datasets but subject to input drift.

Best Practices​

  • Pair Model Comparison with Anomaly Alerts on Cost and P95 Latency. A model swap that wins on quality but blows a cost threshold should surface immediately.
  • Ignore Model Comparison when the agent is in heavy development. Prompt changes and model changes together produce noise; freeze the prompt before comparing.
  • Keep at least one week of overlap when switching production traffic between models. Same-window comparison is always more trustworthy than serial comparison.