Datasets
A dataset is a saved slice of traces that an evaluation pipeline runs against. Saving a dataset makes an evaluation reproducible — you can re-score the same inputs after a model or prompt change and compare results directly, instead of scoring different batches of live traffic each time.
What a Dataset Can Contain
A dataset is always defined as a filter over existing traces. The filter is composed of any combination of:
- Agent — One or more agents whose traces should be included.
- Model — Restrict to traces that used a specific model (e.g.
gpt-4o,claude-3.5-sonnet). - Status —
OK,ERROR, or both. - Time window — Absolute start/end, or a rolling window (
last 7 days). - Attribute match — Any custom OpenTelemetry attribute recorded on the trace (e.g.
tenant-id,session-id).
A saved dataset snapshots the filter, not the underlying traces. Each time a pipeline runs against the dataset, it re-resolves the filter against the current trace store. Use a fixed time window in the filter to freeze the trace set for regression testing.
Creating a Dataset
Datasets are created from the Evaluations creation form — when the pipeline asks which traces to score, the Dataset dropdown lets you pick an existing dataset or create a new one inline. The new-dataset form asks for a name, description, and the filter above.
Reusable datasets make sense when:
- You will run the same evaluation repeatedly (e.g. on every release).
- You want different evaluators to run against the same inputs (e.g. RAGAS + LLM-as-a-judge on the same corpus).
- You want to compare models fairly in Model Comparison.
Best Practices
- For regression testing, pin a fixed time window and a fixed agent filter. Drifting filters make run-to-run comparison meaningless.
- Keep dataset sizes tractable (hundreds to low thousands of traces). Judges and RAGAS both scale with size and cost.
- Name datasets with the scenario they represent, not the date they were created (
rag-faithfulness-gold-setbeatsdataset-apr-09).