> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sphinx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks (beta)

> Evaluate Sphinx answers against your questions and criteria, inspect results, and track scores over time.

> [!NOTE]
>
> Benchmarks is in **beta**. Reach out to [Sphinx](mailto:support@sphinx.ai) to activate it for your workspace. Include your workspace name. The interface and behavior may change during the beta.

Use Benchmarks to test Sphinx on questions your team cares about. You define questions and criteria for a good answer, run the benchmark, and inspect the generated answers and judge results.

## Before you start

* Ask Sphinx to activate Benchmarks for your workspace.
* Prepare representative questions and confirmed criteria for correct answers.
* Select the workspace whose knowledge you want to evaluate.

## Open Benchmarks

Open **Settings** at the bottom left and select **Benchmarks**. The page contains **Average score trend**, **Questions**, and **Benchmark runs**.

![Benchmarks page with average score trend, question list, Add question, and Run benchmark controls](https://mintcdn.com/sphinx-977f1673/O9wCzUTcMLhLJ368/images/ui/benchmarks.jpg?fit=max\&auto=format\&n=O9wCzUTcMLhLJ368\&q=85\&s=011047cc35210cb4fedd7fbe0d0fc14d)

Benchmarks in a demo account with synthetic data.

## Add questions and criteria

1. Click **Add question**.
2. Enter the question in **Question**.
3. Enter a specific requirement in **Criterion 1**. Use **Add criterion** for additional requirements. Each criterion is scored separately, and questions have equal weight in the final score.
4. Click **Add question** to save it.

For example:

| Field | Demo content |
| - | - |
| Question | How long should someone listen to a song for it to count as a finance-qualified play? |
| Criterion 1 | The answer says at least 30 qualifying seconds. |
| Criterion 2 | The answer distinguishes the finance definition from product engagement definitions. |

Use the search field to find saved questions. Each question has controls to view, edit, or delete it.

> [!TIP]
>
> Write questions with enough context to have a clear expected answer. If different teams use different definitions, name the team or workflow. Review criteria with the knowledge owner so an outdated expected answer does not turn a valid response into a misleading failure.

## Run and inspect a benchmark

1. Click **Run benchmark**.
2. Follow the run under **Benchmark runs**. Use **Status** to filter runs.
3. Open a completed run to see **Benchmark run results**. You can also click a run in the average score trend.
4. Review the number of test cases and the **Mean**, **Median**, **Lowest**, and **Highest** scores.
5. For each test case, inspect the **Given answer**, each **Criterion**, and the **Judge result**, including the score and explanation.

Scores are displayed on a 0–100 scale. A high score means the answer satisfied the configured criteria according to the judge. Read individual results before drawing a conclusion about the knowledge base.

## Evaluate generalization

Include representative questions that were not used to develop your knowledge base. Keep the questions, criteria, model, and agent setup comparable when measuring a change. For a broader agent evaluation, compare answer quality alongside cost and latency; the score trend alone does not describe those tradeoffs. See [Learning organizational context](https://docs.sphinx.ai/concepts/context-learning.md).

## Track changes over time

Use **1d**, **7d**, or **30d** in **Average score trend** to change the displayed time range. Question-level trends help you identify regressions that an overall average can hide.

When a score falls, check both the answer and the criterion. Correct missing or conflicting knowledge through [ingest](https://docs.sphinx.ai/knowledge-base/ingest.md) or [controversy resolution](https://docs.sphinx.ai/knowledge-base/controversies.md), review the resulting changes, and rerun the benchmark. If the expected answer changed, update the criterion deliberately and account for that change when comparing runs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.