Skip to main content
Benchmarks is in beta. Reach out to Sphinx to activate it for your workspace. Include your workspace name. The interface and behavior may change during the beta.
Use Benchmarks to test Sphinx on questions your team cares about. You define questions and criteria for a good answer, run the benchmark, and inspect the generated answers and judge results.

Before you start

  • Ask Sphinx to activate Benchmarks for your workspace.
  • Prepare representative questions and confirmed criteria for correct answers.
  • Select the workspace whose knowledge you want to evaluate.

Open Benchmarks

Open Settings at the bottom left and select Benchmarks. The page contains Average score trend, Questions, and Benchmark runs.
Benchmarks page with average score trend, question list, Add question, and Run benchmark controls

Benchmarks in a demo account with synthetic data.

Add questions and criteria

  1. Click Add question.
  2. Enter the question in Question.
  3. Enter a specific requirement in Criterion 1. Use Add criterion for additional requirements. Each criterion is scored separately, and questions have equal weight in the final score.
  4. Click Add question to save it.
For example: Use the search field to find saved questions. Each question has controls to view, edit, or delete it.
Write questions with enough context to have a clear expected answer. If different teams use different definitions, name the team or workflow. Review criteria with the knowledge owner so an outdated expected answer does not turn a valid response into a misleading failure.

Run and inspect a benchmark

  1. Click Run benchmark.
  2. Follow the run under Benchmark runs. Use Status to filter runs.
  3. Open a completed run to see Benchmark run results. You can also click a run in the average score trend.
  4. Review the number of test cases and the Mean, Median, Lowest, and Highest scores.
  5. For each test case, inspect the Given answer, each Criterion, and the Judge result, including the score and explanation.
Scores are displayed on a 0–100 scale. A high score means the answer satisfied the configured criteria according to the judge. Read individual results before drawing a conclusion about the knowledge base.

Evaluate generalization

Include representative questions that were not used to develop your knowledge base. Keep the questions, criteria, model, and agent setup comparable when measuring a change. For a broader agent evaluation, compare answer quality alongside cost and latency; the score trend alone does not describe those tradeoffs. See Learning organizational context.

Track changes over time

Use 1d, 7d, or 30d in Average score trend to change the displayed time range. Question-level trends help you identify regressions that an overall average can hide. When a score falls, check both the answer and the criterion. Correct missing or conflicting knowledge through ingest or controversy resolution, review the resulting changes, and rerun the benchmark. If the expected answer changed, update the criterion deliberately and account for that change when comparing runs.