QualityComing soon

Evaluations

Compare models on your own data, scored by an LLM judge or by criteria you define.

Coming soonYour own dataLLM-as-judge or custom criteria

What is Evaluations?

Evaluations test AI models on your own data and score the results, often with an LLM as the judge or with criteria you define. They help you choose a model or prompt based on evidence rather than impressions. Spro's evaluations are planned and not yet available.

Model choice is easy to get wrong when judged by a few examples. Evaluations run the same test set through each candidate model, score the answers and show the results side by side.

This page describes planned behaviour. Scoring methods and dataset formats will be published with the feature.

Key features

What this feature does for your requests.

  • Run a test set through several models
  • LLM-as-judge scoring
  • Custom scoring criteria
  • Side-by-side results

How it works

4 steps, from setup to the first request.

  1. 01

    Build a test set

    Collect real inputs with expected outcomes.

  2. 02

    Choose models to compare

    Pick the candidates to test.

  3. 03

    Run and score

    Each model answers and is scored.

  4. 04

    Compare the results

    Decide on evidence.

Is it the right fit?

Specific situations where this feature helps, and where it does not.

Good fit if

  • You are choosing a model or prompt for a feature with clear success criteria
  • You change models or prompts and need to catch regressions
  • You have a test set of real inputs

Not a good fit if

  • You have no examples of good answers yet
  • Quality is subjective and nobody can agree on criteria
  • You only need latency or cost numbers. Usage analytics cover those

Evaluations FAQs

Common questions about Evaluations.

Route every AI request through SPRO.

One OpenAI-compatible API for 350+ models, with intelligent routing, 29 built-in guardrails, observability, and automatic failover.

Talk to sales