Performance

Response caching

Return a saved answer for repeated or similar requests, without calling the model.

Exact-match cacheSemantic cachePer-project isolation

What is Response caching?

Response caching is an AI gateway feature that stores model responses and returns them for matching requests without calling the model again. Spro offers exact-match caching, which matches identical requests, and semantic caching, which matches requests with the same meaning. Caching is opt-in per request with the x-spro-cache-config header.

Many AI apps send the same questions again and again: FAQs, product lookups and repeated prompts in tests. Each call costs tokens and waits for the model. A response cache answers those calls from storage, so the model is not called at all.

Spro keeps the cache at the gateway, in front of every provider. A cache hit returns in milliseconds and uses no tokens. The cache is off unless the request sends the cache header, and every entry is scoped to the project that created it.

Key features

What this feature does for your requests.

  • Exact-match cache keyed on messages, model and sampling settings
  • Semantic cache that matches the meaning of the last message
  • Time to live per entry: 600 seconds by default, up to 86400
  • Similarity threshold that controls how close a semantic match must be
  • x-spro-cache header reports hit, miss, bypass or error

How it works

4 steps, from setup to the first request.

  1. 01

    Send the cache header

    Add x-spro-cache-config with the type and ttl as JSON.

  2. 02

    First call stores

    A miss calls the model and saves the response.

  3. 03

    Repeat calls hit

    Matching requests return the saved response with no model call.

  4. 04

    Check the result

    Read x-spro-cache on each response: hit, miss or bypass.

Is it the right fit?

Specific situations where this feature helps, and where it does not.

Good fit if

  • FAQ or support bots that get the same questions every day
  • Test suites that replay identical prompts on every run
  • Lookups whose answer does not change within the cache lifetime

Not a good fit if

  • Answers that must reflect live data or change within minutes
  • Personal answers, where one user's cached reply must not reach another user
  • Creative generation, where variety is the point

Response caching FAQs

Common questions about Response caching.

Route every AI request through SPRO.

One OpenAI-compatible API for 350+ models, with intelligent routing, 29 built-in guardrails, observability, and automatic failover.

Talk to sales