Zero-Shot Verifiers

Evaluate execution traces using built-in Synth AI verifiers without training

Zero-Shot Verifiers

Zero-shot verifiers allow you to use Synth AI’s advanced evaluation logic (including MapReduce and RLM architectures) immediately, without providing a training dataset. These are ideal for rapid prototyping of custom verifiers.

Built-in Verifier IDs

When calling the /graphs/completions endpoint, you can use these reserved IDs to route to built-in verifier graphs:

Graph ID	Best For	Max Tokens
zero_shot_verifier	Automatic routing based on trace size	Varies
zero_shot_verifier_rubric_single	Standard traces (1-2 messages)	50,000
zero_shot_verifier_rubric_mapreduce	Long traces (10+ messages)	500,000
zero_shot_verifier_rubric_rlm	Massive contexts / tool-heavy traces	1,000,000+

Request Structure

Zero-shot verifiers accept a trace and an optional rubric.

import os
from synth_ai.sdk.graphs.completions import GraphCompletionsSyncClient

client = GraphCompletionsSyncClient(
    base_url="https://api.synth-ai.com",
    api_key=os.environ["SYNTH_API_KEY"],
)

response = client.run(
    job_id="zero_shot_verifier",
    input_data={
        "trace": {
            "session_id": "test_session_123",
            "session_time_steps": [...]
        },
        "rubric": {
            "outcome": {
                "criteria": [
                    {"name": "Accuracy", "description": "Is it correct?", "weight": 1.0}
                ]
            }
        }
    }
)
print(response.output)

Few-Shot In-Context Learning

You can improve zero-shot performance by providing calibration_examples directly in the inference request:

{
  "job_id": "zero_shot_verifier",
  "trace": { ... },
  "calibration_examples": [
    {
      "input": { "trace": { ... } },
      "output": { "score": 1.0, "reasoning": "Excellent response" }
    }
  ]
}