Banking77 Factory: Results

Run a Banking77 prompt-improvement Factory and work with the Results it produces, including optional evaluation and current-best selection.

This walkthrough shows the Result lifecycle on a concrete objective: improving a Banking77 intent-classification prompt. The optimized prompt is a Result. Evaluation and current-best selection are optional metadata on that Result — you never rewrite or replace the Result to record them.

Banking77 is the example; the same Result surface works for any Factory output: report, prompt, policy, dataset, model, artifact, or draft code change.

1. Install and authenticate

uv add "synth-ai[research]"
export SYNTH_API_KEY="sk_..."
export SYNTH_RESEARCH_PROJECT_ID="project_..."

2. Read the Results a Factory produced

Every directly valuable output a Factory produces is a Result. List them from the Result surface; ordinary Results carry no evaluation and no selection.

import os
from synth_ai import SynthClient

research = SynthClient().research
factory_id = os.environ["BANKING77_FACTORY_ID"]

for result in research.factories.results.list(factory_id):
    print(result.result_id, result.kind, result.readiness, result.title)
    print("  content:", result.content_url)

Filter the listing by Effort, run, kind, readiness, evaluation status, or current-best state — for example, research.factories.results.list(factory_id, kind="prompt", current_best=True).

3. Attach a benchmark-owned evaluation

When a benchmark grades an optimized prompt, attach the grader’s record to the Result. The backend never grades; it stores exactly what the grader proved. The Result’s identity and content do not change.

result = research.factories.results.get(factory_id, result_id)

research.factories.results.evaluate(
    factory_id,
    result.result_id,
    evaluation={
        "evaluator": "factorybench",
        "objective": "held_out_accuracy",
        "score": 0.913,
        "baseline_score": 0.887,
        "verdict": "passed",
    },
)

graded = research.factories.results.get(factory_id, result.result_id)
print(graded.evaluation.status, graded.evaluation.score, graded.evaluation.verdict)

Only a Result backed by a registered candidate accepts an evaluation. An evaluation posted for the wrong Result identity is rejected.

4. Select the current best

Mark a passing Result as the current best for a named objective/scope. This is idempotent and historical: it appends a selection event and never deletes or rewrites the earlier Result.

decision = research.factories.results.select_current_best(
    factory_id,
    result_id=result.result_id,
    scope="banking77_accuracy",
    reason="highest passing held-out accuracy",
)
print(decision.changed, decision.result.selection.is_current_best)

# Append-only current-best history
for event in research.factories.results.list_selection_events(factory_id):
    print(event.action, event.result_id, event.scope, event.score)

Selecting another passing Result later changes the current-best state without deleting or rewriting this Result. research.factories.results.restore_current_best(...) re-selects a prior Result as current best.

5. Confirm the Result is conserved

The Result you graded and selected is the same Result throughout — selection is state layered on top of it, not a new object.

final = research.factories.results.get(factory_id, result.result_id)
assert final.result_id == result.result_id
print("kind:", final.kind)
print("evaluation:", final.evaluation.status, final.evaluation.score)
print("current best:", final.selection.is_current_best, "scope:", final.selection.scope)

MCP equivalent

The same lifecycle is available to an MCP client through smr_list_factory_results, smr_get_factory_result, smr_evaluate_factory_result, smr_select_factory_result_current_best, smr_restore_factory_result_current_best, and smr_list_factory_result_selection_events.

Legacy compatibility

The research.factories.candidates and research.factories.champions namespaces (and their smr_*_candidate / smr_*_champion MCP tools) remain available over the same backend authority for existing integrations. New code should use Results; candidate/champion are legacy optimization terminology.