Run an Eval Job From Your Terminal

Create and run an eval job using Synth AI’s terminal user interface

This guide walks through creating an eval job directly from the Synth AI TUI.

Prerequisites

Install the Synth AI SDK in your Python environment:

pip install synth-ai

Launch the TUI

In the same environment where synth-ai is installed, run:

synth-ai

This opens the TUI where you can view existing jobs and create new ones.

Create a New Job

Press n to create a new job. You’ll be prompted to select or create a LocalAPI file.

LocalAPI File

A LocalAPI is a FastAPI app that defines your evaluation task. It includes:

• A dataset to evaluate against
• A rollout handler that calls your LLM and scores responses
• Task metadata for the Synth backend

You can use an existing LocalAPI file or create a new one.

For a working example, download the Banking77 classification task, either through cURL or from GitHub directly:

curl -L -o localapi.py \
  https://raw.githubusercontent.com/synth-laboratories/synth-ai/main/demos/gepa_banking77/localapi_banking77.py

If creating a new file, choose where to save it:

You can optionally open the file in your editor to review or customize it:

Select Job Type

After selecting your LocalAPI file, choose the job type. Select eval to run an eval job:

Job Execution

When you start the job, the TUI:

1. Loads and validates your LocalAPI module
2. Starts a local FastAPI server
3. Creates a Cloudflare Tunnel so Synth AI can reach your machine
4. Submits the eval job to Synth AI
5. Runs rollouts against your dataset and aggregates scores

View Results

The TUI displays the job status and results as the eval runs:

The eval details show:

• Progress: Completed rollouts out of total
• Mean Reward: Average score across all rollouts
• Avg Reward: Running average
• Pass Rate: Percentage of rollouts that passed