Skip to main content
synth-ai experiment manages experiment submissions to the local queue system. Use this command to submit GEPA and MIPRO prompt learning experiments that will be processed by the queue worker.

​
Overview

The experiment queue allows you to:
  • Submit multiple experiments for batch processing
  • Control parallelism to manage resource usage
  • Monitor experiment progress in real-time
  • Cancel experiments and manage job execution

​
Prerequisites

  1. Queue Worker Running: Start the queue worker first 
    synth-ai queue start
  2. Environment Variables: Set required environment variables 
    export EXPERIMENT_QUEUE_DB_PATH="$HOME/.experiment_queue/db.sqlite"

​
Quick Start

# 1. Start the queue worker
synth-ai queue start

# 2. Submit an experiment
synth-ai experiment submit experiment.json

# 3. Monitor progress
synth-ai experiments --watch

# 4. Check specific experiment
synth-ai experiment status <experiment_id>

​
Commands

​
experiment submit

Submit a new experiment to the queue. 
synth-ai experiment submit <file|-> [--inline]
Arguments:
  • <file> - Path to JSON file with experiment specification, or - to read from stdin
Options:
  • --inline - Treat the argument as inline JSON instead of a file path
Example JSON Format: 
{
  "name": "My GEPA Experiment",
  "description": "Testing GEPA optimization",
  "parallelism": 2,
  "jobs": [
    {
      "job_type": "gepa",
      "config_path": "path/to/config.toml",
      "config_overrides": {
        "prompt_learning.gepa.rollouts": 100
      }
    }
  ]
}
Examples: 
# Submit from file
synth-ai experiment submit experiment.json

# Submit inline JSON
synth-ai experiment submit --inline '{"name": "Test", "jobs": [{"job_type": "gepa", "config_path": "config.toml"}]}'

# Submit from stdin
cat experiment.json | synth-ai experiment submit -

​
experiment list

List experiments with optional status filtering. 
synth-ai experiment list [--status STATUS] [--limit N] [--json]
Options:
  • --status STATUS - Filter by status (can be used multiple times): QUEUED, RUNNING, COMPLETED, FAILED, CANCELED
  • --limit N - Maximum number of experiments to list (default: 20)
  • --json - Emit JSON output instead of tables
Examples: 
# List all experiments
synth-ai experiment list

# List only running experiments
synth-ai experiment list --status RUNNING

# List completed and failed experiments
synth-ai experiment list --status COMPLETED --status FAILED

# Get JSON output
synth-ai experiment list --json

​
experiment status

Show detailed status for a specific experiment. 
synth-ai experiment status <experiment_id> [--watch] [--json] [--interval SECONDS]
Options:
  • --watch - Continuously refresh the status view
  • --json - Emit JSON payload instead of formatted output
  • --interval SECONDS - Refresh interval for watch mode (default: 2.0)
Examples: 
# Get current status
synth-ai experiment status exp_abc123

# Watch status updates
synth-ai experiment status exp_abc123 --watch

# Get JSON output
synth-ai experiment status exp_abc123 --json

​
experiment results

Show aggregate results for a completed experiment. 
synth-ai experiment results <experiment_id> [--json]
Options:
  • --json - Emit JSON payload instead of formatted output
Example: 
synth-ai experiment results exp_abc123

​
experiment cancel

Cancel an experiment and revoke in-flight jobs. 
synth-ai experiment cancel <experiment_id>
Example: 
synth-ai experiment cancel exp_abc123

​
experiments

Show live dashboard view of experiments. 
synth-ai experiments [--status STATUS] [--recent N] [--watch] [--json] [--interval SECONDS]
Options:
  • --status STATUS - Filter by status (can be used multiple times)
  • --recent N - Number of recent experiments to show (default: 5)
  • --watch - Continuously refresh the dashboard
  • --json - Emit JSON instead of tables
  • --interval SECONDS - Refresh interval for watch mode (default: 2.0)
Examples: 
# Show dashboard
synth-ai experiments

# Watch mode with auto-refresh
synth-ai experiments --watch

# Filter by status
synth-ai experiments --status RUNNING --status QUEUED

# JSON output
synth-ai experiments --json

​
Experiment JSON Format

​
Required Fields

  • name (string) - Experiment name
  • jobs (array) - Array of job specifications

​
Optional Fields

  • description (string) - Experiment description
  • parallelism (integer) - Maximum concurrent jobs (default: 1, max: 8)
  • metadata (object) - Custom metadata

​
Job Specification

Each job in the jobs array requires:
  • job_type (string) - Either "gepa" or "mipro"
  • config_path (string) - Path to TOML configuration file
  • config_overrides (object, optional) - Override config values using dot notation

​
Config Overrides

Use dot notation to override nested config values: 
{
  "config_overrides": {
    "prompt_learning.gepa.rollouts": 100,
    "prompt_learning.termination_config.max_cost_usd": 10.0,
    "prompt_learning.termination_config.time_limit_seconds": 3600
  }
}

​
Examples

​
Basic GEPA Experiment

{
  "name": "Banking77 GEPA",
  "description": "GEPA optimization for Banking77 classification",
  "parallelism": 1,
  "jobs": [
    {
      "job_type": "gepa",
      "config_path": "examples/banking77_gepa.toml",
      "config_overrides": {
        "prompt_learning.gepa.rollouts": 50
      }
    }
  ]
}

​
Multiple Jobs with Parallelism

{
  "name": "Multi-Job Experiment",
  "parallelism": 2,
  "jobs": [
    {
      "job_type": "gepa",
      "config_path": "config1.toml"
    },
    {
      "job_type": "mipro",
      "config_path": "config2.toml"
    },
    {
      "job_type": "gepa",
      "config_path": "config3.toml"
    }
  ]
}

​
With Environment File Override

{
  "name": "Experiment with Custom Env",
  "jobs": [
    {
      "job_type": "gepa",
      "config_path": "config.toml",
      "config_overrides": {
        "prompt_learning.env_file_path": "/path/to/.env"
      }
    }
  ]
}

​
Workflow

  1. Start Queue Worker: synth-ai queue start
  2. Submit Experiment: synth-ai experiment submit experiment.json
  3. Monitor Progress: synth-ai experiments --watch or synth-ai experiment status <id> --watch
  4. View Results: synth-ai experiment results <id> when complete
  5. Stop Worker: synth-ai queue stop when done

​
Status Values

  • QUEUED - Experiment is queued, waiting for worker to process
  • RUNNING - Experiment is currently running
  • COMPLETED - All jobs completed successfully
  • FAILED - One or more jobs failed
  • CANCELED - Experiment was canceled

​
Troubleshooting

​
Experiment Stuck in QUEUED

Symptoms: Experiment remains in QUEUED status Checklist:
  1. Verify worker is running: synth-ai queue status
  2. Check Beat scheduler is enabled (default)
  3. Review worker logs: tail -f logs/experiment_queue_worker.log

​
Job Validation Errors

Error: Config file not found or Environment file not found Solution: Ensure all paths in the experiment JSON are absolute or relative to the current working directory. The config file and any referenced environment files must exist.

​
Parallelism Limits

Error: Input should be less than or equal to 8 Solution: The parallelism field has a maximum value of 8. Reduce the parallelism value in your experiment JSON.

​
Next Steps