Experiment Queue Management

Manage Celery workers for processing experiment queue jobs locally.

synth-ai queue manages the Celery worker that processes experiment jobs locally. The queue system allows you to submit multiple prompt learning experiments (GEPA/MIPRO) and have them processed automatically by a background worker.

Overview

The experiment queue system consists of:

• Queue Worker: A Celery worker that processes jobs from the queue
• Beat Scheduler: Periodically checks for queued jobs and dispatches them (runs every 5 seconds)
• Database: SQLite database stores experiment metadata and job status
• Redis Broker: Redis handles task queuing and result storage

Prerequisites

1. Redis: Install and start Redis

   # macOS
   brew install redis
   brew services start redis
   
   # Verify Redis is running
   redis-cli ping  # Should return: PONG
   

2. Environment Variables: Set required environment variables

   export EXPERIMENT_QUEUE_DB_PATH="$HOME/.experiment_queue/db.sqlite"
   export EXPERIMENT_QUEUE_BROKER_URL="redis://localhost:6379/0"  # Optional, defaults to this
   export EXPERIMENT_QUEUE_RESULT_BACKEND_URL="redis://localhost:6379/1"  # Optional, defaults to this
   

Quick Start

# 1. Start the queue worker
synth-ai queue start

# 2. Submit an experiment (see experiment command docs)
synth-ai experiment submit experiment.json

# 3. Check status
synth-ai queue status

# 4. Stop the worker when done
synth-ai queue stop

Commands

queue start

Start the experiment queue worker in the background.

synth-ai queue start [OPTIONS]

Options:

• --concurrency INTEGER - Worker concurrency (default: 1)
• --loglevel TEXT - Celery log level: debug, info, warning, error (default: info)
• --pool {solo|prefork|threads|gevent|eventlet} - Worker pool type (default: prefork)
• --background / --foreground - Run in background or foreground (default: --background)
• --beat / --no-beat - Run Celery Beat scheduler (default: --beat)
• --extra TEXT - Extra arguments forwarded to celery worker (can be used multiple times)

Examples:

# Start worker in background with default settings
synth-ai queue start

# Start worker in foreground for debugging
synth-ai queue start --foreground

# Start worker with debug logging
synth-ai queue start --loglevel debug

# Start worker without Beat scheduler (manual dispatch only)
synth-ai queue start --no-beat

Behavior:

• Automatically kills any existing workers before starting
• Validates that EXPERIMENT_QUEUE_DB_PATH is set
• Ensures only one worker instance runs at a time
• Logs are written to logs/experiment_queue_worker.log in background mode
• Beat scheduler runs periodic queue checks every 5 seconds

queue stop

Stop all running experiment queue workers.

synth-ai queue stop

Example:

synth-ai queue stop
# Output: ✅ Stopped 1 worker(s)

queue status

Show status of experiment queue workers and configuration.

synth-ai queue status
# or simply:
synth-ai queue

Example Output:

============================================================
Experiment Queue Status
============================================================

Configuration:
  EXPERIMENT_QUEUE_DB_PATH: /Users/me/.experiment_queue/db.sqlite
  EXPERIMENT_QUEUE_BROKER_URL: redis://localhost:6379/0 (default)
  EXPERIMENT_QUEUE_RESULT_BACKEND_URL: redis://localhost:6379/1 (default)

✅ 1 worker(s) running:

Worker 1:
  PID: 12345
  Database: /Users/me/.experiment_queue/db.sqlite
  Broker: redis://localhost:6379/0
  Result Backend: redis://localhost:6379/1

Environment Variables

Variable	Required	Default	Description
EXPERIMENT_QUEUE_DB_PATH	✅ Yes	None	Path to SQLite database file for experiment metadata
EXPERIMENT_QUEUE_BROKER_URL	No	redis://localhost:6379/0	Redis URL for Celery broker
EXPERIMENT_QUEUE_RESULT_BACKEND_URL	No	redis://localhost:6379/1	Redis URL for Celery result backend
EXPERIMENT_QUEUE_TRAIN_CMD	No	Auto-detected	Training command override (defaults to venv Python)

Workflow

1. Start Worker: synth-ai queue start launches the worker
2. Submit Experiments: Use synth-ai experiment submit to add jobs to the queue
3. Automatic Processing: Beat scheduler dispatches jobs every 5 seconds
4. Monitor: Use synth-ai queue status or synth-ai experiments --watch to monitor progress
5. Stop Worker: synth-ai queue stop when done

Troubleshooting

Worker won’t start

Error: EXPERIMENT_QUEUE_DB_PATH environment variable must be set

Solution: Set the environment variable before starting:

export EXPERIMENT_QUEUE_DB_PATH="$HOME/.experiment_queue/db.sqlite"
synth-ai queue start

Redis connection errors

Error: Error connecting to Redis

Solution: Ensure Redis is running:

# Check if Redis is running
redis-cli ping

# Start Redis if needed
brew services start redis  # macOS
# or
redis-server  # Linux

Jobs not being dispatched

Symptoms: Jobs remain in QUEUED status

Checklist:

1. Verify worker is running: synth-ai queue status
2. Check Beat scheduler is enabled: synth-ai queue start --beat (default)
3. Check Redis connectivity: redis-cli ping
4. Review worker logs: tail -f logs/experiment_queue_worker.log

Multiple workers detected

Warning: Some workers are using different database paths!

Solution: The system enforces a single database path. Stop all workers and restart:

synth-ai queue stop
export EXPERIMENT_QUEUE_DB_PATH="/path/to/your/db.sqlite"
synth-ai queue start

Logs

Worker logs are written to logs/experiment_queue_worker.log when running in background mode. To monitor logs in real-time:

tail -f logs/experiment_queue_worker.log

Related Commands

• synth-ai experiment submit - Submit experiments to the queue
• synth-ai experiments - View experiment dashboard
• synth-ai train - Direct training (bypasses queue)

Next Steps

• Submit experiments to the queue
• Monitor experiment progress