Skip to main content
This cookbook shows how to train a policy graph for intent classification using the Banking77 dataset - 77 unique banking-related customer intents.

​
The Task

Classify customer messages into one of 77 banking intents: 
Input: "When will my card arrive?"
Output: "card_arrival"

Input: "My card payment was declined."
Output: "declined_card_payment"

​
Dataset Format

The Banking77 ADAS dataset follows this structure: 
{
  "version": "1.0",
  "metadata": {
    "name": "banking77-classification",
    "description": "Intent classification for banking customer queries",
    "task_description": "Classify the customer message into one of 77 banking intents"
  },
  "tasks": [
    {
      "id": "banking_001",
      "input": {
        "customer_message": "When will my card arrive?"
      }
    },
    {
      "id": "banking_002",
      "input": {
        "customer_message": "My card payment was declined."
      }
    },
    {
      "id": "banking_003",
      "input": {
        "customer_message": "I need to change my PIN"
      }
    }
  ],
  "gold_outputs": [
    {
      "task_id": "banking_001",
      "output": {
        "intent": "card_arrival"
      }
    },
    {
      "task_id": "banking_002",
      "output": {
        "intent": "declined_card_payment"
      }
    },
    {
      "task_id": "banking_003",
      "output": {
        "intent": "change_pin"
      }
    }
  ],
  "judge_config": {
    "mode": "rubric"
  }
}

​
Training the Graph

​
Using the CLI

uvx synth-ai train \
  --dataset banking77_dataset.json \
  --policy-model gpt-4o-mini \
  --rollout-budget 200

​
Using Python

from synth_ai.sdk.api.train.adas import ADASJob

job = ADASJob.from_dataset(
    "banking77_dataset.json",
    policy_model="gpt-4o-mini",
    rollout_budget=200,
)
job.submit()

# Stream progress
for event in job.stream_events():
    if event["type"] == "generation_complete":
        print(f"Gen {event['data']['generation']}: {event['data']['best_score']:.2f}")

result = job.wait()
print(f"Best accuracy: {result['best_score']:.2%}")

​
Available Intent Labels

The 77 Banking77 intents include:
CategoryExample Intents
Cardcard_arrival, card_linking, card_not_working, card_swallowed, declined_card_payment
Accountactivate_my_card, age_limit, apple_pay_or_google_pay, atm_support
Transferbalance_not_updated_after_bank_transfer, beneficiary_not_allowed, cancel_transfer
PINchange_pin, compromised_card, contactless_not_working
…77 total intents

​
Running Inference

Once trained, classify new messages: 
job = ADASJob.from_existing("adas_abc123", api_key=api_key)

result = job.run_inference({
    "customer_message": "How do I add money to my account?"
})

print(result["output"]["intent"])  # "top_up"

​
Batch Classification

import asyncio

messages = [
    {"customer_message": "Cancel my card please"},
    {"customer_message": "Where is my refund?"},
    {"customer_message": "I lost my card"},
]

results = asyncio.run(job.run_batch(messages))
for msg, res in zip(messages, results):
    print(f"{msg['customer_message'][:30]}... → {res['output']['intent']}")

​
Evaluation Metrics

Banking77 uses exact match scoring:
  • Score = 1.0 if predicted intent exactly matches gold intent
  • Score = 0.0 otherwise
The judge normalizes intent names to snake_case before comparison.

​
Configuration Tips

​
For Higher Accuracy

job = ADASJob.from_dataset(
    "banking77_dataset.json",
    policy_model="gpt-4o",  # Stronger model
    rollout_budget=500,      # More optimization budget
    graph_structure="dag",   # Allow chain-of-thought
)

​
For Lower Cost

job = ADASJob.from_dataset(
    "banking77_dataset.json",
    policy_model="gpt-4o-mini",
    rollout_budget=100,
    graph_structure="single_prompt",
)

​
Downloading the Graph

Export your trained graph for inspection or local use: 
graph_yaml = job.download_prompt()
print(graph_yaml)
Example output: 
graph_type: policy
nodes:
  - id: classifier
    type: llm
    model: gpt-4o-mini
    prompt: |
      Classify this banking customer message into one of the following intents:
      {available_intents}

      Customer message: {customer_message}

      Respond with only the intent name in snake_case.
    output_schema:
      type: object
      properties:
        intent:
          type: string