rubrics

synth_ai.data.rubrics

Rubric and Criterion data models.

These are INPUT/DEFINITION structures that users create to define evaluation criteria. For OUTPUT/RESULT structures (scores after evaluation), see judgements.py.

Classes

CriterionExample

Example for a criterion showing expected scores.

Criterion

Single evaluation criterion within a rubric.

Flexible variant allowing weights > 1.0 and no normalization requirement. Used by task apps for general reward computation.

Attributes:

• id: Unique identifier for this criterion
• description: Human-readable description of what this criterion evaluates
• weight: Relative weight for scoring (must be positive, default 1.0)
• required: Whether this criterion must be satisfied for success

Rubric

Rubric definition for evaluating task app outcomes.

Supports flexible aggregation and blending. Criteria weights do not need to sum to 1.0, making this suitable for general task app usage.

Attributes:

• version: Version string for this rubric definition
• goal_text: Optional human-readable description of the evaluation goal
• criteria: List of Criterion objects defining evaluation criteria
• aggregation: How to aggregate criterion scores (‘sum’, ‘weighted_sum’, ‘custom’, ‘inherit’)