LLM Evaluation Framework

# LLM Evaluation Framework

You are an AI quality engineer. Design a rigorous evaluation framework for an LLM-powered feature.

## The Feature

Feature description: [WHAT THE AI FEATURE DOES — e.g., summarizes customer emails, classifies support tickets, generates product descriptions]
Model used: [e.g., Claude, GPT-4, or a fine-tuned model]
Input format: [DESCRIBE TYPICAL INPUT]
Expected output: [DESCRIBE IDEAL OUTPUT]
Production risk level: [LOW / MEDIUM / HIGH — consequences of a bad output]

## Test Case Design

Design 5 test case categories with 3 examples each:
1. **Golden path** — Typical, clean inputs that should always work well
2. **Edge cases** — Unusual but valid inputs
3. **Adversarial inputs** — Inputs designed to trip up the model
4. **Out-of-scope inputs** — Things the model should refuse or redirect
5. **Regression cases** — Known past failures to prevent regressions

## Evaluation Metrics

Define the evaluation metrics for this feature. For each metric:
- Name and definition
- How to measure it (human rater rubric, automated check, or LLM-as-judge prompt)
- Acceptable threshold to ship

## Evaluation Pipeline

Design the end-to-end eval pipeline: how to run evals, how to score results, and what triggers a block on shipping.