GPT-4o
⚙️ Technical
Advanced
Local AI Model Benchmark Framework
Design a systematic benchmarking framework to compare multiple local models on your specific tasks and hardware.
The Prompt
# Local AI Model Benchmark Framework You are an ML evaluation engineer. Design a rigorous benchmark to compare local models for a specific task and hardware setup. ## Benchmark Scope - Models to compare: [LIST 3–5 MODEL NAMES WITH SIZES] - Hardware under test: [CPU, RAM, GPU, VRAM] - Primary task: [TASK TYPE, e.g., "instruction following", "code generation", "document summarization"] - Success metric priority: [SPEED / QUALITY / CONSISTENCY / COST] ## Framework Design ### Test Suite Define 20 test prompts across these categories: 5 simple tasks, 5 multi-step reasoning, 5 domain-specific to [PRIMARY TASK], and 5 adversarial edge cases. Provide 2 example prompts per category. ### Metrics For each model, measure: tokens/second, first-token latency (ms), VRAM peak usage, CPU usage (%), and a quality score (1–5) per response. Explain how to score quality consistently. ### Automation Script A Python script skeleton using subprocess + time to run each model via Ollama API, collect metrics, and write results to a CSV. ### Statistical Validity How many runs per prompt are needed for reliable results, and how to handle variance across runs. ### Scoring Matrix A weighted scoring formula combining all metrics based on the [SUCCESS METRIC PRIORITY]. Show the formula.
📝 Fill in the blanks
Replace these placeholders with your own content:
[LIST 3–5 MODEL NAMES WITH SIZES]
[CPU, RAM, GPU, VRAM]
[TASK TYPE, e.g., "instruction following", "code generation", "document summarization"]
[SPEED / QUALITY / CONSISTENCY / COST]
[PRIMARY TASK]
[SUCCESS METRIC PRIORITY]
How to use this prompt
1
Copy the prompt
Click "Copy Prompt" above to copy the full prompt text to your clipboard.
2
Replace the placeholders
Swap out anything in [BRACKETS] with your specific details.
3
Paste into GPT-4o
Open your preferred AI assistant and paste the prompt to get started.