Investigation: What Should an AI Evaluation Actually Measure?

Question

What should an evaluation measure when an AI system is part of a larger product workflow?

Current Understanding

Model accuracy matters, but it is not enough. The product can fail because retrieval selected weak context, because the task was ambiguous, because the fallback path was unclear, or because the user needed calibrated uncertainty rather than a confident answer.

Evidence Needed

• Stable task definitions.
• Versioned examples.
• Human review for ambiguous cases.
• Production signals that connect failure to user impact.

Working Hypothesis

Good AI evaluation will look less like a single score and more like an observability system for behavior.