How We Test AI Summarizers

1. Key findings captured (2 points)

The summary correctly identifies the primary conclusions, results, or main argument of the source document. Deductions for missing major findings or mischaracterizing conclusions.

2. Methods described accurately (1.5 points)

For research papers: the approach, dataset, or analytical method is correctly described. For meeting notes: the decisions and their rationale. For legal documents: the operative clauses and their conditions.

3. Figures, tables, and data referenced (1.5 points)

Key quantitative findings, charts, or data tables in the source are reflected in the summary. Full points require at least 70% of significant data points mentioned.

4. Nuance and conditionality preserved (2 points)

The single most important criterion for high-stakes content. Deductions for: inverting conditional statements (omitting 'unless', 'except', 'subject to'), collapsing caveats into certainty, removing hedge language where the hedge is material.

5. Disciplinary / domain terminology accuracy (1.5 points)

Specialist terms are used correctly. For academic papers: no substitution of incorrect synonyms. For legal: correct clause-type labels. For medical: correct anatomical and pharmacological terminology.

6. Output format usefulness (1.5 points)

Is the output format appropriate for the use case? Meeting summaries should have action items and owners. Research papers should distinguish findings from methods. Contracts should flag clauses by type. Free-form prose gets half credit for this criterion regardless of quality.