Explain what makes an evaluation dataset effective for NLP prompts.

• A. A good evaluation dataset is small, uniform, and tailored to one narrow task.
• B. A good evaluation dataset is diverse, representative of target use, and has clear, verifiable ground truth.
• C. A good evaluation dataset is biased toward the test set to inflate performance.
• D. A good evaluation dataset is diverse, representative of target use, free from leakage, balanced across classes, with clear, verifiable ground truth and multiple human raters if possible.

Answer: (D)

An effective evaluation dataset for NLP prompts must be diverse and representative of how the model will be used, with ground-truth labels that are clear and verifiable, and it should be designed to measure genuine capability rather than memorization. Diversity means including a wide range of prompts, styles, languages, and task variations so the evaluation tests how the model handles real-world variation, not just a narrow slice. Representativeness ensures the prompts reflect the target user base and typical use cases, so performance translates to actual deployments.

Crucially, the data should be free from leakage—no overlap between training data and evaluation prompts—so scores reflect true generalization rather than memorized content. Balancing across classes prevents skewed results toward the most common outputs, giving a fair view of how well the model handles all possible responses. Clear ground truth makes scoring consistent and reproducible, while multiple human raters (when possible) help quantify annotation reliability and reduce the impact of individual judgments, especially on nuanced or ambiguous prompts.

The other options fall short because they either narrow the test too much, inflate performance by peeking at the test data, or omit key safeguards that ensure valid, reliable evaluation.