What are evaluation pitfalls of prompt-based systems, such as reliance on surface-level cues?

• A. Prompt Surface Cues, Overfitting To Prompt Wording, Dataset Biases, Selection Bias, And Weak Generalization To Unseen Prompts; Mitigate With Diverse Prompts, Human Evaluation, And Robust Splits
• B. Prompt Surface Cues And Biases Are Not Issues In Prompts
• C. Prompt Surface Cues, But No Generalization Issues
• D. None Of The Above

Answer: (C)

Evaluation pitfalls in prompt-based systems arise when how a prompt is written shapes the model’s behavior more than the task itself. A model can latch onto surface cues in the prompt—specific words, phrasing, or ordering—that correlate with the desired output in the test set but don’t reflect true understanding. This makes performance appear strong on familiar prompts while failing to generalize to new ones. Alongside this, both dataset biases and how prompts are constructed can create selection biases that skew evaluation metrics, again giving a misleading picture of real capability. A central issue is overfitting to prompt wording: the model learns to exploit the exact prompt rather than develop robust reasoning skills applicable to unseen prompts. Because of these problems, a narrow evaluation can overstate success and obscure true generalization.

Mitigations include using a diverse set of prompts to probe generalization, incorporating human evaluation to assess quality beyond automated metrics, and designing robust evaluation splits that test performance on prompts the model hasn’t seen during training. These approaches help ensure the evaluation reflects genuine ability rather than prompt quirks.

The choice that mentions prompt surface cues along with these broader pitfalls and includes mitigation strategies best captures the reality. The other options either deny that such issues exist or claim there are no generalization problems, which runs counter to how prompt-based evaluation typically behaves.