What is instruction following evaluation and how is it measured in prompt-tuned models?

• A. It measures model speed.
• B. It assesses how well the model follows natural-language instructions; measured by task completion rate, output quality, and alignment; benchmarks and human eval.
• C. It is purely qualitative with no benchmarks.
• D. It measures memory usage.

Answer: (B)

Instruction following evaluation measures how faithfully a model adheres to natural-language instructions and delivers the requested output. In prompt-tuned models, this is quantified by task completion rate (did the model accomplish the defined goal across tasks), output quality (accuracy, fluency, relevance, and usefulness of the response), and alignment (consistency with the instruction and avoidance of undesired behavior). Benchmarks and human evaluation are used to quantify these aspects: standardized instruction-following benchmarks provide automatic scores across many tasks, while human raters judge alignment and quality where automatic metrics fall short. This approach gives a robust view of how well the model follows instructions, beyond speed or memory usage.