What is a key limitation of ROUGE metrics?

• A. ROUGE metrics measure lexical overlap.
• B. ROUGE scores are unaffected by lexical overlap.
• C. ROUGE metrics only use the length of the summary.
• D. ROUGE metrics rely on lexical overlap and may not capture quality.

Answer: (D)

ROUGE evaluates automatically by counting how much lexical overlap there is between the produced summary and reference summaries. This makes it simple to compute and often correlates with human judgments in some cases, but it has a key limitation: it may not capture quality aspects such as fluency, coherence, factual accuracy, or true paraphrase ability because it only looks at word-level overlap. A well-done abstractive summary can paraphrase ideas with different wording and still be valuable, yet ROUGE may score it poorly if the exact overlapping phrases aren’t present. Conversely, a longer, more overlap-heavy output can score high even if it’s repetitive or less clear. The other options describe features or incorrect assumptions about ROUGE’s behavior, whereas the core limitation centers on relying on lexical overlap and potentially missing true quality.