Explain attention mechanism and why it enables transformer models to capture long-range dependencies.

• A. Attention only considers local neighborhoods.
• B. Attention computes weighted sums over all positions to focus on relevant tokens; allows long-range dependencies.
• C. Attention reduces dimensionality.
• D. Attention ignores positional information.

Answer: (B)

Attention lets the model decide, for each position, which other positions in the sequence matter most to form its representation. It does this by computing a score between a query for the current position and keys for all positions, then converting those scores into weights with softmax. The final representation is a weighted sum of the corresponding values. Because this sum can draw information from every position, a token anywhere in the sequence can directly influence another token’s representation, so long-range dependencies are captured without having to pass information step by step through many layers or time steps. This global, content-based weighting lets the model focus on the most contextually relevant tokens, even if they’re far away. At the same time, positional information is still provided to distinguish order, so the mechanism doesn’t ignore position.