What is LoRA and how does it help in fine-tuning models?

• A. LoRA increases the total number of trainable parameters.
• B. LoRA decomposes weight updates into low-rank matrices to reduce trainable parameters.
• C. LoRA is a data augmentation technique.
• D. LoRA has no effect on training efficiency.

Answer: (B)

LoRA, or Low-Rank Adaptation, focuses on making fine-tuning large models parameter-efficient by expressing updates to weight matrices as low-rank products. Instead of adjusting every element of a big weight matrix, you keep the pretrained weights fixed (or mostly fixed) and learn a small set of parameters that form a low-rank addition to those weights. Concretely, you replace W with W plus a delta W, where delta W = A B for small matrices A and B, with rank r much smaller than the dimensions of W. This means the number of trainable parameters drops from the size of W to size(A) plus size(B), which is typically much smaller. During fine-tuning you train only A and B while W stays unchanged; at inference you can merge delta W into W or apply it as a fixed modification. This approach is especially useful for transformer projections (like attention or feed-forward layers) because it lets the model adapt to new tasks with a tiny parameter budget while preserving the original weights. The other options either describe increasing trainable parameters, a different technique, or claim no efficiency impact, which doesn’t capture what LoRA does.