Fine-Tuning & LoRA

Question 1

LoRA stands for…

Accepted Answer

Low-Rank Adaptation — LoRA (Low-Rank Adaptation) injects trainable low-rank matrices into frozen transformer weights, achieving task adaptation with far fewer trainable parameters.

Question 2

In LoRA, the weight update is expressed as W' = W + BA. What are B and A?

Accepted Answer

Two low-rank matrices of rank r ≪ d — B and A are the two low-rank trainable matrices. Their product BA has rank r, capturing the most important directional update with far fewer parameters than a full d×k matrix.

Question 3

For a weight matrix of dim d=4096 with LoRA rank r=8, how many trainable parameters does LoRA add?

Accepted Answer

65,536 parameters total — LoRA adds two matrices: B (d×r) and A (r×k). With d=k=4096 and r=8, that is 2 × 8 × 4096 = 65,536 — roughly 256× fewer than the full d² = 16.8M parameters.

Question 4

QLoRA's key contribution over standard LoRA is…

Accepted Answer

Quantizing the frozen base model to 4-bit — QLoRA quantizes the frozen pretrained weights to 4-bit (NormalFloat4), drastically cutting memory. The small LoRA adapters stay in BF16/FP16 for numerical stability. This enables 65B fine-tuning on a single 48GB GPU.

Question 5

LoRA is typically applied to which transformer weight matrices?

Accepted Answer

The four attention matrices: query, key, value and output projection — LoRA is most commonly applied to all four attention matrices — W_q, W_k, W_v and W_o — and uniformly to every matrix of the same type in the model. Under a fixed 18M-parameter budget on GPT-3 175B, Hu et al. found all four at rank 2 beat one matrix at rank 8 or two at rank 4 on WikiSQL and MultiNLI; query and value are the best pair only if you can afford just two. Applying LoRA to the feedforward layers as well often helps further.

Question 6

What is the LoRA scaling hyperparameter alpha (α) used for?

Accepted Answer

Scaling the adapter output by α/r — The adapter output is scaled by α/r before adding to W. Higher α relative to r amplifies the adapter's contribution. In practice α ≈ r or 2r is common — this lets you change r without re-tuning the effective step size.

Question 7

PEFT stands for…

Accepted Answer

Parameter-Efficient Fine-Tuning — PEFT (Parameter-Efficient Fine-Tuning) is the umbrella term for methods that update only a small subset of parameters — including LoRA, adapters, prefix tuning, and prompt tuning.

Question 8

After training, how can LoRA be deployed with zero inference latency overhead?

Accepted Answer

Merging the update into frozen weights — Since ΔW = BA has the same shape as W, you can precompute and absorb it: W' = W + BA. The resulting W' is loaded like a normal weight — no extra ops, no latency hit.

Question 9

Instruction tuning (a form of SFT) is designed to…

Accepted Answer

Fine-tune on labeled (instruction, response) pairs — Instruction tuning (SFT stage) fine-tunes the pretrained model on a diverse set of (instruction, desired response) examples, improving its ability to generalize to new instructions at inference time.

Question 10

When does fine-tuning clearly beat few-shot prompting?

Accepted Answer

When you have many labels and a stable task — Fine-tuning amortizes the cost of demonstrating a task over inference calls and removes the need for long few-shot prompts. But it requires significant labeled data and a stable task definition.

Key insight

The LoRA decomposition

Test your knowledge

LoRA stands for…

The short version

In more depth

Fine-Tuning & LoRA

LoRA: Low-Rank Adaptation

QLoRA

When to fine-tune?

Interview Questions

References & Further Reading

Related topics

Love this visualization?