The fine-tuning ladder, with a worked LoRA example

What it is

"Fine-tuning" is used to mean five different interventions with wildly different costs, and the first job in any conversation about it is establishing which rung of the ladder someone means:

Rung	What changes	Cost	Fixes
1. Prompting	Nothing	Minutes	Unclear instructions
2. Few-shot / prompt optimisation	Nothing	Hours	Format, style, edge cases
3. RAG	Retrieved context	Days	Missing knowledge
4. PEFT / LoRA	~0.1-2% of weights	Days to weeks	Behaviour, format, tone, task
5. Full fine-tune / continued pretraining	All weights	Weeks, GPUs	Domain shift, new language

The ladder is ordered by cost and you climb it only when the rung below has failed for a diagnosed reason. The single most common expensive mistake in applied LLM work is jumping to rung 4 or 5 for a problem that lives on rung 3.

The rule that resolves most cases: RAG for knowledge, fine-tuning for behaviour. If the model does not know something, that is retrieval. If the model knows it and will not do what you want with it (wrong format, wrong tone, wrong reasoning pattern, ignores a constraint), that is fine-tuning. Fine-tuning is a poor way to install facts: the facts become stale, you cannot cite sources, and updating one fact means another training run.

LoRA (Low-Rank Adaptation) is the technique that made rung 4 practical. Instead of updating a weight matrix W, freeze it and learn a low-rank update:

$$W' = W + \Delta W = W + BA \quad\text{where } B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d,k)$$

What this is confused with: LoRA is not a smaller model and not distillation. The base model is unchanged and full-size; LoRA adds a small trainable delta. At inference you can either keep them separate (allowing many adapters on one base) or merge them (W ← W + BA) for zero added latency.

The problem it solves

Full fine-tuning an 8B model requires roughly 8x the model size in GPU memory:

Weights (bf16):                   16 GB
Gradients (bf16):                 16 GB
Adam optimiser state (fp32 m, v): 64 GB
Master weights (fp32):            32 GB
                                 ──────
                                 128 GB   before activations

That needs multiple 80 GB GPUs for a model that serves happily on one. And you get one model per task: three fine-tuned variants means three 16 GB checkpoints and three deployments.

LoRA changes both:

Base weights (frozen, bf16):      16 GB
LoRA params (r=16, ~0.5% of W):   ~40 MB
Gradients (LoRA only):            ~40 MB
Optimiser state (LoRA only):     ~160 MB
                                 ──────
                                 ~16.3 GB   -> fits on one GPU

About 8x less memory, and the adapter is 40 MB instead of 16 GB. You can store hundreds of task adapters and swap them per request against one loaded base model.

Mechanics

Why low rank works

The hypothesis, from the LoRA paper: the weight update needed to adapt a pretrained model to a downstream task has low intrinsic rank. The model already contains the capability; adaptation is a small directional nudge, not a re-derivation.

W is [4096, 4096]                        = 16.8 M parameters
LoRA with r=16:
    A is [16, 4096]  = 65,536
    B is [4096, 16]  = 65,536
                       ───────
                       131,072 params      = 0.78% of W

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.05):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False                    # frozen

        self.A = nn.Parameter(torch.zeros(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        # B starts at ZERO, so BA = 0 and the model is unchanged at step 0.
        # Training therefore begins from exactly the base model's behaviour.

        self.scaling = alpha / r
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.dropout(x) @ self.A.T @ self.B.T * self.scaling

B initialised to zero is not incidental. It means ΔW = 0 at initialisation, so training starts from exactly the pretrained model rather than from a perturbed version. Without it, the first steps would fight a random perturbation.

alpha/r scaling decouples the learning rate from the rank. With alpha fixed, doubling r halves the per-component contribution, so you can change rank without retuning the learning rate. The convention alpha = 2r is common and works.

Which modules to adapt

The original paper applied LoRA only to W_q and W_v. Subsequent practice found adapting all linear layers works better, and it is now the default:

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",     # attention
                  "gate_proj", "up_proj", "down_proj"]        # FFN

The FFN modules matter because, as the transformer page shows, the feed-forward network is about 72 percent of the parameters. Adapting only attention leaves most of the model untouched.

Choosing rank

r = 4-8      style, tone, output format. Small behavioural changes.
r = 16-32    task adaptation: a new task the base model can nearly do.
r = 64-128   substantial behavioural change, larger datasets (>50k examples).
r > 128      diminishing returns; consider full fine-tuning instead.

Higher rank is not reliably better. With a small dataset, high rank overfits: more capacity means more room to memorise. The honest procedure is to try r=16, measure, and increase only if it is underfitting (training loss still falling at the end and eval tracking it).

QLoRA: the memory that made this accessible

QLoRA quantises the frozen base to 4-bit while training LoRA in bf16:

Base weights (NF4 4-bit):    4.5 GB
LoRA params + grads + opt:   0.25 GB
Activations:                 ~4 GB
                            ────────
                            ~9 GB    -> a 24 GB consumer GPU fine-tunes a 7B model

Three components: NF4 (a 4-bit datatype information-theoretically optimal for normally-distributed weights), double quantisation (quantising the quantisation constants, saving about 0.4 bits per parameter), and paged optimisers (using unified memory to survive gradient-checkpointing spikes).

The reported result is that QLoRA matches 16-bit LoRA quality, which is the claim that made it standard. The cost is speed: dequantising on every forward pass is roughly 30 to 50 percent slower than bf16 LoRA. Use QLoRA when memory-constrained, plain LoRA when not.

Serving many adapters

# One base model in memory; adapters swapped per request.
# vLLM, LoRAX, and TGI all support this natively.
base = load_model("llama-3-8b")           # 16 GB, loaded ONCE
adapters = {
    "support-tone":   load_lora("s3://.../support/"),      # 40 MB
    "code-review":    load_lora("s3://.../review/"),       # 40 MB
    "sql-generation": load_lora("s3://.../sql/"),          # 40 MB
}

One GPU serves dozens of task-specific models. With full fine-tuning, three tasks means three 16 GB models and three deployments. This is the operational argument for LoRA and it is frequently more decisive than the training-cost argument.

Alternatively, merge for zero latency overhead:

merged = base_weight + (B @ A) * (alpha / r)      # one model, no runtime cost

Merged means no adapter swapping and no per-request overhead; unmerged means one base serving many tasks. Choose by whether you need one specialised model or many.

A worked example: a support classifier, all five rungs

A SaaS company routing support tickets into 14 categories and drafting a first reply. Baseline was a prompted GPT-4-class model.

Rung 1: prompting.

routing accuracy:        71.2%
draft acceptance:        34%       (agent sends without editing)
cost per ticket:         $0.031
p50 latency:             2,400ms

Errors were concentrated: three category pairs accounted for 61 percent of routing mistakes, and they were genuinely ambiguous distinctions specific to this product ("billing dispute" versus "subscription change").

Rung 2: few-shot with 8 curated examples.

routing accuracy:        71.2% -> 78.9%
draft acceptance:        34% -> 41%
cost per ticket:         $0.031 -> $0.048     (+55%, the examples are input tokens)
p50 latency:             2,400 -> 3,100ms

A real gain, and it cost 55 percent more per ticket forever, because the examples are re-sent with every request. Few-shot has an ongoing cost that fine-tuning does not, which is the crossover argument at volume.

Rung 3: RAG over the knowledge base.

routing accuracy:        78.9% -> 79.4%     (+0.5, essentially noise)
draft acceptance:        41% -> 68%         (+27 points)

This is the ladder's diagnostic working. RAG barely moved routing, because routing is a behaviour problem: the model must learn this company's category boundaries, and no retrieved document teaches that. It moved draft quality enormously, because drafting needs knowledge: the refund policy, the current pricing, the escalation path.

Two problems, two rungs, and running both experiments was what revealed it.

Rung 4: LoRA for the routing behaviour.

Data: 42,000 historical tickets with human-verified categories.
Split: 38k train, 2k validation, 2k test (stratified, time-ordered split
       so test is the most recent, avoiding leakage from label drift).

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=32, lora_alpha=64, lora_dropout=0.05,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
    task_type="CAUSAL_LM")

model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable: 83,886,080 || all: 8,113,197,056 || trainable%: 1.034

Training:
  base model:        Llama-3-8B-Instruct
  hardware:          1x A100 80GB
  epochs:            3
  effective batch:   32 (batch 8, grad accumulation 4)
  learning rate:     2e-4, cosine schedule, 100 warmup steps
  wall clock:        4h 20m
  cost:              ~$14

routing accuracy:        79.4% -> 94.1%
  on the 3 ambiguous pairs: 41% -> 91%
cost per ticket:         $0.048 -> $0.004      (8B self-hosted vs API)
p50 latency:             3,100 -> 340ms

The ambiguous-pair improvement from 41 to 91 percent is the whole story. Those distinctions are not in any document and cannot be prompted concisely, because they are implicit in how this company's agents have historically labelled tickets. 38,000 examples teach them; a paragraph of instructions cannot.

What failed, and it is instructive. Their first attempt used r=8 and one epoch:

r=8, 1 epoch:    routing 84.2%    <- underfitting; train loss still falling
r=32, 3 epochs:  routing 94.1%
r=64, 3 epochs:  routing 93.8%    <- no better; slight overfit on validation
r=32, 6 epochs:  routing 92.1%    <- overfitting: train 99.2%, test 92.1%

Rank and epochs both have an interior optimum, and the only way to find it is a small sweep. Their instinct that "more capacity is better" was wrong in both directions.

Rung 5: they considered full fine-tuning and did not do it.

estimate: 4x A100 for ~18 hours = ~$430 per run
expected gain over LoRA: ~1-2 points, from published comparisons
adapters possible: one model per task, 16 GB each

The deciding factor was operational, not quality. They had three tasks (routing, drafting, summarisation) and LoRA meant one base model with three 40 MB adapters served from one GPU. Full fine-tuning meant three 16 GB models and three deployments, for one to two points.

Final architecture, and it uses two rungs at once:

routing:   LoRA adapter (behaviour)        94.1% accuracy
drafting:  RAG + a separate LoRA adapter   81% draft acceptance
           (RAG for knowledge, LoRA for tone and format)

                     baseline    final
routing accuracy      71.2%      94.1%
draft acceptance      34%        81%
cost per ticket       $0.031     $0.006
p50 latency           2,400ms    380ms
GPUs                  0 (API)    2 (1 + 1 standby)

The lesson is the diagnosis rather than the technique. Routing was a behaviour problem and drafting was a knowledge problem, they looked identical from the outside ("the model gets tickets wrong"), and they needed different rungs. A team that had jumped straight to fine-tuning would have fixed routing and left drafting at 41 percent; a team committed to RAG would have fixed drafting and left routing at 79.

Production evidence

LoRA (Hu et al., Microsoft, 2021) reported matching full fine-tuning on GLUE with 0.01 percent of the trainable parameters on GPT-3 175B. The low-intrinsic-rank hypothesis was the paper's contribution and it has held up across model families.

QLoRA (Dettmers et al., 2023) fine-tuned a 65B model on a single 48 GB GPU and reported matching 16-bit performance, which is what moved fine-tuning from a well-resourced-lab activity to something a small team can do. NF4, double quantisation and paged optimisers are all from that paper.

Hugging Face PEFT is the standard implementation and supports LoRA, QLoRA, prefix tuning, prompt tuning, IA³ and DoRA. Its adoption is what made the technique routine.

vLLM, LoRAX (Predibase) and TGI all support multi-adapter serving, dynamically loading adapters against a shared base. That production support is the strongest evidence for the operational argument: serving many adapters from one base is a supported deployment pattern, not a hack.

S-LoRA (Sheng et al., 2023) demonstrated serving thousands of concurrent adapters on one GPU with unified paging for adapter weights, which is the extreme version of the same idea.

DoRA (Liu et al., 2024) decomposes the weight update into magnitude and direction, applying LoRA only to the direction, and reports consistently better results than LoRA at equal parameter count. It is the most credible recent refinement and is supported in PEFT.

The debate

RAG or fine-tuning? The rule is knowledge versus behaviour and it holds up under pressure. Facts belong in retrieval because they change, they need citations, and updating one fact should not require a training run. Behaviour belongs in fine-tuning because format, tone, category boundaries and reasoning patterns cannot be conveyed concisely in a prompt and are stable over time. Most real systems need both, as the worked example shows, and framing it as a choice is the error.

Is fine-tuning ever right for knowledge? Narrowly: when the "knowledge" is really a large stable vocabulary or notation the base model handles badly (a proprietary schema, a domain notation, an internal DSL). That is closer to continued pretraining than to instruction tuning, and the tell is that retrieval does not help because the model cannot use the retrieved text properly.

LoRA or full fine-tuning? LoRA, in nearly every case, and the deciding argument is usually operational rather than quality. Published comparisons put LoRA within 1 to 3 points of full fine-tuning on task adaptation, and LoRA gives you many-adapters-one-base serving, 40 MB artifacts, one-GPU training, and trivial rollback (delete the adapter). Full fine-tuning earns its place for genuine domain shift: a new language, a new modality, a domain whose token distribution differs sharply from pretraining. Those are continued-pretraining problems and LoRA's low-rank constraint genuinely limits them.

How much data do you need? Fewer examples than people expect for behaviour, more than they expect for quality. Roughly: 500 to 1,000 high-quality examples for style and format, 5,000 to 50,000 for task adaptation, and beyond that returns flatten quickly. Data quality dominates quantity past a few thousand: 2,000 carefully verified examples routinely beat 20,000 noisy ones, because the model learns the noise faithfully. The LIMA result (1,000 curated examples producing strong instruction-following) is the canonical demonstration.

What about catastrophic forgetting? Full fine-tuning on a narrow task degrades general capability measurably. LoRA is much more resistant, because the base weights are frozen and the update is rank-constrained, but it is not immune: aggressive training with high rank on a narrow distribution still shifts behaviour. Always evaluate on general benchmarks as well as the target task, and the failure mode to watch for is a model that is excellent at the fine-tuned task and worse at everything adjacent.

Follow-up Q&A

"RAG or fine-tuning, and how do you decide?"

Knowledge versus behaviour. If the model does not know a fact, that is retrieval, because facts change, need citations, and should not require a training run to update. If the model knows the material and will not do what you want with it (wrong format, wrong tone, does not respect your category boundaries), that is fine-tuning. The diagnostic in practice is to run both cheaply: in one case RAG moved routing accuracy 0.5 points and draft acceptance 27 points, which said immediately that routing was behaviour and drafting was knowledge, and those looked identical from the outside.

"How does LoRA work and why is low rank sufficient?"

Freeze W and learn ΔW = BA with B [d, r] and A [r, k] for small r. The hypothesis is that adapting a pretrained model has low intrinsic rank: the capability is already present and adaptation is a directional nudge rather than a re-derivation. B is initialised to zero so ΔW = 0 at step 0 and training starts from exactly the base model. At r=16 on a 4096x4096 matrix that is 0.78 percent of the parameters, and memory drops roughly 8x because gradients and optimiser state only cover the adapter.

"What rank would you use?"

Start at 16 for task adaptation and measure. 4 to 8 for style and format, 16 to 32 for a task the base model can nearly do, 64 to 128 only with a large dataset. Higher rank is not reliably better: with limited data it overfits, and in one measured sweep r=64 was slightly worse than r=32 on the same data. Both rank and epoch count have an interior optimum, so a small sweep is not optional.

"How much data, and what quality?"

500 to 1,000 for style, 5,000 to 50,000 for task adaptation, and diminishing returns beyond. Quality dominates past a few thousand examples: the model learns your noise faithfully, so 2,000 verified examples typically beat 20,000 noisy ones. Split time-ordered rather than randomly if the labels can drift, so the test set is the most recent data and you are not leaking future labelling conventions into training.

"When is full fine-tuning worth it over LoRA?"

Genuine domain shift rather than task adaptation: a new language, a domain whose token distribution differs sharply from pretraining, or continued pretraining on a large corpus. For ordinary task adaptation LoRA is within a couple of points and wins operationally: 40 MB artifacts, one-GPU training, many adapters on one base, and rollback by deleting a file. In one case the deciding factor was three tasks meaning three 16 GB models and three deployments versus one base and three 40 MB adapters.

"What is QLoRA and what does it cost you?"

The frozen base is quantised to 4-bit NF4 while LoRA trains in bf16, with double quantisation and paged optimisers. It takes a 7B fine-tune to about 9 GB, so a consumer GPU works, and it reportedly matches 16-bit LoRA quality. The cost is speed: dequantising on every forward pass is 30 to 50 percent slower. Use it when memory-bound, plain LoRA otherwise.

Common misconceptions

"Fine-tuning teaches the model facts." It teaches behaviour reliably and facts poorly. Facts learned this way go stale, cannot be cited, and updating one requires another training run. That is what retrieval is for.

"LoRA is a compressed model." The base model is unchanged and full-size. LoRA is a small additive delta, and at inference you either keep it separate (many adapters, one base) or merge it (W ← W + BA) for zero overhead.

"Higher rank is better." It is more capacity and more room to overfit. Rank has an interior optimum that depends on dataset size, and going from 32 to 64 made things slightly worse in one measured sweep.

"More training data always helps." Past a few thousand examples, quality dominates quantity, and noisy labels are learned faithfully. Curating 2,000 examples usually beats collecting 20,000.

"Fine-tuning is expensive." A LoRA run on an 8B model was $14 and four hours on one A100 in the worked example. What is expensive is the data curation and the evaluation, which is where the real project cost sits.

Interview delivery note

Say this verbatim: "RAG for knowledge, fine-tuning for behaviour. If the model does not know a fact, retrieve it, because facts change and should not need a training run. If the model knows the material and will not do what you want with it, that is fine-tuning, and I would start with LoRA at rank 16 and measure." The rule plus a concrete starting point, which is what separates a position from a description.

The senior-versus-staff separator is running both experiments to diagnose which problem you have. A senior engineer knows the knowledge-versus-behaviour rule. A staff engineer notices that RAG moved routing 0.5 points and drafting 27 points, concludes those are two different problems that looked identical from the outside, and ships both rungs. The diagnosis is the work; the technique is the easy part.

The second signal is the operational argument for LoRA over full fine-tuning: three tasks means three 16 GB models and three deployments, or one base with three 40 MB adapters served from one GPU. Quality is within a couple of points either way, and the deployment difference is what actually decides it.

The Staff & Team Lead Interview Reference