\n d = {d}, r = {clampedR} → full = {fmt(full)}, LoRA = {fmt(lora)}.{' '}\n {saves ? (\n <>\n LoRA is {ratio.toFixed(1)}× smaller.\n \n ) : (\n <>\n No saving — at r ≥ d/2 the full update is as cheap or cheaper.\n \n )}\n
\n
\n );\n}\n",
+ "frameTitle": "The LoRA budget",
+ "frameCaption": "Break-even is r = d/2. Try toy · 4 at r=2 (break-even — the savings vanish), then Llama attn · 4096 at r=16 (~128× smaller). Real matrices are thousands wide, so a small r always wins big.",
+ "frameSize": "normal",
+ "frameExpand": true
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-42",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Set it to the toy $4$ at $r=2$ and the bars are equal — no win. Now jump to a real attention matrix ($4096$) at $r=16$: the full update is 16.7 million numbers, LoRA’s is 131 thousand — ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "128× smaller",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ", for the ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "same-shaped",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " update. Because real weight matrices are thousands wide and $r$ stays tiny, LoRA sits far below the $d/2$ break-even and the savings are enormous. That 0.2% from the opening is this ratio, summed over every matrix in the model.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-43",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "This also answers “why not just make $A$ and $B$ full-size?” Full-size factors would be ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "more",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " numbers than $\\Delta W$ itself — and no longer low-rank. The whole point is the skinny shared bottleneck.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "heading-44",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Why untrained matrices don’t break the model",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-45",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Here’s the objection that trips everyone up: $A$ and $B$ are new matrices. They know ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "nothing",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — not language, not your task. If we bolt them onto a carefully pretrained model, shouldn’t they inject garbage and wreck it?",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-46",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The fix is a beautiful one-liner. In our letter convention, $A$ starts random and ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "$B$ starts as all zeros",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " (some libraries swap the letters). Because one factor is zero, the product is exactly zero:",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "math-47",
+ "type": "math",
+ "props": {
+ "latex": "\\Delta W = A\\,B = A \\cdot 0 = 0\n\\quad\\Longrightarrow\\quad\nh = Wx + 0 = Wx"
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-48",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "At step zero, the adapter contributes ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "nothing",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". The model behaves ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "exactly",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " like the untouched base. Training then moves the zero matrix off zero and reshapes the random matrix with it, so the adapter grows from no-op to useful correction.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "table-49",
+ "type": "table",
+ "props": {
+ "textColor": "default"
+ },
+ "content": {
+ "type": "tableContent",
+ "columnWidths": [
+ null,
+ null,
+ null
+ ],
+ "rows": [
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "Moment",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "Adapter math",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "What the model receives",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ },
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "Before training",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "$B = 0$, so $ABx = 0$",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "$h = Wx$ — exactly the frozen base",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ },
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "During training",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "$B$ becomes nonzero, $A$ also adjusts",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "$h = Wx + ABx$ — base output plus a learned nudge",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ },
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "After training",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "$AB$ is the learned low-rank update",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "Same base model, different activations flowing forward",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ }
+ ]
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-50",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "So the untrained adapter never starts as “random garbage that corrupts the model.” It starts as a ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "zero delta",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". The only thing training is allowed to learn is the extra nudge the frozen model needs: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "“given the features the frozen model already computed here, push the activation this way.”",
+ "styles": {
+ "italic": true
+ }
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "heading-51",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "How it’s wired: a side path, not a new layer",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-52",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "So where does the adapter physically sit? Not as a new layer stacked on top, and not as a separate model running in parallel. It sits ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "beside each weight matrix it adapts",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". The same input $x$ flows into both the frozen $W$ and the adapter, and their outputs are ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "summed",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ":",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "math-53",
+ "type": "math",
+ "props": {
+ "latex": "h = W x + A B x = (W + \\Delta W)\\,x"
+ },
+ "children": []
+ },
+ {
+ "id": "svg-54",
+ "type": "svg",
+ "props": {
+ "code": "",
+ "caption": "LoRA is a side path. The input feeds both the frozen W and the low-rank adapter (down to r, back up); the two outputs are added. The base weights never change — but the activation h passed to the next layer does."
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-55",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The static diagram shows the wiring. The animation below shows what happens ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "inside",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " the lower path: $B$ compresses many input features into $r$ bottleneck slots, $A$ expands those few slots into a full-size correction, and the correction gets added to the frozen output. Toggle the adapter off to see the step-0 case, where $B = 0$ makes every bottleneck slot zero.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "customComponent-56",
+ "type": "customComponent",
+ "props": {
+ "componentName": "SidePathFlow",
+ "source": "import { useEffect, useMemo, useRef, useState, type CSSProperties } from 'react';\n\nconst mono: CSSProperties = { fontFamily: 'var(--font-mono)', fontSize: 12, color: 'var(--ink-2)' };\n\nconst X = [0.82, 0.34, 0.68, 0.22, 0.74, 0.46, 0.58, 0.28];\nconst WX = [0.52, 0.38, 0.64, 0.44, 0.57, 0.35];\nconst B_MIX = [\n [0.9, 0.16, 0.7, 0.12, 0.62, 0.2, 0.74, 0.3],\n [0.12, 0.82, 0.28, 0.78, 0.22, 0.72, 0.36, 0.68],\n [0.48, 0.22, 0.16, 0.66, 0.86, 0.42, 0.24, 0.58],\n];\nconst A_MIX = [\n [0.76, 0.28, 0.52],\n [0.18, 0.78, 0.34],\n [0.58, 0.2, 0.76],\n [0.32, 0.56, 0.82],\n [0.72, 0.34, 0.22],\n [0.28, 0.74, 0.5],\n];\n// W is d_out × d_in (6 × 8) — the full, dense frozen matrix.\nconst W_MIX = Array.from({ length: 6 }, (_, r) =>\n Array.from({ length: 8 }, (_, c) => 0.55 + 0.35 * Math.abs(Math.sin((r + 1.3) * (c + 1.7)))),\n);\n\nfunction wavg(values: number[], weights: number[]): number {\n const total = weights.reduce((a, b) => a + b, 0);\n return values.reduce((a, v, i) => a + v * weights[i], 0) / total;\n}\nfunction nodeY(i: number, n: number, cy: number, gap: number): number {\n return cy + (i - (n - 1) / 2) * gap;\n}\n// smoothstep reveal: 0 before a, 1 after b\nfunction ss(a: number, b: number, t: number): number {\n if (t <= a) return 0;\n if (t >= b) return 1;\n const u = (t - a) / (b - a);\n return u * u * (3 - 2 * u);\n}\n\ntype Pt = [number, number];\n// Position along a polyline at fraction f (0..1), by arc length.\nfunction pointAt(pts: Pt[], f: number): Pt {\n const seg: number[] = [];\n let total = 0;\n for (let i = 1; i < pts.length; i++) {\n const d = Math.hypot(pts[i][0] - pts[i - 1][0], pts[i][1] - pts[i - 1][1]);\n seg.push(d);\n total += d;\n }\n let dist = f * total;\n for (let i = 0; i < seg.length; i++) {\n if (dist <= seg[i]) {\n const k = seg[i] ? dist / seg[i] : 0;\n return [pts[i][0] + (pts[i + 1][0] - pts[i][0]) * k, pts[i][1] + (pts[i + 1][1] - pts[i][1]) * k];\n }\n dist -= seg[i];\n }\n return pts[pts.length - 1];\n}\n\n// deliberate grid so every label sits on its object\nconst X_CX = 52;\nconst X_CY = 150;\nconst X_GAP = 15;\nconst WX_CX = 258;\nconst WX_CY = 64;\nconst WX_GAP = 11;\nconst Z_CX = 298;\nconst Z_CY = 194;\nconst Z_GAP = 22;\nconst D_CX = 452;\nconst D_CY = 194;\nconst D_GAP = 12;\nconst PLUS_X = 556;\nconst PLUS_Y = 128;\nconst H_CX = 628;\nconst H_CY = 128;\nconst H_GAP = 12;\nconst B_X = 150;\nconst B_Y = 178;\nconst A_X = 356;\nconst A_Y = 160;\nconst CELL = 9;\nconst CGAP = 3;\nconst W_X = 148;\nconst W_Y = 44;\nconst W_CELL = 5.5;\nconst W_GAP = 1.5;\n\n// dot routes, sampled to hug the actual (curved) arrows\nconst TOP_ROUTE: Pt[] = [\n [60, 122],\n [106, 82],\n [140, 64],\n [212, 64],\n [258, 64],\n [350, 74],\n [429, 88],\n [495, 104],\n [541, 120],\n [556, 128],\n [628, 128],\n];\nconst BOT_ROUTE: Pt[] = [\n [60, 176],\n [108, 189],\n [148, 194],\n [243, 194],\n [298, 194],\n [356, 194],\n [389, 194],\n [452, 194],\n [487, 179],\n [511, 164],\n [529, 150],\n [543, 138],\n [556, 128],\n [628, 128],\n];\n\nfunction Vector({\n values,\n cx,\n cy,\n gap,\n color,\n opacity = 1,\n}: {\n values: number[];\n cx: number;\n cy: number;\n gap: number;\n color: string;\n opacity?: number;\n}) {\n return (\n \n {values.map((value, i) => (\n \n ))}\n \n );\n}\n\nfunction Grid({\n x,\n y,\n mix,\n active,\n color = 'var(--accent)',\n cell = CELL,\n gap = CGAP,\n base = 0.1,\n scale = 0.34,\n}: {\n x: number;\n y: number;\n mix: number[][];\n active: number;\n color?: string;\n cell?: number;\n gap?: number;\n base?: number;\n scale?: number;\n}) {\n return (\n \n {mix.map((row, r) =>\n row.map((value, c) => (\n \n )),\n )}\n \n );\n}\n\nexport default function SidePathFlow() {\n const [enabled, setEnabled] = useState(true);\n const [t, setT] = useState(0);\n const raf = useRef(0);\n\n useEffect(() => {\n let start: number | null = null;\n const loop = (now: number) => {\n if (start === null) start = now;\n setT(((now - start) / 3400) % 1);\n raf.current = requestAnimationFrame(loop);\n };\n raf.current = requestAnimationFrame(loop);\n return () => cancelAnimationFrame(raf.current);\n }, []);\n\n const on = enabled;\n const z = useMemo(() => (on ? B_MIX.map((w) => wavg(X, w)) : [0, 0, 0]), [on]);\n const delta = useMemo(() => A_MIX.map((w) => wavg(z, w) * 0.9), [z]);\n\n // reveal factors, driven by where the single dot is along its route\n const wxOn = ss(0.28, 0.42, t);\n const bOn = on ? ss(0.12, 0.3, t) : 0;\n const zOn = on ? ss(0.34, 0.46, t) : 0;\n const aOn = on ? ss(0.5, 0.68, t) : 0;\n const abOn = on ? ss(0.66, 0.76, t) : 0;\n const hOn = ss(0.88, 1, t); // both dots deliver to h at the end\n\n const topDot = pointAt(TOP_ROUTE, t);\n const botDot = pointAt(BOT_ROUTE, t);\n\n return (\n
\n \n \n \n \n \n \n\n {/* input x */}\n \n x\n d_in features\n\n {/* TOP: frozen path x -> W -> Wx */}\n \n \n \n frozen W · not trained\n d_out × d_in · dense\n \n \n Wx\n\n {/* BOTTOM: adapter path x -> B -> z -> A -> ABx */}\n \n \n B\n r × d_in · wide\n\n \n \n z = Bx\n\n \n \n A\n d_out × r · tall\n\n \n \n ABx\n\n {/* merge into h */}\n \n \n \n +\n \n\n {/* h = grey Wx base + accent ABx correction, both revealed as the dots arrive */}\n \n {WX.map((base, i) => {\n const cy = nodeY(i, WX.length, H_CY, H_GAP);\n const corr = on ? Math.min(1, delta[i]) : 0;\n return (\n \n \n {corr > 0.01 && (\n \n )}\n \n );\n })}\n \n h\n {on ? 'h = Wx + ABx' : 'h = Wx'}\n\n {/* one dot per route, riding exactly on its arrow */}\n \n {on && }\n \n\n
\n );\n}\n",
+ "frameTitle": "One base, many adapters",
+ "frameCaption": "The 1.2 GB base loads once and never changes. Each skill is a ~4 MB adapter snapped on top and swapped per request — so a serving stack can offer hundreds of fine-tunes without hundreds of full models.",
+ "frameSize": "normal",
+ "frameExpand": true
+ },
+ "children": []
+ },
+ {
+ "id": "heading-88",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The real numbers: a 0.6B model on a laptop",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-89",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Back to the opening. The model was ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "Qwen3-0.6B",
+ "styles": {
+ "code": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — 596 million parameters. With LoRA at $r=16$ on its linear layers, the trainable adapter was ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "1.4 million parameters — 0.24% of the model",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". Training touched only those; the 596M base stayed frozen, so the memory that would have gone to gradients and optimizer state for 596M parameters simply never got allocated. Peak memory: ~3 GB. Wall-clock: ~40 seconds for a few hundred steps.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-90",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Push it further and you get ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "QLoRA",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ": quantize the frozen base to 4-bit (it’s read-only, so precision matters less), and even the ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "frozen",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " copy shrinks ~4×. That’s how a 7B model fine-tunes in ~8 GB — a single consumer GPU — instead of ~112 GB.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "heading-91",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "When not to reach for LoRA",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-92",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "LoRA is a low-rank ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "adjustment",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " to existing knowledge. It shines at installing behaviors and skills the base can already almost do. It’s the wrong tool when:",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-93",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "You need to inject ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "large-scale new knowledge",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " (a new language, a whole domain the base never saw). That’s a high-rank change; you likely need continued pretraining or full fine-tuning.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-94",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Your rank is ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "too low for a genuinely complex task",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — the adapter underfits. Raise $r$, or reconsider.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-95",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The honest framing: LoRA bets your change is simple. When that bet holds — which is most day-to-day fine-tuning — you get 99% of full fine-tuning’s benefit for ~0.2% of its cost. When it doesn’t, no bottleneck will save you.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "separator-96",
+ "type": "separator",
+ "props": {},
+ "children": []
+ },
+ {
+ "id": "heading-97",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Recap — the whole idea in one breath",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-98",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Fine-tuning adds an update $\\Delta W$ to each weight. Full fine-tuning learns that update as a matrix ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "as big as the model",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — expensive. LoRA bets the update is ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "low-rank",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " (it lives in a few directions), so it learns $\\Delta W = A B$ — two skinny matrices sharing a small rank $r$ — instead. One is zero-initialized so the adapter starts as a no-op and grows a delta; it rides on top of the frozen model, changing only the activation passed forward, never the weights themselves. On real (large) matrices this is dozens to hundreds of times cheaper, which is how you fine-tune a billion-parameter model, on a laptop, by changing 0.2% of it.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-99",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "That’s not a magic trick. It’s a bet about the shape of change — and it’s usually right.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ }
+ ],
+ "tableVariants": {
+ "table-14": {
+ "border": "lined",
+ "zebra": true
+ },
+ "table-49": {
+ "border": "lined",
+ "zebra": true
+ },
+ "table-80": {
+ "border": "lined",
+ "zebra": true
+ }
+ }
+}
diff --git a/src/content/posts/lora-from-zero/LoRABudget.tsx b/src/content/posts/lora-fine-tuning-guide/LoRABudget.tsx
similarity index 100%
rename from src/content/posts/lora-from-zero/LoRABudget.tsx
rename to src/content/posts/lora-fine-tuning-guide/LoRABudget.tsx
diff --git a/src/content/posts/lora-from-zero/MultiLoRASwap.tsx b/src/content/posts/lora-fine-tuning-guide/MultiLoRASwap.tsx
similarity index 100%
rename from src/content/posts/lora-from-zero/MultiLoRASwap.tsx
rename to src/content/posts/lora-fine-tuning-guide/MultiLoRASwap.tsx
diff --git a/src/content/posts/lora-from-zero/RankReconstruction.tsx b/src/content/posts/lora-fine-tuning-guide/RankReconstruction.tsx
similarity index 100%
rename from src/content/posts/lora-from-zero/RankReconstruction.tsx
rename to src/content/posts/lora-fine-tuning-guide/RankReconstruction.tsx
diff --git a/src/content/posts/lora-from-zero/SidePathFlow.tsx b/src/content/posts/lora-fine-tuning-guide/SidePathFlow.tsx
similarity index 100%
rename from src/content/posts/lora-from-zero/SidePathFlow.tsx
rename to src/content/posts/lora-fine-tuning-guide/SidePathFlow.tsx
diff --git a/src/content/posts/lora-from-zero/index.mdx b/src/content/posts/lora-fine-tuning-guide/index.mdx
similarity index 61%
rename from src/content/posts/lora-from-zero/index.mdx
rename to src/content/posts/lora-fine-tuning-guide/index.mdx
index 944df31..7bfe408 100644
--- a/src/content/posts/lora-from-zero/index.mdx
+++ b/src/content/posts/lora-fine-tuning-guide/index.mdx
@@ -1,39 +1,40 @@
---
-title: 'LoRA From Zero: Fine-Tuning a Model by Changing 0.2% of It'
-summary: 'Full fine-tuning learns a weight update as big as the model itself. LoRA learns that same update as two tiny matrices instead — tuning 0.2% of the parameters, on a laptop. Here is exactly why that works.'
+title: 'Intuitive Guide to LoRA: Fine-Tuning a Model by 0.2% of weights'
+summary: 'You don''t need a massive tech budget or a cluster of high-end GPUs to train your own AI. LoRA allows developers to fine-tune giant models right on a standard laptop. Here is the zero-jargon, first-principles explanation of the clever shortcut that leveled the playing field.'
authors: ['dinesh']
date: '2026-07-19'
-readMin: 17
+updated: '2026-07-20'
+readMin: 12
topic: 'Training'
topicId: 'training'
tags: ['lora', 'fine-tuning', 'peft', 'low-rank', 'adapters', 'qlora']
---
+import RankReconstruction from './RankReconstruction';
import Interactive from '../../../components/Interactive';
import LoRABudget from './LoRABudget';
-import RankReconstruction from './RankReconstruction';
import SidePathFlow from './SidePathFlow';
import MultiLoRASwap from './MultiLoRASwap';
-Last week I fine-tuned a 0.6-billion-parameter language model on my laptop. It went from failing a task 87% of the time to nailing it 100% of the time — in about **40 seconds**, using **3 GB** of memory, by training just **0.2%** of the model's parameters.
+Last week I fine-tuned a 0.6-billion-parameter language model on my laptop. It went from failing a task 87% of the time to nailing it 100% of the time — in about **40 seconds**, using **3 GB** of memory, by training just **0\.2%** of the model’s parameters.
-Every number in that sentence should feel slightly impossible. Fine-tuning a model usually means updating *all* its weights — that's the whole idea of "tuning." How do you change a model's behavior by leaving 99.8% of it frozen? What are you even training, if not the model?
+Every number in that sentence should feel slightly impossible. Fine-tuning a model usually means updating _all_ its weights — that’s the whole idea of “tuning.” How do you change a model’s behavior by leaving 99.8% of it frozen? What are you even training, if not the model?
-The answer is **LoRA** — Low-Rank Adaptation — and it's one of those ideas that feels like a magic trick until you see the mechanism, at which point it becomes obviously, almost boringly, correct. This post builds it from the ground up: what fine-tuning actually is, why it's so expensive, the one bet LoRA makes, and exactly how two tiny matrices pull it off. Two playgrounds along the way let you feel the trick instead of taking my word for it.
+The answer is **LoRA** — Low-Rank Adaptation — and it’s one of those ideas that feels like a magic trick until you see the mechanism, at which point it becomes obviously, almost boringly, correct. This post builds it from the ground up: what fine-tuning actually is, why it’s so expensive, the one bet LoRA makes, and exactly how two tiny matrices pull it off. Two playgrounds along the way let you feel the trick instead of taking my word for it.
-Let's start before LoRA, with plain fine-tuning.
+Let’s start before LoRA, with plain fine-tuning.
## Fine-tuning is just `W + ΔW`
-A neural network is a stack of weight matrices. Running it is a chain of matrix multiplies: an input vector $x$ meets a weight matrix $W$, producing an output $h = Wx$ that flows to the next layer. Those $W$ matrices — hundreds of millions to billions of numbers — *are* the model. All its knowledge lives in them.
+A neural network is a stack of weight matrices. Running it is a chain of matrix multiplies: an input vector $x$ meets a weight matrix $W$, producing an output $h = Wx$ that flows to the next layer. Those $W$ matrices — hundreds of millions to billions of numbers — _are_ the model. All its knowledge lives in them.
-**Fine-tuning** changes that knowledge. You show the model examples of the behavior you want, and for each weight matrix you compute an *update* — call it $\Delta W$ — that nudges the model toward the target. Then you apply it:
+**Fine-tuning** changes that knowledge. You show the model examples of the behavior you want, and for each weight matrix you compute an _update_ — call it $\Delta W$ — that nudges the model toward the target. Then you apply it:
$$
W' = W + \Delta W
$$
-That's the whole loop: run a forward pass, use backpropagation to compute a $\Delta W$ that lowers the loss, add it to $W$, repeat.
+That’s the whole loop: run a forward pass, use backpropagation to compute a $\Delta W$ that lowers the loss, add it to $W$, repeat.
-This works perfectly. It's how essentially every fine-tuned model was made for years. But look closely at that $\Delta W$: it has *exactly the same shape as $W$*. If $W$ is a $1000 \times 1000$ matrix, then $\Delta W$ is a million numbers too. You're learning and storing a second copy of the model's size, for every matrix, plus optimizer bookkeeping on top. This is **full fine-tuning**, and its cost is the whole problem.
+This works perfectly. It’s how essentially every fine-tuned model was made for years. But look closely at that $\Delta W$: it has _exactly the same shape as $W$_. If $W$ is a $1000 \times 1000$ matrix, then $\Delta W$ is a million numbers too. You’re learning and storing a second copy of the model’s size, for every matrix, plus optimizer bookkeeping on top. This is **full fine-tuning**, and its cost is the whole problem.
## The problem: `ΔW` is as big as the model
-Here's what "as big as the model" costs in practice. For each trainable number you don't just store the number — you store its gradient and (for the Adam optimizer) two running statistics, roughly **~16 bytes per parameter** during training.
+Here’s what “as big as the model” costs in practice. For each trainable number you don’t just store the number — you store its gradient and (for the Adam optimizer) two running statistics, roughly **\~16 bytes per parameter** during training.
-That's why full fine-tuning a 7B model doesn't fit on your laptop, or even on one datacenter GPU. And it feels wasteful: to teach a model to emit clean JSON, do we really need to relearn a fresh million-number update for *every* weight matrix? That question — *is the update really as complex as the model?* — is the crack LoRA pries open.
+That’s why full fine-tuning a 7B model doesn’t fit on your laptop, or even on one datacenter GPU. And it feels wasteful: to teach a model to emit clean JSON, do we really need to relearn a fresh million-number update for _every_ weight matrix? That question — _is the update really as complex as the model?_ — is the crack LoRA pries open.
## The bet: the update is simpler than the model
-LoRA's founding bet is a claim about $\Delta W$, not about $W$:
+LoRA’s founding bet is a claim about $\Delta W$, not about $W$:
-The pretrained model is genuinely complex — it took trillions of tokens to build. But the **change** needed to adapt it to one narrow task is *simple*. In linear-algebra terms, $\Delta W$ is **low-rank**: its information lives in only a handful of directions, even though it's written as a big matrix.
+The pretrained model is genuinely complex — it took trillions of tokens to build. But the **change** needed to adapt it to one narrow task is _simple_. In linear-algebra terms, $\Delta W$ is **low-rank**: its information lives in only a handful of directions, even though it’s written as a big matrix.
-"Low-rank" has a precise, visual meaning, and it's worth *seeing* before we use it. Any matrix can be built up as a sum of simple rank-1 pieces, ordered from most to least important. A **low-rank** matrix is one where the first few pieces already capture almost everything — the rest is nearly noise.
+“Low-rank” has a precise, visual meaning, and it’s worth _seeing_ before we use it. Any matrix can be built up as a sum of simple rank-1 pieces, ordered from most to least important. A **low-rank** matrix is one where the first few pieces already capture almost everything — the rest is nearly noise.
Drag the rank slider below. On the left is a full matrix; on the right is the same matrix rebuilt from only its first $r$ pieces. Watch how few you need before the two are indistinguishable:
@@ -96,17 +97,17 @@ Drag the rank slider below. On the left is a full matrix; on the right is the sa
-Notice the collapse: by rank 4 or 5 the reconstruction is essentially perfect, at a fraction of the numbers. **If a fine-tune's $\Delta W$ has this property — and empirically it does — then storing the full $\Delta W$ is enormously wasteful.** You could store just the first few pieces. That's precisely what LoRA does.
+Notice the collapse: by rank 4 or 5 the reconstruction is essentially perfect, at a fraction of the numbers. **If a fine-tune’s $\Delta W$ has this property — and empirically it does — then storing the full $\Delta W$ is enormously wasteful.** You could store just the first few pieces. That’s precisely what LoRA does.
## The trick: write `ΔW` as `A · B`
-Instead of learning a full $\Delta W$, LoRA learns it *already factored* into two skinny matrices whose product reconstructs a low-rank update:
+Instead of learning a full $\Delta W$, LoRA learns it _already factored_ into two skinny matrices whose product reconstructs a low-rank update:
$$
\Delta W \approx A \, B, \qquad A \in \mathbb{R}^{\,d_{out}\times r}, \quad B \in \mathbb{R}^{\,r\times d_{in}}
$$
-The shared inner dimension $r$ — the **rank** — is small (think 8, 16, 32). One matrix is *tall and skinny*, the other *short and wide*, and they meet at the tiny $r$:
+The shared inner dimension $r$ — the **rank** — is small (think 8, 16, 32). One matrix is _tall and skinny_, the other _short and wide_, and they meet at the tiny $r$:
-**Why two matrices instead of one?** Because a single matrix the size of $\Delta W$ would just *be* $\Delta W$ — no savings. The magic is that a tall × wide product *reconstructs* a big matrix from few numbers, and forcing them through the narrow $r$ is exactly what makes the result low-rank. `A·B` isn't notation for a single thing; it's a genuine factorization that trades a big matrix for two small factors.
+**Why two matrices instead of one?** Because a single matrix the size of $\Delta W$ would just _be_ $\Delta W$ — no savings. The magic is that a tall × wide product _reconstructs_ a big matrix from few numbers, and forcing them through the narrow $r$ is exactly what makes the result low-rank. `A·B` isn’t notation for a single thing; it’s a genuine factorization that trades a big matrix for two small factors.
-Letter conventions differ between sources — some write $\Delta W = BA$ with the letters swapped. Don't track the letters; track the **shapes**: one matrix is `(d_out × r)`, the other `(r × d_in)`, and they share the small `r`. Applied to an input, the wide one squeezes it *down* to `r` numbers, the tall one expands it back *up*.
+Letter conventions differ between sources — some write $\Delta W = BA$ with the letters swapped. Don’t track the letters; track the **shapes**: one matrix is `(d_out × r)`, the other `(r × d_in)`, and they share the small `r`. Applied to an input, the wide one squeezes it _down_ to `r` numbers, the tall one expands it back _up_.
-And crucially, you never choose the big dimensions. $d_{out}$ and $d_{in}$ come from the model's architecture; the library reads them off each weight and sizes $A$ and $B$ automatically. **The only number you pick is $r$.**
+And crucially, you never choose the big dimensions. $d_{out}$ and $d_{in}$ come from the model’s architecture; the library reads them off each weight and sizes $A$ and $B$ automatically. **The only number you pick is $r$.**
### The bottleneck: shrink to `r`, then expand
-This factorization has a physical meaning worth holding onto, because it *is* the intuition for why LoRA works. Reading the adapter in the order it actually runs — right to left, since $ABx = A(Bx)$ — it does two things:
+This factorization has a physical meaning worth holding onto, because it _is_ the intuition for why LoRA works. Reading the adapter in the order it actually runs — right to left, since $ABx = A(Bx)$ — it does two things:
-- **$B$ is a _down-projection_.** It takes the full $d_{in}$-dimensional input and squeezes it into just $r$ numbers. With only $r$ slots to write into, it is *forced* to keep the few input directions that matter for this task and discard the rest — it writes a summary.
-- **$A$ is an _up-projection_.** It takes those $r$ summary numbers and expands them back into a full $d_{out}$-dimensional adjustment — deciding how each captured direction should nudge the output.
+- **$B$ is a **_**down-projection**_**.** It takes the full $d_{in}$\-dimensional input and squeezes it into just $r$ numbers. With only $r$ slots to write into, it is _forced_ to keep the few input directions that matter for this task and discard the rest — it writes a summary.
+- **$A$ is an **_**up-projection**_**.** It takes those $r$ summary numbers and expands them back into a full $d_{out}$\-dimensional adjustment — deciding how each captured direction should nudge the output.
-So the update can never be arbitrary: it must pass through an $r$-wide **bottleneck** — read at most $r$ features going in, write at most $r$ patterns coming out. *That squeeze is the low-rank property, enforced by construction.* And it's enough, because — as the reconstruction playground showed — the change a fine-tune actually needs lives in only a handful of directions. LoRA doesn't *hope* the update is simple; it builds a pipe too narrow for it to be anything else, and lets training find the best simple update that fits through.
+So the update can never be arbitrary: it must pass through an $r$\-wide **bottleneck** — read at most $r$ features going in, write at most $r$ patterns coming out. _That squeeze is the low-rank property, enforced by construction._ And it’s enough, because — as the reconstruction playground showed — the change a fine-tune actually needs lives in only a handful of directions. LoRA doesn’t _hope_ the update is simple; it builds a pipe too narrow for it to be anything else, and lets training find the best simple update that fits through.
## Where the savings actually come from
@@ -166,19 +167,19 @@ $$
For a square $d \times d$ matrix, that simplifies to a clean rule: **LoRA saves when $r < d/2$.**
-The surprise: on a *small* matrix, LoRA saves nothing. A $4\times4$ update is 16 numbers; factored at $r=2$ it's $2\times(4+4) = 16$ — identical. That's the break-even point. The savings are a *big-matrix* phenomenon, and they grow explosively with size. Feel it:
+The surprise: on a _small_ matrix, LoRA saves nothing. A $4\times4$ update is 16 numbers; factored at $r=2$ it’s $2\times(4+4) = 16$ — identical. That’s the break-even point. The savings are a _big-matrix_ phenomenon, and they grow explosively with size. Feel it:
-Set it to the toy $4$ at $r=2$ and the bars are equal — no win. Now jump to a real attention matrix ($4096$) at $r=16$: the full update is 16.7 million numbers, LoRA's is 131 thousand — **128× smaller**, for the *same-shaped* update. Because real weight matrices are thousands wide and $r$ stays tiny, LoRA sits far below the $d/2$ break-even and the savings are enormous. That 0.2% from the opening is this ratio, summed over every matrix in the model.
+Set it to the toy $4$ at $r=2$ and the bars are equal — no win. Now jump to a real attention matrix ($4096$) at $r=16$: the full update is 16.7 million numbers, LoRA’s is 131 thousand — **128× smaller**, for the _same-shaped_ update. Because real weight matrices are thousands wide and $r$ stays tiny, LoRA sits far below the $d/2$ break-even and the savings are enormous. That 0.2% from the opening is this ratio, summed over every matrix in the model.
-This also answers "why not just make $A$ and $B$ full-size?" Full-size factors would be *more* numbers than $\Delta W$ itself — and no longer low-rank. The whole point is the skinny shared bottleneck.
+This also answers “why not just make $A$ and $B$ full-size?” Full-size factors would be _more_ numbers than $\Delta W$ itself — and no longer low-rank. The whole point is the skinny shared bottleneck.
-## Why untrained matrices don't break the model
+## Why untrained matrices don’t break the model
-Here's the objection that trips everyone up: $A$ and $B$ are new matrices. They know *nothing* — not language, not your task. If we bolt them onto a carefully pretrained model, shouldn't they inject garbage and wreck it?
+Here’s the objection that trips everyone up: $A$ and $B$ are new matrices. They know _nothing_ — not language, not your task. If we bolt them onto a carefully pretrained model, shouldn’t they inject garbage and wreck it?
The fix is a beautiful one-liner. In our letter convention, $A$ starts random and **$B$ starts as all zeros** (some libraries swap the letters). Because one factor is zero, the product is exactly zero:
@@ -188,23 +189,23 @@ $$
h = Wx + 0 = Wx
$$
-At step zero, the adapter contributes *nothing*. The model behaves **exactly** like the untouched base. Training then moves the zero matrix off zero and reshapes the random matrix with it, so the adapter grows from no-op to useful correction.
+At step zero, the adapter contributes _nothing_. The model behaves **exactly** like the untouched base. Training then moves the zero matrix off zero and reshapes the random matrix with it, so the adapter grows from no-op to useful correction.
| Moment | Adapter math | What the model receives |
-| ------ | ------------ | ----------------------- |
+| --- | --- | --- |
| Before training | $B = 0$, so $ABx = 0$ | $h = Wx$ — exactly the frozen base |
| During training | $B$ becomes nonzero, $A$ also adjusts | $h = Wx + ABx$ — base output plus a learned nudge |
| After training | $AB$ is the learned low-rank update | Same base model, different activations flowing forward |
-So the untrained adapter never starts as "random garbage that corrupts the model." It starts as a **zero delta**. The only thing training is allowed to learn is the extra nudge the frozen model needs: *"given the features the frozen model already computed here, push the activation this way."*
+So the untrained adapter never starts as “random garbage that corrupts the model.” It starts as a **zero delta**. The only thing training is allowed to learn is the extra nudge the frozen model needs: _“given the features the frozen model already computed here, push the activation this way.”_
-## How it's wired: a side path, not a new layer
+## How it’s wired: a side path, not a new layer
-So where does the adapter physically sit? Not as a new layer stacked on top, and not as a separate model running in parallel. It sits *beside each weight matrix it adapts*. The same input $x$ flows into both the frozen $W$ and the adapter, and their outputs are **summed**:
+So where does the adapter physically sit? Not as a new layer stacked on top, and not as a separate model running in parallel. It sits _beside each weight matrix it adapts_. The same input $x$ flows into both the frozen $W$ and the adapter, and their outputs are **summed**:
$$
h = W x + A B x = (W + \Delta W)\,x
@@ -245,7 +246,7 @@ $$
-The static diagram shows the wiring. The animation below shows what happens *inside* the lower path: $B$ compresses many input features into $r$ bottleneck slots, $A$ expands those few slots into a full-size correction, and the correction gets added to the frozen output. Toggle the adapter off to see the step-0 case, where $B = 0$ makes every bottleneck slot zero.
+The static diagram shows the wiring. The animation below shows what happens _inside_ the lower path: $B$ compresses many input features into $r$ bottleneck slots, $A$ expands those few slots into a full-size correction, and the correction gets added to the frozen output. Toggle the adapter off to see the step-0 case, where $B = 0$ makes every bottleneck slot zero.
@@ -254,13 +255,13 @@ The static diagram shows the wiring. The animation below shows what happens *ins
This settles two common misreadings at once:
- The base weights $W$ **never change** — they stay frozen, read-only.
-- But the **activation $h$ passed to the next layer does change** — it's the original $Wx$ plus the adapter's nudge. The correction ripples forward through the rest of the frozen network.
+- But the **activation $h$ passed to the next layer does change** — it’s the original $Wx$ plus the adapter’s nudge. The correction ripples forward through the rest of the frozen network.
-One more detail: "adapting the last few layers" doesn't mean one adapter per layer. Each *targeted weight matrix* (in a transformer: attention's query/key/value/output, the MLP projections) gets its own $A$–$B$ pair. "The adapter" you save at the end is the whole collection of these small pairs — a few megabytes total.
+One more detail: “adapting the last few layers” doesn’t mean one adapter per layer. Each _targeted weight matrix_ (in a transformer: attention’s query/key/value/output, the MLP projections) gets its own $A$–$B$ pair. “The adapter” you save at the end is the whole collection of these small pairs — a few megabytes total.
-Zooming all the way out, here's where those pairs actually live inside the model:
+Zooming all the way out, here’s where those pairs actually live inside the model:
-