From Softmax Attention to Kimi Delta Attention

Reframing softmax attention to see the bottleneck#

For a query at position $t$, causal softmax attention computes the output as a weighted sum over all previous tokens:

$o_t = \sum_{i \le t} \frac{\exp(q_t^\top k_i)}{\sum_{j \le t} \exp(q_t^\top k_j)} \, v_i$

the output at $t$ is a blend of every past value $v_i$, weighted by how well each token's key $k_i$ matches the current query $q_t$, where the match scores are normalized across all past tokens so the weights sum to 1.

The two costs (the reason linear attention exists)#

Compute. To produce all outputs for a sequence of length $T$, you compute $q_t^\top k_i$ for every pair. That's $T^2$ dot products. Double the sequence, quadruple the work.

Memory. To compute $o_t$ you need every past $k_i$ and $v_i$. Nothing lets you throw any away, because the next query might match an old key strongly. So at decode you cache all past keys and values (the KV cache), and it grows linearly with sequence length.

Why you can't compress the cache: the exponential. Because $\exp$ sits between $q_t$ and $k_i$, the weight for token $i$ can't be computed until $q_t$ is known. You cannot pre-summarize the past into a small fixed state, because the past has to wait for the future query before it knows how much to contribute.

The question that sets up everything: what if we removed the $\exp$ and replaced it with something that lets us reassociate the math, so the entire past collapses into a single fixed-size state each new token updates in constant work?

---

Linear attention, folding the past into a fixed state#

Why the exponential blocks compression#

$\exp(q_t^\top k_i)$ is non-separable. There's no way to split it into a piece depending only on $q_t$ times a piece depending only on $k_i$. Query and key are fused inside one nonlinear function, so no useful precomputation on keys alone is possible.

What becomes possible without it#

Drop the exponential, let the weight be the plain dot product $q_t^\top k_i$. Now query and key are only multiplied, which is separable, so the sum can be rearranged:

$o_t = \sum_{i \le t} (q_t^\top k_i)\, v_i$

Since $q_t^\top k_i$ is a scalar, move it around freely:

$(q_t^\top k_i)\, v_i = v_i\,(k_i^\top q_t) = (v_i k_i^\top)\, q_t$

Substitute back and pull $q_t$ out of the sum (it doesn't depend on $i$):

$o_t = \left( \sum_{i \le t} v_i k_i^\top \right) q_t$

Define the state:

$S_t = \sum_{i \le t} v_i k_i^\top$

This is the compression. $S_t$ is a single matrix of fixed size $d_v \times d$. It does not grow with $t$. It's a running sum:

$\boxed{S_t = S_{t-1} + v_t k_t^\top \qquad\qquad o_t = S_t\, q_t}$

Each new token costs constant work: form the outer product $v_t k_t^\top$, add it to the running state, read out with $q_t$. No scanning, no growing cache. Compute drops from $T^2$ to linear, memory becomes a constant $d_v \times d$ state.

Caveat: real linear attention keeps a small nonlinearity via a feature map $\phi$, so the weight is $\phi(q_t)^\top \phi(k_i)$. Still separable, so the same state trick applies with $\phi(k)$ in place of $k$. What matters is separability, not zero nonlinearity.

The outer product is the storage primitive#

Everything about the memory rests on one operation: the outer product $v k^\top$. This is what writes a single key-value pair into the matrix. Build up its behavior in three steps.

Step 1: an outer product turns a pair into a matrix. $v k^\top$ is a column vector times a row vector, which produces a matrix (not a number). Each entry:

$(v k^\top)[r, c] = v[r] \cdot k[c]$

Every component of $v$ is paired with every component of $k$. With $v = [2, 0]$ and $k = [1, 0]$:

$v k^\top = \begin{bmatrix}2\\0\end{bmatrix}\begin{bmatrix}1 & 0\end{bmatrix} = \begin{bmatrix}2\cdot 1 & 2\cdot 0\\ 0\cdot 1 & 0\cdot 0\end{bmatrix} = \begin{bmatrix}2 & 0\\ 0 & 0\end{bmatrix}$

Step 2: the key decides where the value is filed. All the content above landed in column 1, because $k = [1,0]$ has its weight in position 1. Change to $k = [0,1]$ and the same value files into column 2:

$\begin{bmatrix}2\\0\end{bmatrix}\begin{bmatrix}0 & 1\end{bmatrix} = \begin{bmatrix}0 & 2\\ 0 & 0\end{bmatrix}$

Think of filing a document. The key is the address (picks which column/direction the value is written into); the value is the contents (what gets stored there). One outer product is one filled slot, holding the binding "$k \rightarrow v$."

Step 3: reading a slot back with a query. Multiply the slot by a query vector $x$. It simplifies:

$(v k^\top)\, x = v\,(k^\top x)$

$k^\top x$ is a single scalar measuring how aligned the query is with the stored key. It acts as a volume knob on the value: query along $k$, get $v$ at full strength; query perpendicular to $k$ ($k^\top x = 0$), get nothing; partial alignment gives a scaled-down $v$. Check with the slot from Step 1, queried by $x = [1,0]$:

$\begin{bmatrix}2&0\\0&0\end{bmatrix}\begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix} = v$

Putting many slots together. A real memory holds many pairs. Add their outer products into one matrix:

$S = \sum_i v_i k_i^\top$

Addition preserves the shape, so $S$ stays fixed-size no matter how many pairs you store. Each pair was filed into its own key-direction, and they coexist in one matrix. Read the whole memory at once by querying with a key $k_j$:

$S k_j = \sum_i v_i \,(k_i^\top k_j)$

Every stored value $v_i$ comes back, each weighted by how much its key $k_i$ aligns with your query $k_j$. Values whose keys point toward $k_j$ dominate; the rest fade out. That is a soft lookup, and it's the entire reason the matrix behaves like an associative memory.

Worked storage example. Store $k_1=[1,0], v_1=[5,0]$ and $k_2=[0,1], v_2=[0,9]$:

$v_1 k_1^\top = \begin{bmatrix} 5 & 0 \\ 0 & 0\end{bmatrix}, \quad v_2 k_2^\top = \begin{bmatrix} 0 & 0 \\ 0 & 9\end{bmatrix}, \quad S = \begin{bmatrix} 5 & 0 \\ 0 & 9\end{bmatrix}$

Different keys wrote into different columns, so values never collided. Reading with $k_1$: $S k_1 = [5,0] = v_1$. Reading with $k_2$: $S k_2 = [0,9] = v_2$. Clean, because the keys are orthogonal.

Interference appears when keys are not orthogonal. If $k_2 = [0.7, 0.7]$, then querying $k_1=[1,0]$ gives $v_1(1) + v_2(0.7)$: the wrong value leaks in. Non-orthogonal keys write into overlapping columns, so reads mix.

Where are the savings? The keys are not stored#

The key is consumed at write time and discarded. In $S_t = S_{t-1} + v_t k_t^\top$, once $k_t$'s contribution is folded in, $k_t$ ceases to exist as a separate object. Retrieval never matches against stored keys, it just multiplies by the query, $o_t = S_t q_t$. The key-matching was already baked into the geometry of $S$ at write time.

	KV cache (softmax)	Linear attention
Stores	every $(k_i, v_i)$, size $T\cdot(d+d_v)$	one matrix $S$, size $d_v \times d$
Grows with $T$?	yes	no
Recovery	exact, fully separable	lossy, superimposed

The keys aren't stored twice, they're stored zero times. They were spent to build $S$ and discarded. The price is that summed outer products can't be pulled apart again: lossy compression.

$S q$ is attention, in the opposite association order#

Substitute $S_t = \sum_i v_i k_i^\top$ into the read and push $q_t$ inside:

$o_t = S_t\, q_t = \left(\sum_{i \le t} v_i k_i^\top\right) q_t = \sum_{i \le t} v_i (k_i^\top q_t) = \sum_{i \le t} (q_t^\top k_i)\, v_i$

That last form is exactly standard (softmax-free) attention. Same sum, same scores, same weighted blend. The only difference is when you group:

$\underbrace{\sum_i (q_t^\top k_i)\, v_i}_{\text{score first, then sum}} \quad=\quad \underbrace{\Big(\sum_i v_i k_i^\top\Big) q_t}_{\text{sum first, then score}}$

• Score-first (softmax) can't pre-sum, because scores need the future query. Holds the full list. Memory and work grow with $t$.
• Sum-first (linear) pre-sums the past into $S$ before the query arrives. Holds one fixed matrix. Constant memory and work. The softmax's $\exp$ is exactly what forbids the regrouping. Linear attention insists on a regroupable form and accepts the blur as the cost.

---

DeltaNet, giving the memory an eraser#

Flaw in plain linear attention: it only ever adds. It never removes. Two problems:

1. Interference. Non-orthogonal keys leak into each other on read.
2. No overwrite. The same key with two values stores a smeared superposition $v_a k^\top + v_b k^\top$, with no way to replace the old binding.

DeltaNet's fix: before writing, check what memory already returns for this key, and write only the correction.

$\hat v_t = S_{t-1} k_t \qquad \delta_t = v_t - S_{t-1} k_t$

The gap $\delta_t$ is the delta. Write only that correction, scaled by write strength $\beta_t$:

$S_t = S_{t-1} + \beta_t\,(v_t - S_{t-1} k_t)\, k_t^\top$

Expand and group the $S_{t-1}$ terms:

$\boxed{S_t = S_{t-1}\underbrace{(I - \beta_t k_t k_t^\top)}_{\text{erase}} + \underbrace{\beta_t v_t k_t^\top}_{\text{write}}}$

The erase operator $(I - \beta_t k_t k_t^\top)$ strips out existing content along the $k_t$ direction before the fresh value goes in. $\beta_t$ acts like a learning rate: 0 leaves memory untouched, 1 fully wipes and replaces along $k_t$.

Note: $k_t^\top k_t$ vs $k_t k_t^\top$#

• $k_t^\top k_t = \|k_t\|^2 = \sum_i k_t[i]^2 \ge 0$ is a scalar (squared norm). Never zero for a nonzero vector, so the erase always subtracts something.
• $k_t k_t^\top$ is a matrix (outer product), the thing inside $(I - \beta_t k_t k_t^\top)$.

Worked collision example#

Tokens (note $k_3 = k_1$, a deliberate collision), $\beta = 1$ throughout:

$k_1=[1,0], v_1=[2,0] \quad k_2=[0,1], v_2=[0,3] \quad k_3=[1,0], v_3=[5,0]$

Plain linear attention (add only):

$S_1 = \begin{bmatrix}2&0\\0&0\end{bmatrix} \to S_2 = \begin{bmatrix}2&0\\0&3\end{bmatrix} \to S_3^{\text{plain}} = \begin{bmatrix}7&0\\0&3\end{bmatrix}$

Column 1 became $7 = 2 + 5$: the values smeared together.

DeltaNet (erase before write). At token 3, $I - k_3 k_3^\top = \begin{bmatrix}0&0\\0&1\end{bmatrix}$. Apply to $S_2$ first:

$\begin{bmatrix}0&0\\0&1\end{bmatrix}\begin{bmatrix}2&0\\0&3\end{bmatrix} = \begin{bmatrix}0&0\\0&3\end{bmatrix}$

Column 1 (token 1's stale value) is zeroed; column 2 (token 2) untouched. Then write $v_3$:

$S_3^{\text{DeltaNet}} = \begin{bmatrix}0&0\\0&3\end{bmatrix} + \begin{bmatrix}5&0\\0&0\end{bmatrix} = \begin{bmatrix}5&0\\0&3\end{bmatrix}$

Reading with $q=[1,0]$ gives $[5,0] = v_3$ cleanly. The stale $v_1$ is gone, not smeared.

Honest limit: $S$ has fixed capacity $d_v \times d$. It can't hold unlimited separable bindings. Past capacity, something gives: overwrite or interference. This is the fundamental price of refusing to let memory grow, and it's exactly why Kimi Linear is a hybrid (3 KDA layers : 1 full-attention layer): the full-attention layer keeps an exact view as a safety net.

---

Gated DeltaNet, a fading knob#

Flaw in DeltaNet: the erase only touches the single direction $k_t$. Every other direction sits frozen forever. No general fading, so stale content clogs the fixed-capacity matrix.

Fix: a decay gate $\alpha_t$ that shrinks the whole state before each update:

$\boxed{S_t = \alpha_t\underbrace{(I - \beta_t k_t k_t^\top)\,S_{t-1}}_{\text{DeltaNet erase}} + \beta_t v_t k_t^\top}$

$\alpha_t \in (0,1)$ is a scalar computed per token. It uniformly scales down everything stored, before the targeted erase and write.

• $\alpha_t = 1$: nothing fades, back to plain DeltaNet.
• $\alpha_t = 0$: entire past wiped, restart from this token.
• $\alpha_t = 0.98$: gentle decay, ~2% lost per step, soft half-life.

Because $\alpha_t$ is per token, the model learns when to hold and when to flush (like an LSTM forget gate, applied to a matrix-valued memory).

Decay chain. A value written at step $i$ and read at step $t$ has been multiplied by $\alpha_{i+1}\alpha_{i+2}\cdots\alpha_t$. That product gives each memory a controllable lifespan.

Position for free. How much a value has faded signals how long ago it was written. So the gates encode recency and ordering, which is why these architectures can drop explicit positional encodings like RoPE.

Flaw that motivates KDA: $\alpha_t$ is a single scalar. Every feature dimension decays at the same rate. But different features live on different timescales (document-wide topic vs. local detail). One shared knob is forced into a compromise.

---

Concept 5: Kimi Delta Attention, per-channel forgetting#

The one change: replace the scalar $\alpha_t$ with a diagonal matrix $\text{Diag}(\alpha_t)$, where $\alpha_t$ is a vector with one decay value per channel:

$\boxed{S_t = (I - \beta_t k_t k_t^\top)\,S_{t-1}\,\text{Diag}(\alpha_t) + \beta_t v_t k_t^\top}$

$\text{Diag}(\alpha_t)$ holds $[\alpha_t^{(1)}, \ldots, \alpha_t^{(d)}]$ on its diagonal. Each key-channel of the memory now fades at its own rate. The erase and write are unchanged; only the forgetting got finer.

Numerical contrast#

State $S = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}$.

Gated DeltaNet, scalar $\alpha_t = 0.7$ (both channels lose 30%, no choice):

$0.7 \cdot \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix} = \begin{bmatrix} 1.4 & 0 \\ 0 & 2.1 \end{bmatrix}$

KDA, per-channel $\alpha_t = [0.9,\ 0.5]$ (each column its own rate):

$\begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}\text{Diag}([0.9,\ 0.5]) = \begin{bmatrix} 1.8 & 0 \\ 0 & 1.5 \end{bmatrix}$

Channel 1 keeps 90%, channel 2 is halved. One update, two forgetting speeds.

Why it matters#

Long-range channels (overall subject) want $\alpha \approx 1$ and persist. Short-range channels (local detail) want small $\alpha$ and flush fast. A single scalar can't serve both. Per-channel decay lets each feature choose, making better use of the fixed-size memory, which is KDA's headline for long-context recall.

Richer position. With per-channel decay, each channel carries its own decay timeline, so the memory holds a multi-rate sense of position. This is why KDA needs no explicit RoPE at all, and the positional signal is more expressive than a scalar could give.

---

The full arc#

Stage	Update	What it adds
Softmax attention	$o_t = \sum_i \text{softmax}(q_t^\top k_i)\, v_i$	exact recall, but growing cache
Linear attention	$S_t = S_{t-1} + v_t k_t^\top$	fixed matrix via associativity (lossy)
DeltaNet	$S_t = S_{t-1}(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top$	erase, so writes overwrite not smear
Gated DeltaNet	$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t v_t k_t^\top$	scalar decay, stale memory fades
KDA	$S_t = (I - \beta_t k_t k_t^\top) S_{t-1} \text{Diag}(\alpha_t) + \beta_t v_t k_t^\top$	per-channel decay

---

Reference#

Kimi Linear: An Expressive, Efficient Attention Architecture (arXiv:2510.26692)