An investment committee with different specializations outperforms any single analyst. Multi-head attention: each head learns different relationship patterns.— Day 17 Principle
I. Multiple Heads
Instead of one large head, use multiple smaller heads in parallel. 4 heads of size 16 vs 1 head of 64. Outputs are concatenated and projected.
class MultiHeadAttention(nn.Module):
def __init__(self, n_heads, head_size):
super().__init__()
self.heads = nn.ModuleList([Head(head_size) for _ in range(n_heads)])
self.proj = nn.Linear(n_heads * head_size, n_embd)
def forward(self, x):
out = torch.cat([h(x) for h in self.heads], dim=-1)
return self.proj(out)
II. Positional Encoding
Self-attention is permutation-invariant. Positional encodings add position information.
pos_emb = nn.Embedding(block_size, n_embd)
x = tok_emb + pos_emb(torch.arange(T))
Learned vs Sinusoidal
Original Transformer used sinusoidal. Modern LLMs use learned or RoPE. Learned is simpler for fixed context.
IV. The Matrix
Deep Intuition
Surface Only
Quick
🎯
DO FIRST
MultiHeadAttention with 4 heads + positional embeddings.
⏭
IF TIME
Visualize per-head attention patterns.
Slow
🖐
CAREFULLY
Remove positional encoding. Observe degradation.
🚫
AVOID
RoPE or ALiBi. Master learned embeddings first.
V. Today’s Deliverables
- MultiHeadAttention with concat + projection
- Positional embeddings
- Ablation without positions
- Per-head visualization
- Parameter comparison
Two of three Transformer pillars complete. Tomorrow: the full block.— Day 17 Closing