A matrix-factorized, ultra-low memory optimizer developed by Jassim Ibrahim Farooqi as part of ongoing research at OpenBlade into extreme-efficiency training architectures. By decoupling second moments into marginal rank-1 row and column projections and applying dynamic RMS update scaling, Avalanche operates within a constant ~10 MB memory ceiling—freeing hundreds of gigabytes of VRAM on foundation models.
pip install git+https://github.com/OpenBladeAI/Avalanche.git
Avalanche delivers full precision flexibility. While second-moment states always remain factorized in $O(M + N)$ (~10 MB VRAM), momentum velocity ($m_t$) adapts seamlessly to your GPU headroom without discarding any gradient information:
| Variant Class | Mode Flag | Momentum State | Quantization & Mechanics | Katana-1.1B VRAM | Net Savings |
|---|---|---|---|---|---|
| Avalanche (Zero-Momentum) | mode="none" |
0.0 B / param | Zero momentum buffer ($m=0$). Factorized $r \in \mathbb{R}^M, c \in \mathbb{R}^N$. | ~10.0 MB | 99.99% Reduction |
| Avalanche4Bit | mode="4bit" |
0.5 B / param | Packed 4-bit signed nibbles ($[-7, 7]$ in uint8) + Lossless Error-Feedback (EF) |
~2.69 GB | 68.1% Reduction |
| Avalanche8Bit | mode="8bit" |
1.0 B / param | Signed 8-bit integers ($[-127, 127]$ in int8) + Lossless Error-Feedback (EF) |
~3.80 GB | 54.9% Reduction |
| AvalancheFP16 | mode="fp16" |
2.0 B / param | IEEE 754 half-precision float + dynamic intermediate buffer cast-away | ~4.92 GB | 41.6% Reduction |
| AvalancheBF16 | mode="bf16" |
2.0 B / param | Bfloat16 precision with full 8-bit dynamic exponent range matching FP32 | ~4.92 GB | 41.6% Reduction |
| AvalancheFP32 | mode="fp32" |
4.0 B / param | Single-precision float velocity with rank-1 factorized second moments | ~7.16 GB | 15.0% Reduction |
Standard Adam requires 8 bytes per parameter (two FP32 states). Avalanche breaks this scaling law, shifting complexity from parameter area $O(M \cdot N)$ to parameter perimeter $O(M + N)$.
| Model Scale | Parameters | Standard FP32 AdamW ($m + v$) | 8-Bit Adam (bitsandbytes) | Avalanche Footprint | Net VRAM Savings |
|---|---|---|---|---|---|
| Small / Edge | 125 Million | ~1.00 GB | ~250 MB | ~1.8 MB | 99.82% Reduction |
| Medium / Foundation | 1.12 Billion | ~8.96 GB | ~2.24 GB | ~9.6 MB | 99.89% Reduction |
| 7B Base (e.g. LLaMA) | 7.00 Billion | ~56.00 GB | ~14.00 GB | ~24.5 MB | 99.96% Reduction |
| 13B Mid-Tier | 13.00 Billion | ~104.00 GB | ~26.00 GB | ~42.0 MB | 99.96% Reduction |
| 70B Frontier | 70.00 Billion | ~560.00 GB | ~140.00 GB | ~78.0 MB | 99.99% Reduction |
* Optimizer state memory measured directly from PyTorch allocated tensor buffers.
Mathematical breakdown of the four interlocking innovations that enable Avalanche's ~10 MB constant memory footprint.
For a weight tensor $W \in \mathbb{R}^{M \times N}$, Avalanche tracks two low-dimensional marginal vectors: row mean squared gradients $r \in \mathbb{R}^M$ and column mean squared gradients $c \in \mathbb{R}^N$.
For a 4096×4096 matrix, memory drops from 16.7M floats (67.1 MB) to 8,192 floats (32.8 KB) — a 99.95% reduction on every linear layer.
To eliminate loss spikes in mixed precision (BF16/FP16) without manual clipping, Avalanche calculates the root-mean-square norm of the raw update vector and bounds it to 1.0.
Guarantees update vectors remain bounded, completely preventing loss explosions even under sharp gradient transients.
When updating giant matrices (e.g., 65k vocabulary embeddings with >4M parameters), reconstructing full $\hat{v}$ at once would cause a temporary VRAM allocation spike.
Slices updates into bounded tiles, strictly enforcing a flat memory ceiling regardless of vocabulary size or embedding dimension.
First-moment tracking ($m$) consumes 50% of AdamW's memory. In deep transformer architectures, per-coordinate variance normalization dominates convergence speed.
Eliminates gigabytes of momentum tensors without sacrificing convergence rate or final loss.
Avalanche inherits from torch.optim.Optimizer. It works identically to torch.optim.AdamW with zero code refactoring required.
import torch
import torch.nn as nn
from avalanche import (
Avalanche, # Zero-momentum (99.99% reduction, ~10 MB VRAM)
Avalanche4Bit, # 4-bit packed momentum with lossless error-feedback (0.5 B/param)
Avalanche8Bit, # 8-bit momentum with lossless error-feedback (1.0 B/param)
AvalancheBF16, # Native Bfloat16 momentum (2.0 B/param)
AvalancheFP16, # Native FP16 momentum (2.0 B/param)
AvalancheFP32, # FP32 momentum with factorized 2nd moments (4.0 B/param)
)
# 1. Initialize your model
model = nn.Transformer(d_model=1024, nhead=16).cuda()
# 2. Drop in your desired Avalanche precision variant
optimizer = Avalanche4Bit(
model.parameters(),
lr=1e-4,
betas=(0.9, 0.999), # Momentum enabled!
lossless=True, # Error feedback active (no gradients/steps discarded)
weight_decay=0.01,
)
# 3. Standard training pass
for x, targets in dataloader:
optimizer.zero_grad(set_to_none=True)
logits = model(x)
loss = criterion(logits, targets)
loss.backward()
optimizer.step()Avalanche is completely standalone. Download the single-file implementation directly into your project or install via pip.
@software{openblade2026avalanche,
author = {Jassim Ibrahim Farooqi and OpenBlade},
title = {Avalanche: Ultra-Low VRAM Matrix-Factorized Optimizer for Deep Neural Networks},
year = {2026},
publisher = {GitHub},
howpublished = {\url{https://github.com/OpenBladeAI/avalanche}},
url = {https://github.com/OpenBladeAI/avalanche},
note = {An OpenBlade release. Ongoing research and optimizations into extreme-efficiency training architectures.}
}