NVFP4 Is a Scaling System, Not Just a 4-Bit Number
What the payload, block scale, and tensor scale each do, and when a 4-bit claim holds up
Suppose a model tensor is stored in NVFP4. It is tempting to picture every entry as a miniature, self-contained FP16 value: four bits go in, one approximate real number comes out, and the rest is a hardware detail.
That picture is incomplete. The four bits only select from a tiny set of normalized values. To reconstruct a value in the tensor’s actual range, NVFP4 also needs a scale shared by a small block and another scale shared by the whole tensor.
The question I want to resolve is therefore narrower than “Is FP4 good?” It is: what job do the payload, block scale, and tensor scale each perform, and when can that representation help a real workload?
I will assume familiarity with tensors, ordinary floating point, quantization, and matrix multiplication. Everything specific to FP4 and NVIDIA’s implementation starts from scratch.
Four bits do not provide many choices
A floating-point layout is often named by how many exponent and mantissa bits it contains. E2M1 has one sign bit, two exponent bits, and one mantissa bit. Together, they make a four-bit payload, so E2M1 is one kind of FP4.
Ignoring the sign, its finite magnitudes are:
0, 0.5, 1, 1.5, 2, 3, 4, 6
The sign bit mirrors the nonzero values below zero. This is a very small codebook, meaning the set of values the bit patterns can select. NVIDIA documents this E2M1 payload as the element format used by NVFP4 (Transformer Engine documentation).
Now consider a block whose useful values cluster near 0.01, and another whose values cluster near 100. A bare E2M1 codebook cannot closely cover both neighborhoods. Its largest finite magnitude is 6, and the gaps between available values are large relative to many tensors.
Quantization solves this by treating the codebook values as numbers in a local unit rather than final tensor values. A scale defines that unit. The payload says “four units”; the scale says how large one unit is here.
The smallest example: the same payload, two real values
NVFP4 reconstructs a value with three factors:
x = x_e2m1 × s_block × s_global
Their scopes are more important than their names:
x_e2m1is the four-bit code stored for one element.s_blockis a scale shared by a local block of 16 elements.s_globalis a scale shared by the whole tensor.
Take two 16-value blocks and fix the tensor-wide scale at s_global = 0.25.
Tensor-wide scale: s_global = 0.25
Block A: s_block = 1.0 Block B: s_block = 1.5
payload = 4 payload = 4
| |
v v
4 × 1.0 × 0.25 = 1.0 4 × 1.5 × 0.25 = 1.5Both elements contain the same E2M1 payload. Yet one reconstructs to 1.0 and the other to 1.5, because each block uses a different local unit. The payload is not a standalone measurement; it is a code interpreted within two shared coordinate systems.
The explorer below lets you vary those three factors separately. Choose an exact E2M1 payload, then move the block scale or tensor scale. The block control changes the unit for one toy 16-value neighborhood; the tensor control changes the unit shared by every block. The blue result is the reconstructed value, not a quantization-error or model-quality measurement.
Toy explorer: one stored code, two scaling levels
Toy calculation, not benchmark data. The controls use exact codebook values and simplified scale choices to expose the reconstruction rule.
With the defaults, the stored code 4 becomes 1.5 in real tensor units.
Try leaving the payload at 4 and changing only the block scale from 1.0 to 1.5. The selected four-bit value does not move, but the reconstructed result does. Changing the tensor scale then moves every local unit together. This is the division of labor that the phrase “two-level scaling” can otherwise hide.
Why two levels instead of one? The block scale adapts the codebook to a small neighborhood, where values are more likely to have a similar range. The global scale places those local scales within the range that their own storage format can represent. Transformer Engine describes the same hierarchy and derives the global scale from the tensor’s maximum absolute value before deriving local scales from block maxima (Transformer Engine documentation).
The format names can now be attached without hiding their roles:
- The element payload is E2M1, a four-bit floating-point code.
- Each group of 16 payloads shares an E4M3 scale. E4M3 is an eight-bit floating-point format with four exponent and three mantissa bits, so its scale can take fractional values between powers of two.
- The tensor shares an FP32 global scale, stored in ordinary 32-bit floating point.
This hierarchy is what two-level scaling means in NVFP4: a local scale per block, followed by a global scale per tensor. Per-block scaling adapts to variation among small groups. Per-tensor scaling provides one range-setting factor for the entire tensor. Neither phrase says that every operation or every piece of model state is stored at that precision.
“4-bit” is a payload count, not a memory measurement
Once the local scale is visible, a common storage shortcut breaks.
One block contains 16 four-bit payloads and one eight-bit local scale:
16 × 4 + 8 = 72 bits
Amortized across the block:
72 / 16 = 4.5 bits per value
So the basic payload-plus-local-scale representation is already 4.5 bits per value, not four. The tensor’s FP32 global scale adds a small amortized cost whose size depends on the number of elements in that tensor.
The next explorer makes that amortization visible. Move the slider to change the number of 16-value blocks in one toy tensor. It counts four payload bits per value, one eight-bit local scale per block, and one 32-bit global scale for the tensor. It intentionally excludes padding, alignment, duplicate layouts, high-precision components, and runtime buffers, so its result is a representation calculation rather than an application-memory forecast.
Toy explorer: payload bits plus scale metadata
Defined assumptions: 16 values per block, 4 payload bits per value, one 8-bit block scale, and one 32-bit tensor scale. No padding or runtime state is included.
At 1,024 values, the single global scale is already thinly amortized. The local scale still contributes 0.5 bit per value.
At the smallest setting, the one global scale is expensive because only 16 values share it. As the toy tensor grows, that one-time cost fades, while the local-scale cost remains fixed at 8 / 16 = 0.5 bit per value. The output approaches 4.5 bits per value but never turns the calculation into a full-memory estimate.
Even 4.5 bits is not a full runtime-memory prediction. Transformer Engine documents padded and swizzled scale layouts and a TN-only NVFP4 GEMM layout; training may also keep rowwise and columnwise representations of weights (Transformer Engine documentation). Alignment, padding, higher-precision layers, activations, optimizer state, KV cache, workspaces, and framework allocations all remain outside the small calculation.
Comparing payloads alone, four bits are one quarter of a BF16 payload and one half of an FP8 payload. Those ratios are useful for understanding the upper-level opportunity. They do not establish a fourfold reduction in total application memory.
Generic FP4, MXFP4, and NVFP4
“FP4” is not a complete deployment specification. It can identify a four-bit element format such as E2M1 without telling us how a tensor is scaled. Two systems can store the same E2M1 payload and reconstruct it differently.
MXFP4 makes the comparison concrete. MXFP4, the FP4 member of the OCP microscaling family, also uses E2M1 payloads. It shares one E8M0 scale across 32 values. E8M0 has eight exponent bits and no mantissa, so its scale is restricted to powers of two. NVFP4 instead shares an E4M3 fractional scale across 16 values and adds the FP32 scale for the tensor (NVIDIA’s NVFP4 technical overview).
| Property | Generic FP4 / E2M1 | MXFP4 | NVFP4 |
|---|---|---|---|
| Element payload | May be E2M1 | E2M1 | E2M1 |
| Local scale | Not specified by the name | E8M0, powers of two | E4M3, fractional FP8 |
| Values per local scale | Not specified | 32 | 16 |
| Second scaling level | Not specified | None in the MX definition | One FP32 scale per tensor |
The table is a comparison of representations, not model quality. NVFP4 gets two ways to fit a local distribution more closely: its scale covers half as many elements, and that scale can sit between powers of two. NVIDIA illustrates lower quantization error from these choices, but a representation-error example is not an end-to-end accuracy result (NVIDIA’s NVFP4 technical overview). Smaller blocks also mean more scale metadata and handling.
The better mental model is now precise: generic FP4 names a payload class; MXFP4 and NVFP4 specify different systems around an E2M1 payload.
A format is not a training recipe
If scaling were the whole story, low-precision training would amount to casting tensors and calling the same matrix multiplications. NVIDIA’s documented NVFP4 training path contains several additional mechanisms.
Two-dimensional weight scaling changes the grouping used for weights. Transformer Engine uses 16-by-16 tiles by default so rowwise and columnwise weight representations can support the forward and backward matrix multiplications. Activations and gradients use one-dimensional 16-value blocks.
Stochastic rounding chooses probabilistically between neighboring representable values instead of always taking the nearest one. For a gradient between two E2M1 values, the probabilities are chosen so the expected rounded value equals the original value. Transformer Engine enables this for gradient quantization. “Unbiased in expectation” does not mean an individual rounding is exact, and the added randomness has reproducibility implications.
A Random Hadamard Transform is an orthogonal mixing operation applied to selected operands before quantization. Its purpose here is to spread isolated large values across coordinates so that one outlier is less likely to set an unhelpful scale for an entire block. Transformer Engine currently documents this path for BF16 inputs and gradients used in selected weight-gradient operations (Transformer Engine documentation).
The broader recipe also retains higher precision for sensitive layers or operations. In other words, “NVFP4 training” is mixed-precision training built around NVFP4 matrix multiplications, not a claim that the whole training state and every computation use four bits.
NVIDIA reports pretraining a 12-billion-parameter model for 10 trillion tokens with loss and downstream accuracy comparable to an FP8 baseline using this combined method (NVFP4 training paper). This is evidence for the model, data, schedule, hardware, and recipe in that experiment. It is not evidence that naive E2M1 conversion works for arbitrary architectures, or that every model will match FP8.
Hardware support still needs a full sentence
NVFP4 is designed for native block-scaled matrix operations on NVIDIA Blackwell Tensor Cores. But “Blackwell supports NVFP4” leaves out which operation, device, software path, and tensor are involved.
Transformer Engine 2.16 documents NVFP4 training on compute capabilities SM100 and SM103, and inference on SM100 or later. It exposes the recipe to PyTorch and JAX through NVFP4BlockScaling, while also documenting layout and distributed-training constraints (Transformer Engine documentation). These are versioned software conditions, not permanent properties of every Blackwell deployment.
Inference has at least two separate quantization targets:
- Weights and activations are operands used by model computations.
- The KV cache stores attention keys and values from earlier tokens so autoregressive decoding does not recompute them.
TensorRT-LLM’s quantization matrix, inspected in the source package after its 2026-07-14 update, listed NVFP4 weights and activations on Blackwell SM100/103 and SM120. Its NVFP4 KV-cache support was narrower, limited to SM100/103, and the documented KV-cache combination required FP8 weights and activations (TensorRT-LLM quantization guide). Because this support changes with releases, the live matrix should be checked for the exact TensorRT-LLM version before deployment.
Training GEMMs, weight-and-activation inference, and KV-cache compression are therefore three different decisions. Each needs an exact compute capability, framework version, model and layout, calibration or training path, and quality test.
Where speed and accuracy claims can fail
Native Tensor Core execution creates a credible path to higher matrix throughput and lower operand bandwidth. End-to-end performance improves only if supported matrix multiplications dominate the workload.
Quantization and scale handling consume work. Attention, communication, unsupported operators, memory bandwidth, small tensor shapes, and irregular layouts may dominate instead. A faster GEMM can simply move the bottleneck elsewhere. Peak hardware throughput is not a substitute for measuring prefill latency, decode latency, generation throughput, or training step time at the intended shapes and batch sizes.
Accuracy needs a separate boundary. Finer local scaling can reduce the numerical error introduced while representing a tensor. Model quality also depends on where quantization is applied, how calibration is performed, whether training adapts to it, which computations stay at higher precision, and how errors propagate through the network. Lower error in a toy tensor does not prove equal language-model accuracy, much less equal behavior on rare or sensitive cases.
The strongest large-scale result cited above was produced by NVIDIA for a particular pretraining experiment. Independent behavior on a target model, task distribution, context length, and deployment stack remains unknown until it is measured.
A decision test for an NVFP4 claim
When a benchmark or deployment proposal says “NVFP4,” I would unpack it in this order:
- Name the quantized object. Is it weights, activations, training GEMM operands, or the KV cache?
- Count the representation. Include local and global scales, padding, duplicate layouts, and any higher-precision components instead of counting only E2M1 payload bits.
- Verify the supported path. Record the compute capability, software versions, model family, tensor layout, and allowed precision combination.
- Describe how the artifact was made. Distinguish a supported pre-quantized checkpoint, calibrated post-training quantization, and training with the full mixed-precision recipe.
- Validate quality on the intended workload. Compare with BF16 or FP8 on held-out tasks, long contexts, outlier-heavy inputs, and rare behaviors that matter.
- Measure the system boundary. Report checkpoint size, peak memory, KV-cache size where relevant, end-to-end latency or step time, throughput, and quality under the same conditions.
If those questions are unanswered, “4-bit” tells us the width of the element payload and little more.
NVFP4’s useful idea is not merely squeezing a floating-point value into four bits. It is giving the same tiny E2M1 codebook different local units, then putting those units into a tensor-wide range. That hierarchy can represent tensors more closely than a coarser power-of-two scheme, and Blackwell hardware can execute supported operations natively. The remaining memory, accuracy, and speed claims belong to the whole mixed-precision system and the workload running through it, not to the four payload bits alone.