GPUMODE H100 MATMUL V2 #1

November 2026

Matmuls are super popular and are extremely optimized. On H100s, cuBLAS gets near maximal perf so beating it is pretty rough without per shape specialization. I'm writing up tricks I used to get to #1 for the GPU MODE matmul_v2 leaderboard.

The task is C = A·B in FP16, and only one shape is ranked: M = 4096, N = 5120, K = 4096. Every timed call follows an L2 flush, and the output must match torch.matmul with rtol = 1e-5. On FP16 outputs that tolerance is close to bit-for-bit equality with cuBLAS, and it shaped most of the design. The problem is 171.8 GFLOP, which at the H100's dense FP16 peak of about 989 TFLOPS takes 174 µs.

Kernel correctness rules out the usual tricks

FP addition is not associative, so the order in which we sum K matters. For the ranked shape, cuBLAS adds K in order into one accumulator per output element. This matters bc to match it we basically cannot use split-K and stream-K, which are semi-standard ways to balance the end of GEMM across SMs. TLDR, you can't mess with reduction order.

I chose Gluon for the kernel. Gluon is practically made for agents, so I figured it would help others the most if I did it this way, also I find Gluon really neat for kernel authoring. I'm sure CUDA would've been "better" but you can do that with my kernels if you'd like. In the GPU MODE linalg series, I used CUDA only and got #9 on the Cholesky comp, but I wasn't too satisfied with the level of effort I needed to put in, and the amount of agents I threw. I literally could NOT afford even a quarter of that usage this time.

This was the rough starting point design:

gl.warp_specialize([
    (wg_partition, (..., 0)),   # consumer wg 0: rows 0-63
    (wg_partition, (..., 1)),   # consumer wg 1: rows 64-127
    (load_partition, (...)),    # producer: tma loads only
], [4, 1], [232, 40])           # workers: warps, regs/thread

One note: Hopper's setmaxnreg lets the producer drop to 40 registers per thread so each consumer thread can hold 232. A consumer needs 128 of them just for the 64×256 FP32 accumulator.

4th pipeline stage

When profiling, I discovered the consumers were spending 29% of their stall samples waiting for data. 3 of the 48 KB stages didn't keep enough bytes in flight to cover the load latency. A 4th stage needs another 48 KB, and the 64 KB buffer for staging the output tile was in the way. Writing the output in two 128×128 halves through one 32 KB buffer freed exactly enough: 4 × 48 + 32 = 224 KB, of the 227 KB available. Data stalls fell to 2.4% and the time dropped by 1.4%. From then on the consumers mostly waited on their own tensor-core instructions.

Tile order

The 132 tiles running in parallel determine what the 50 MB L2 can share. Tiles are grouped 16 tile-rows at a time. The groups alternate direction: the second group runs right to left, so at the turn it reuses the B columns that just loaded.[2] This optimization cut DRAM reads from 239 MB to 181 MB and the time by about 0.35%.

Filling the last wave

640 tiles over 132 SMs is 4.85 waves.[3] After 4 full waves, 112 tiles are left for 132 SMs. The kernel finishes when the busiest SM does, so 112 SMs run a 5th tile while 20 sit idle.

Because the second group runs right to left, those 112 tiles are its leftmost 7 columns: a clean 16 × 7 tile rectangle, 1,792 columns wide. I cut it into 448-column windows and split each window between two CTAs. Each CTA computes an L-shaped piece, 64×256 for one warpgroup and 64×192 for the other, which is 0.875 of a full tile. That makes 128 pieces for 128 SMs, and the busiest SM does 4.875 tiles of work instead of 5.

0192256448 rows 0–6364–127 CTA 0 · warpgroup 0 · 64×256 CTA 0 · warpgroup 1 · 64×192 CTA 1 · warpgroup 0 · 64×192 CTA 1 · warpgroup 1 · 64×256 CTA 0 loads B columns 0–255 CTA 1 loads B columns 192–447
One 448-column window from the last wave, split between two CTAs. Each CTA still loads full-size A and B boxes, and each warpgroup only uses part of the B box. The 64 columns between the dashed lines get loaded by both CTAs.

I picked the L shape because it changes very little. The producer loads the same A and B boxes it would for a full tile, so the TMA code, the pipeline and the reduction order all stay the same. The cost is that both CTAs load the shared 64 columns of B. In-kernel timestamps put the last wave at 0.884–0.890 of a full tile (0.875 is ideal), which was worth 0.8% overall.

Spreading the epilogue

At the end of every tile, each SM writes 64 KB of FP16 output, and in-kernel timestamps showed the tensor cores idle for about 1.5 µs at each tile boundary. Deferring the store so it overlaps the next tile's first K-steps made things worse. The idle gap shrank, but the next 8 K-steps slowed from about 560 ns to 720 ns.

To find out why, I ran a broken version that skipped the global write. It was 2.5% faster, so the cost was the write itself, and specifically its shape: a 64 KB burst leaving the SM competes with the 48 KB per K-step that TMA is loading into it.

So I spread the store out. Each consumer warpgroup keeps its finished 64×256 FP16 tile in registers (64 per thread, which fits in the 232 budget) and writes it as four 64×64 TMA stores during the next tile, at K-steps 1, 9, 17 and 25. The second warpgroup is offset by 4 K-steps, so the SM never sends more than 8 KB in any 4 K-step window.

before after, WG 0 after, WG 1 64 KB burst, tensor cores idle 8 KB8 KB8 KB8 KB 08162432 … K-step of the next tile (one cell = one K-step, about 600 ns)
Top: the whole 64 KB goes out at the tile boundary while the tensor cores wait. Bottom: each warpgroup writes one 8 KB piece every 8 K-steps, offset by 4, with no idle gap.

In the consumer loop, a store is one call between issuing a K-step's WGMMAs and waiting on them:

# previous tile's fp16 result, as four 64x64 pieces
parts = split_n(out, 4)
for g in gl.static_range(4):
    acc, s = k_issue(...)          # issue this k-step's wgmmas
    if pending:                    # write one 8 kb piece
        store_piece(c_desc, c_bufs, parts[g],
                    row, col + g * 64, g, WG)
    acc = k_retire(acc, 1, NARROW) # <= 1 wgmma group in flight
    mbarrier.arrive(empty.index(prev), count=1)
    acc, it, prev = k_steps(...)   # 7 more k-steps

This was the biggest single gain, 2.2%. The same 64 KB cost 2–3% as a burst and almost nothing when spread out.

Smaller changes

The 64×192 warpgroup in the L-shaped pieces first ran two WGMMAs per 16-wide slice of K, an n128 and an n64, so it read its A slice from shared memory twice. Gluon's layouts only give you power-of-two N, so I issue one wgmma.m64n192k16 through inline PTX with hand-built shared-memory descriptors.[4] That warpgroup's shared-memory reads drop from 40 KB to 32 KB per K-step, worth about 0.2%.

Passing the tile-grid dimensions as constexpr[5] turns the divisions that map a tile index to a row and column into cheap multiply-and-shift sequences. That gave another 0.15–0.45%.

What didn't work

Idea Result What profiling showed
2-CTA clusters with B multicast 0.6–1.6% slower halves L2-to-SM traffic, which wasn't the bottleneck
192×256 tiles, three consumer warpgroups up to 17% slower m64n256k16 needs at least 154 registers per thread, and 13–16 warps leave at most 128
256×160 tiles 10% slower splitting N = 160 into n128 + n32 instructions cost 10% more cycles
L2 persistence and eviction hints 1.3–3.9% slower the L2 flush before each timed call wipes them
Trimming instructions from the main loop 0.6% slower the loop had at most ~1% instruction overhead to begin with

Final results

The final kernel scored 204.7 µs, #1 on the board.[6] Run side by side in the same container, it takes 208.1 µs to cuBLAS's 222.1 µs, about 6% faster.

The main loop, TMA loads feeding WGMMA, got within about 1% of the tensor-core rate early on. Most of the remaining gain came from the work around it: the last partial wave, the output write and the tile order. The epilogue taught me the most. Moving the same bytes in smaller pieces spread over time was worth more than any change to the math, and breaking the kernel on purpose (skipping the write) was the fastest way to find out what a piece of it actually cost.

Notes

  1. This is called warp specialization: one warp loads data and the rest do math. Producer/consumer is the usual naming for it (CUTLASS, FlashAttention-3). ↩︎

  2. tile-rows 0–15tile-rows 16–31 32 × 20 tiles in two groups of 16 tile-rows. Tiles go down a column, then move over one. The bottom group goes right to left, so it starts on the B columns the top group just loaded. ↩︎

  3. A wave is one tile per SM. Time lost to a partial last wave is called wave quantization. ↩︎

  4. A 64-bit value that tells the tensor cores where an operand sits in shared memory and how it's swizzled. ↩︎

  5. constexpr args are compile-time constants in Triton, so the kernel gets compiled for those exact values. ↩︎

  6. Submission 971538, 204.692 µs. Second place was 204.96 µs at the time. ↩︎