Optimal Block Sizes for High-Performance Memory Access

In high-performance computing, the layout and access pattern of memory are crucial. While linear, contiguous memory is often preferred, the benefits diminish with larger blocks. My experiments aimed to determine the optimal block size for peak performance. I found that 1 MB blocks suffice for most workloads, 128 kB blocks are adequate with at least ~1 cycle per byte, and 4 kB blocks work well above ~10 cycles per byte.

To test this, I used a Ryzen 9 7950X3D and designed an experiment to isolate and control the effects of memory hierarchy. The 'processing kernel' used a span of floats and returned a uint64_t result to prevent optimization that could skew results. The scalar_stats kernel, for example, computes running stats over blocks, achieving ~7 GB/s, or ~0.75 cycles per byte.

The experimental setup involved clobbering the cache to ensure no useful data remained before each run. This was done by folding a 256 MB block of memory into a seed, ensuring the cache was cleared. The scalar_stats kernel was tested across block sizes from 32 bytes to 2 MB, showing that beyond 128 kB, increasing block size offers little benefit.

In a repeated scenario, where data is hit multiple times, smaller working sets can be cached, leading to earlier peak performance. For different workloads, such as the SIMD kernel simd_sum and the heavy_sin kernel, the block size requirements varied. The simd_sum needed 1 MB for peak performance, while heavy_sin performed well with 4 kB blocks.

Combining all experiments, the results showed that block sizes larger than 1 MB are often unnecessary. Most processing tasks take more than 1 cycle per byte, making 128 kB blocks sufficient. The experiments highlighted that while contiguous blocks are beneficial, the required block size for peak performance is smaller than expected.

The benchmarks, available on GitHub, invite further testing on different systems to see if these guidelines generalize. Future experiments could explore multi-threading, striding impacts, and combined read-write workloads. Despite some noise in measurements, the benchmarks provide valuable insights into memory access optimization.