> For the complete documentation index, see [llms.txt](https://blog.ruipan.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blog.ruipan.xyz/earlier-readings-and-notes/cs759-hpc-course-notes/lecture-14-tiling-as-a-programing-pattern-in-cuda.-example-vector-reduction-in-cuda..md).

# Lecture 14: CUDA Case Studies. (1) 1D Stencil Operation. (2) Vector Reduction in CUDA.

## Lecture Summary

* Last time
  * Atomic operations
  * Things that shape the speed of execution of a kernel
    * The concept of "occupancy" and what impacts it (how many threads per block, how many registers/thread, how much ShMem/block)
  * Rules of thumb, for good execution speed in GPU computing
  * The nvcc toolchain, and how code is sent to host or gpu compilers
* Today
  * Case studies: parallel reduction on the GPU & 1D convolution
  * Looking beyond today: some more GPU computing feature, but looking for a while into optimization features

![Application optimization process](/files/-MUaGzNUqo5nvkZ8652d)

## 1D Stencil Operation

![What the algorithm does](/files/-MUaJHU_DXQVJAav8PI6)

![Serial implementation](/files/-MUaF6vRP6ESWAOteYoK)

![Parallel implementation](/files/-MUaFBvQe_ao9L1H6S1U)

![nvprof pointed out spaces for optimizations](/files/-MUaFhSgzV3a-743xa61)

![Use pinned memory (pinned memory cannot be paged out by the OS)](/files/-MUaFmXuI4iYW3mtJnTM)

![Data partitioning example (overlapping compute & memory)](/files/-MUaHGbHf0eZozOEa_6H)

![Performance improvements](/files/-MUaHVUUxxYU36hw0EQF)

![Optimization summary](/files/-MUaHl8Jiw7XsP_gAIWm)

## Vector Reduction in CUDA

![What the algorithm does (summing all entries in an array)](/files/-MUaJQQHlwV7DAtdvMzz)

Problem: Ideally we want to synchronize across all thread blocks, but CUDA does not have global synchronization. Our workaround is to decompose into multiple kernels.

* Optimization goal: Reaching GPU peak performance
  * Choosing the right metric
    * GFLOP/s: for compute-bound kernels
    * Bandwidth: for memory-bound kernels
* Reductions have low arithmetic intensity (1 flop/2 elements loaded), so we should go for peak bandwidth

![Interleaved addressing: highly divergent warps are inefficient, and % operator is very slow](/files/-MUaZQUej2Ywkez82zyK)

![Change which thread works on what. New problem: shared memory bank conflicts](/files/-MUaZafkMRUMDusaJMYU)

![Sequential addressing](/files/-MUa_4Wm9gkhVOKy1Uit)

* Kernel 4: Replace single load w/ two loads and first add of the reduction
* Kernel 5: Loop unrolling (unroll last warp)
* Kernel 6: Completely unrolling (using templates)
* Kernel 7: Multiple elements per thread