Module 1: Introduction to GPU

• Why GPU exists?
• The Kind of work GPU is made for.
• GPU is very different from a CPU
• Inside a Streaming Multiprocessor (SM)
• Threads and warps
• CUDA
• Matrix Multiplication
• Compute-Bound vs Memory-Bound

Module 2: GPU Terminologies

• VRAM
• Memory Bandwidth
• PCIe and NVLink
• Tensor Cores
• SMs, Threads & Warps
• Floating Point Numbers & FP32
• FP16 & BF16
• FP8, INT8 & INT4
• FLOPS & TFLOPS
• Why Precision Matters
• Model Parameters & Model Size
• Tokens & Context Window
• KV Cache
• Batch Size, Throughput & Latency
• GPU Utilization, Occupancy & MFU
• Training
• Inference
• Quantization & Model Formats

Module 3: Running an LLM on a GPU

• The Whole Path
• Loading a Model & the VRAM Budget
• What Actually Moves Through the Model?
• Attention: How Tokens Use Other Tokens
• The KV Cache
• Reducing the KV Cache
• Decode
• Prefill vs Decode
• Batching
• Continuous Batching
• PagedAttention
• Beyond Batching
• Multi-GPU Inference
• Putting It All Together

icon:gpu

skillcat-systems