🎉 CUDA Learn Notes with PyTorch: fp32、fp16/bf16、fp8/int8、flash_attn、sgemm、sgemv、warp/block reduce、dot prod、elementwise、softmax、layernorm、rmsnorm、hist etc.
-
Updated
Sep 21, 2024 - Cuda
🎉 CUDA Learn Notes with PyTorch: fp32、fp16/bf16、fp8/int8、flash_attn、sgemm、sgemv、warp/block reduce、dot prod、elementwise、softmax、layernorm、rmsnorm、hist etc.
This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.
Standard library strided math functions.
Strided array math operations.
Base strided.
Standard library strided array special math functions.
Compute the absolute value.
Standard library special math functions.
Apply a function to each element in an array and assign the result to an element in an output array, iterating from right to left.
Add a description, image, and links to the elementwise topic page so that developers can more easily learn about it.
To associate your repository with the elementwise topic, visit your repo's landing page and select "manage topics."