Fourier Neural Operators Just Got A Turbo Boost: Researchers From Uc Riverside Introduce Turbofno, A Fully Fused Fft-gemm-ifft Kernel Achieving Up To 150% Speedup Over Pytorch

Fourier Neural Operators (FNO) are powerful devices for learning partial differential equation solution operators, but deficiency architecture-aware optimizations, pinch their Fourier furniture executing FFT, filtering, GEMM, zero padding, and iFFT arsenic abstracted stages, resulting successful aggregate kernel launches and excessive world representation traffic. The FFT -> GEMM -> iFFT computational shape has received inadequate attraction regarding GPU kernel fusion and representation layout optimization. Current methods for illustration Quantum ESPRESSO, Octopus, and CP2K make abstracted calls to FFT and BLAS routines. However, they person 3 limitations: partial wave utilization pinch further representation transcript operations, deficiency of autochthonal wave filtering capabilities successful cuFFT, and excessive representation transactions betwixt processing stages.

FNO implements a pipeline that originates pinch a guardant FFT connected input characteristic maps, applies spectral filtering, and reconstructs output done inverse FFT. This process necessitates wave domain truncation and zero-padding steps, which existent frameworks for illustration PyTorch execute arsenic abstracted memory-copy kernels owed to cuFFT’s limitations successful autochthonal input/output trimming support. Leading FFT libraries specified arsenic cuFFT and VkFFT deficiency built-in information truncation capabilities. Traditional 2D FFTs use some 1D-FFT stages on spatial dimensions, but FNO applies spectral weights crossed nan transmission dimension, suggesting an opportunity for decoupling nan FFT stages by keeping nan first 1D FFT on spatial axes while reinterpreting nan 2nd FFT shape on nan hidden dimension.

Researchers from nan University of California, Riverside, CA, USA, person projected TurboFNO, nan first afloat fused FFT-GEMM-iFFT GPU kernel pinch built-in FFT optimizations. The attack originates pinch processing FFT and GEMM kernels from scratch that execute capacity comparable to aliases faster than closed-source state-of-the-art cuBLAS and cuFFT. An FFT version is introduced to efficaciously fuse FFT and GEMM workloads wherever a azygous thread artifact iterates complete nan hidden dimension, aligning pinch nan k-loop successful GEMM. Moreover, 2 shared representation swizzling patterns are designed to execute 100% representation slope utilization erstwhile forwarding FFT output to GEMM and alteration iFFT to retrieve GEMM results straight from shared memory.

TurboFNO integrates optimized implementations of FFT and CGEMM kernels to alteration effective fusion and built-in FFT optimizations. The kernel fusion strategy successful TurboFNO progresses done 3 levels: FFT-GEMM fusion, GEMM-iFFT fusion, and afloat FFT-GEMM-iFFT fusion. Each shape involves aligning nan FFT workflow pinch GEMM, resolving information layout mismatches, and eliminating shared representation slope conflicts. Key techniques see modifying FFT output layout to lucifer GEMM’s input format, applying thread swizzling for conflict-free shared representation access, and integrating inverse FFT arsenic an closing shape of CGEMM to bypass intermediate world representation writes and heighten representation locality.

TurboFNO shows awesome capacity successful some 1D and 2D FNO evaluations. In 1D FNO tests, nan optimized FFT-CGEMM-iFFT workflow achieves up to 100% speedup complete PyTorch, averaging 50% improvement. These gains travel from FFT pruning, which reduces computation by 25%-67.5%. The afloat fused FFT-CGEMM-iFFT kernel delivers up to 150% speedup complete PyTorch and provides an further 10%-20% betterment complete partial fusion strategies. Similarly, successful 2D FNO, nan optimized workflow outperforms PyTorch pinch mean speedups supra 50% and maximum improvements reaching 100%. The 2D afloat fused kernel achieves 50%-105% speedup complete PyTorch without capacity degradation, contempt nan further overhead of aligning FFT workload layout pinch CGEMM dataflow.

In this paper, researchers introduced TurboFNO, nan first afloat fused GPU kernel that integrates FFT, CGEMM, and iFFT for accelerating Fourier Neural Operators. They developed a bid of architecture-aware optimizations to flooded inefficiencies successful accepted FNO implementations, specified arsenic excessive kernel launches and world representation traffic. These see a civilization FFT kernel pinch built-in wave filtering and zero padding, a GEMM-compatible FFT version that mimics k-loop behavior, and shared representation swizzling strategies that amended slope utilization from 25% to 100%. TurboFNO achieves up to 150% speedup and maintains an mean 67% capacity summation crossed each tested configurations.

Here is nan Paper. Also, don’t hide to travel america on Twitter and subordinate our Telegram Channel and LinkedIn Group. Don’t Forget to subordinate our 90k+ ML SubReddit.

🔥 [Register Now] miniCON Virtual Conference connected AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 p.m. PST) + Hands connected Workshop