Neural Denoising in a Shader

Input [1 SPP] ----------------------------- GroundTruth [1024 SPP] ----------------------------- Filtered Result [From 1 SPP]

The Idea

This very cool idea was conceived by Aaron Beis and tackled as a team. We imposed severe constraints on the pipeline in the sense that we wanted maximum results while limiting model size so that we could fit it inside a fragment shader. Also FPS!

Design

We designed a pipeline that uses a denoising filter called WAMNet, short for ‘Wait a Minute Network.’ We coined this name because it took about a minute to learn the filter for a single scene. This filter denoises the input from inside an OpenGL fragment shader. We generated chunks of data with [RGB, normals] as input, and trained a Multi-Layer Perceptron (MLP) with sine activations. The inclusion of normals enforced geometric consistency within the scene and helped preserve high frequencies. After training, we used the model weights to generate the shader code and predict the output color. Here is the report that offers more details. Overall, it was a very fun project to work on !

Results

Here you see a video of the shader denoising in real-time!

Credits

This project served as the spark for all my subsequent decisions in my Masters. These 2 guys, Aaron and Shailesh, have continued to be a constant source of inspiration and still motivate me to explore the depths of the Graphics world. Thanks to them, forward rendering will always hold a special place in my professional life!