With the siggraph 2020 conference coming soon, Facebook researchers published an article today introducing the "neural supersampling" technology.This is an algorithm technology based on artificial intelligence machine learning to improve image clarity, which is not much different from NVIDIA's DLSS "deep learning supersampling" technology. However, the neural supersampling does not need special hardware or software to achieve, and the results are good, basically comparable to the effect of DLSS.
"Close to our research, NVIDIA recently released deep learning supersampling (DLSS), which can process low-resolution images in real-time through neural networks and enlarge sampling.
In this paper, we will introduce a new method to make modern game engine easy to integrate, without special hardware or software, which can be applied to many existing software platforms, acceleration hardware and displays.
We observe that the additional auxiliary information provided by motion vectors plays a key role in neural supersampling. The motion vector determines the geometric correspondence between pixels in a continuous frame. In other words, each motion vector points to sub-pixel locations on which a visible surface point may appear adjacent to the previous frame. These values are usually estimated by computer vision, but such optical flow simulation algorithm is prone to errors. In contrast, the rendering engine can directly produce dense motion vectors, so as to obtain reliable and rich input information, which can be used to render image content for neural supersampling.
Our approach is to maximize image and video quality using innovative spatio-temporal neural network design, combined with additional supporting information, while providing real-time performance ."
In the article, Facebook mentioned that neural supersampling technology may be applied to AR and VR programs to facilitate their oculus platform. However, if the effect is good, it should be applied to other 3D games. Of course, the new technology needs to be tested by practice, and we are looking forward to seeing the actual performance of this neural supersampling.