Anti-Aliasing / Multisampling
Table of Contents
Multi-Sample Anti-Aliasing (MSAA) (Hardware Anti-Aliasing)
This is traditional anti-aliasing that is done in hardware during the rendering of the scene and the quality can be configured in the Video Device Configuration.
Depending on the hardware the quality settings can differ in its resolution.
Hardware Vendors have been tweaking with variations of this idea, trying to find more efficient representations for the edges than increasing the size of the rendertargets, but the problems remain the same.
Going from 3D to 2D with respect to MSAA means converting these enlarged rendertargets into ordinary rendertargets with one pixel per real pixel. This is called the multisample resolve.
Note that Ventuz renders with premultiplied alpha. When doing so, color and alpha are associated together and may not be divided. When resolving 8 bit rendertargets, Direct3D will use a different method for color than for alpha, and that leads to artifacts at the border to a transparent background.
To get rid of the premultiplied alpha artifacts on non-HDR layers you can use the FixSeams option on the Layer 3D Root Node on layers which are alpha blended onto others. When active, the renderer will not use the Hardware MSAA of the Graphics Card but instead a specific shader in Ventuz. This is slower and thus may result in performance problems but the edges of the alpha channels are anti aliased correctly with the quality of MSAA. Keep in mind that this is even slower than the already slow MSAA; only use it when it is really necessary and when using the faster FXAA will not solve the problem on the layer or does not result in the desired quality.
In the NVidia graphics settings, you should set "Antialiasing - Gamma correction" to "Off". This fixes the problem with premultiplied alpha, as the gamma correction is exactly what causes the problem.
Performance
As you might imagine, Multi-Sampling costs render performance, especially in complex scenes.
Therefore, in high-resolution and Cluster setups, Multisampling should be lowered or disabled to increase the rendering performance.
You can check the amount of time the multisample resolve takes in the Performance Statistics.
Fast Approximate Anti-Aliasing (FXAA) (Software Anti-Aliasing)
To combat the disadvantages of MSAA, game developers have created shader based solutions such as Fast Approximate Anti-Aliasing (FXAA).
FXAA is not related to the hardware and therefore can be configured on the 3D Layer Root of any used 3D Layer rather than in the Machine Configuration. Thus you can change the used Anit-Aliasing Method from Layer to Layer.
The NVidia and AMD control panels allow you to enable FXAA or similar algorithms manually, but that is not optimal as the graphics driver does not have all the information it needs to do this properly.
The FXAA Algorithm will search for edges in the readily rendered image and blur the image there a little bit. This may take up a lot less performance than MSAA but will make the result look blurry. There are several parameter configurations in the FXAA Property Group of the 3D Layer Root to improve performance and quality.
Alpha turns on or off the processing of the alpha channel that we integrated into the algorithm. Turning this off may increase quality in Scenes that do not have an Alpha Channel in the resulting texture. Quality changes the step count done in the shader. The more steps, the higher the quality and performance impact. Medium Dithering reduces the quality but enhances the performance, so Less Dither will result in a better image. The Best setting will not use dithering at all.
EdgeThreshold changes the minimum amount of the local contrast to trigger the processing of the algorithm. The higher this threshold, the fewer pixels will be processed and thus the faster the algorithm but the lower the quality. EdgeThresholdMin defines the threshold of the minimum luma to apply the algorithm. This will trim the algorithm from processing darker areas in the image. Again, the higher this value the faster the algorithm. Subpixel will change the amount of sub-pixel anti-aliasing. The higher this value the softer the result. This will not have an impact on the performance, only on the sharpness of the resulting image.
Lastly, PerformanceTest will change each pixel on your Layer according to how much performance the FXAA used up for it. An unchanged Pixel did not trigger the algorithm at all. Green Pixels triggered the FXAA but did not need a lot of steps for the aliasing removal and red pixels needed a lot of steps. With the PerformanceTest turned on you can easily analyze how the FXAA impacts your performance and the quality of your image.