Deferred Shading, AA, alpha-blending.

While playing GTA IV I couldn't help but to notice the use of Alpha to Coverage. For the last few years I've seen this feature used mainly to create smooth edges for alpha-tested polygons when FSAA is enabled¹. In the game, however, it is used for in and out object fading, and for a part of transparent objects as well. As far as I know, their engine uses deferred shading rendering, so this method looks rather interesting and gave me some thoughts about implementing those features, mentioned in the topic, that don't live quite well with deferred rendering method.

So how did I do this? I've seen some nice demos from Humus that shows transparent objects in deferred rendering using method similar to depth peeling. To implement something like this, we need to be able to access every single sample in our multisampled G-buffer, so I've opened DX help along with tutorials 01 and 02, and started to write a demo under DX10 where such features are available.

My first idea was that during the G-buffer fill-up, for transparent objects we will explicitly set SampleMask with the number of active samples that relatively to the overall sample count, nearly corresponds to the alpha value (so, with AAx8 it will be possible to have 8 levels of transparency). Straight away, I've stumbled across some problems with transparent objects being placed in multiple layers. So I had to manually hard-code sample masks with randomly distributed active samples to get the result look right.

In the shader where lightning is computed, for every sample, I calculated two-sided diffuse lightning (using alpha value to linearly interpolate between sides), specular and simplified volumetric lightning. The weight of every sample is 1/sample count. At this point, I have realized that I can set different weights for different samples and after that it is possible to write every transparent surface information into a single sample in G-buffer. I used a simple formula to convert pixel alpha to sample weight, because they aren't equal parameters.
This new method didn't solve the order-independent transparency problem, but made things much easier, now it is possible to set AAx4 while still able to render 4 layers of transparent geometry and the alpha value isn't restricted to 8 levels any more.

As the result I have a demo along with sources that could be run if you have video card with DX10 support and Vista\Windows7 operating system:
Download, 335kb.

Some screenshots:




It's just a demo without pretending to be used in real-world applications (at least for today). There is a lot memory usage for the multisampled G-buffer, and a lot of bandwidth is required.
If the demo speed is satisfying, you can set a LIGHTS constant in "Shaders\LightPS.txt" file up to 32.

¹ AntiAliasing with Transparency
² Stencil-Routed K-Buffer
³ Volumetric lighting demo