Using ATI hardware tessellation in DX9.

Maybe you heard that ATI GPUs starting from 2*** have hardware tessellation unit. Maybe you have seen it working in GPU Mesh Mapper and Render Monkey 1.81. But there isn't any info on how to use it yourself. So with the power of reverse engineering I have finally written my first application that uses hardware tessellation. And I will gladly share the info with you.

To enable tessellator, simply write a few lines of code.
dev->SetRenderState(D3DRS_POINTSIZE, 0x7FA03001);
dev->SetRenderState(D3DRS_MAXTESSELLATIONLEVEL, *(DWORD*)(&floatTesselationFactor)); //from 1.0f to 15.0f

There is however, one issue. If you are changing the primitive type of the geometry, you need to disable tessellation:
dev->SetRenderState(D3DRS_POINTSIZE, 0x7FA03000); //Other values can do the same
dev->SetRenderState(D3DRS_ENABLEADAPTIVETESSELLATION, 0);
And after that, enable it again.

Before you try this out, you also need to change you vertex shader. Now you will get not a single vertex data, but the data of all three vertices of a triangle and barycentric coordinates inside it.

So, if you have something like:
struct VS_INPUT
{
float4 position: POSITION0,
float2 texcoord: TEXCOORD0
};

as your vertex shader input, change it to:

struct VS_INPUT
{
float3 barycentric: BLENDWEIGHT0,

//First vertex
float4 position1: POSITION0,
float2 texcoord1: TEXCOORD0,

//Second vertex
float4 position2: POSITION4,
float2 texcoord2: TEXCOORD4,

//Third vertex
float4 position3: POSITION8,
float2 texcoord3: TEXCOORD8
};

Now, to find parameters for current vertex, use:

float4 position = position1 * barycentric.x + position2 * barycentric.y + position3 * barycentric.z;
float2 texcoord = texcoord1 * barycentric.x + texcoord2 * barycentric.y + texcoord3 * barycentric.z;

Have fun with it!