Inside Direct3D: Stencil Buffers

Contents

Creating a Stencil Buffer

Configuring the Stenciling State

Creating Effects

One aspect of advanced rendering we haven't discussed yet is stenciling, a technique that can be useful for developing commercial applications. If you want your 3D applications to stand apart from the crowd, you'd be wise to combine stenciling with the texturing techniques you learned about in earlier chapters. This chapter will detail how to use stenciling and show you the different types of effects you can generate with it.

Many 3D games and simulations on the market use cinema-quality special effects to add to their dramatic impact. You can use stencil buffers to create effects such as composites, decals, dissolves, fades, outlines, silhouettes, swipes, and shadows. Stencil buffers determine whether the pixels in an image are drawn. To perform this function, stencil buffers let you enable or disable drawing to the render-target surface on a pixel-by-pixel basis. This means your software can "mask" portions of the rendered image so that they aren't displayed.

When the stenciling feature is enabled, Microsoft Direct3D performs a stencil test for each pixel that it plans to write to the render-target surface. The stencil test uses a stencil reference value, a stencil mask, a comparison function, and a pixel value from the stencil buffer that corresponds to the current pixel in the target surface. Here are the specific steps used in this test:

  1. Perform a bitwise AND operation of the stencil reference value with the stencil mask.
  2. Perform a bitwise AND operation on the stencil-buffer value for the current pixel with the stencil mask.
  3. Compare the results of Step 1 and Step 2 by using the comparison function.

By controlling the comparison function, the stencil mask, the stencil reference value, and the action taken when the stencil test passes or fails, you can control how the stencil buffer works. As long as the test succeeds, the current pixel will be written to the target. The default comparison behavior (the value that the D3DCMPFUNC enumerated type defines for D3DCMP_ALWAYS) is to write the pixel without considering the contents of the stencil buffer. You can change the comparison function to any function you want by setting the value of the D3DRENDERSTATE_STENCILFUNC render state and passing one of the members of the D3DCMPFUNC enumerated type.

Creating a Stencil Buffer

Before creating a stencil buffer, you need to determine what stenciling capabilities the target system supports. To do this, call the IDirect3DDevice7::GetCaps method. The dwStencilCaps flags specify the stencil-buffer operations that the device supports. The reported flags are valid for all three stencil-buffer operation render states: D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILPASS, and D3DRENDERSTATE_STENCILZFAIL. Direct3D defines the following flags for dwStencilCaps:

Direct3D embeds the stencil-buffer information with the depth-buffer data. To determine what formats of depth buffers and stencil buffers the target system's hardware supports, call the IDirect3D7::EnumZBufferFormats method, which has the following declaration:

HRESULT IDirect3D7::EnumZBufferFormats (
    REFCLSID riidDevice,
    LPD3DENUMPIXELFORMATSCALLBACK lpEnumCallback,
    LPVOID lpContext
);

Parameter
Description
riidDevice A reference to a globally unique identifier (GUID) for the device whose depth-buffer formats you want enumerated
IpEnumCallback The address of a D3DEnumPixelFormatsCallback function you want called for each supported depth-buffer format
IpContext Application-defined data that is passed to the callback function

If the method succeeds, it returns the value D3D_OK. If it fails, the method returns one of these four values:

The code in listing 1 determines what stencil buffer formats are available and what operations are supported and then creates a stencil buffer. As you can see, this code notes whether the stencil buffer supports more than 1-bit -- some stenciling techniques must be handled differently if only a 1-bit stencil buffer is available.

Clearing a Stencil Buffer

The IDirect3DDevice7 interface includes the Clear method, which you can use to simultaneously clear the render target's color buffer, depth buffer, and stencil buffer. Here's the declaration for the IDirect3DDevice7::Clear method:

    HRESULT IDirect3DDevice7::Clear(
       DWORD dwCount,
       LPD3DRECT lpRects,
       DWORD dwFlags,
       D3DVALUE dvZ,
       DWORD dwStencil
    );

Parameter
Description
dwCount The number of rectangles in the array at lpRects.
IpRects
An array of D3DRECT structures defining the rectangles to be cleared. You can set a rectangle to the dimensions of the render-target surface to clear the entire surface. Each of these rectangles uses screen coordinates that correspond to points on the render-target surface. The coordinates are clipped to the bounds of the viewport rectangle.
dwFlags Flags indicating which surfaces should be cleared. This parameter can be any combination of the following flags, but at least one flag must be used:
  D3DCLEAR_TARGET Clear the render-target surface to the color in the dwColor parameter. D3DCLEAR_STENCIL Clear the stencil buffer to the value in the dwStencil parameter.
  D3DCLEAR_ZBUFFER Clear the depth buffer to the value in the dvZ parameter.
dwColor D3DCLEAR_ZBUFFER Clear the depth buffer to the value in the dvZ parameter..
dvZ A 32-bit RGBA color value to which the render-target surface will be cleared.
dwStencil The new z value that this method stores in the depth buffer. This parameter can range from 0.0 to 1.0, inclusive. The value of 0.0 represents the nearest distance to the viewer, and 1.0 represents the farthest distance.
  The integer value to store in each stencil-buffer entry. This parameter can range from 0 to 2n-1 inclusive, in which n is the bit depth of the stencil buffer.

The IDirect3DDevice7::Clear method still accepts the older D3DCLEAR_TARGET flag, which clears the render target using an RGBA color you provide in the dwColor parameter. This method also still accepts the D3DCLEAR_ZBUFFER flag, which clears the depth buffer to a depth you specify in dvZ (in which 0.0 is the closest distance and 1.0 is the farthest). DirectX 6 introduced the D3DCLEAR_STENCIL flag, which you can use to reset the stencil bits to the value you specify in the dwStencil parameter. This value can be an integer ranging from 0 to 2n-1, in which n is the bit depth of the stencil buffer.

________________________________________________________

Configuring the Stenciling State