12.9 - Stencils¶

The Big Idea¶

A stencil buffer provides fine-grain control over which pixels are processed by the graphics pipeline after they have been processed by a fragment shader. Stencil buffer functionality is hard-coded into the graphics pipeline and is controlled by a programmer through stencil buffer parameters. Understanding the parameter settings is non-trivial, so they will be explained in the simplest possible terms and then details will be added to provide the “total picture.”

There are two fundamental ideas:

• Using a “stencil test” to determine if a fragment should be allowed to modify the color buffer.
• Setting the stencil buffer values to create a “stencil mask.”

Before discussing these two issues, let’s discuss the structure and nature of a stencil buffer.

The Stencil Buffer¶

A stencil buffer must be created when a WebGL context is created for a canvas. It is a 2D array of unsigned bytes that must have the same dimensions as the draw buffer’s color buffer and depth buffer. Here is an example of creating a WebGLRenderingContext that contains a stencil buffer:

canvas = document.getElementById(canvas_id);
context = canvas.getContext('webgl', {"stencil" : true} );

A true or false value can be represented using a single bit. Typically 0 means false and 1 means true. Since a stencil buffer stores an unsigned byte for each pixel, it can potentially represent eight different “stencils” at any given time. WebGL allows a programmer to specify a “mask” that determines which bits are tested when a stencil test is performed. When a value from a stencil buffer is retrieved, a bit-wise AND operation is always performed with a “mask” before it is used. A “mask” is typically specified in hexadecimal notation by starting the value with 0x. The following table shows the eight masks that can be used to differentiate the eight bits in an unsigned byte:

If each stencil buffer value is treated as a single true/false value, the “mask” can be set to 0xFF (all ones). Other division of the bits are possible. For example, two stencils could be created, where the high order four bits (using a mask of 0xF0) represent one stencil and the low order fours bits (using a mask of 0x0F) represents the other stencil. You can conceptualize the stencil buffer as a single “mask”, as eight separate “masks”, or any number in-between.

The Stencil Test¶

When the stencil test is enabled, pixels are allowed to modify the color buffer and the depth buffer only if they pass a “stencil test.” The following pseudocode describes the internal logic of the graphics pipeline and demonstrates that the stencil test happens before the depth test (assuming that the “depth test” has been enabled by gl.enable(gl.DEPTH_TEST)).

void clearBuffers() {
  for (x = 0; x < image_width; x++) {
    for (y = 0; y < image_height; y++) {
      depth_buffer[x][y] = maximum_z_value;
      color_buffer[x][y] = background_color;
     }
  }
}

void renderPixel(x, y, z, color) {
  if (z < depth_buffer[x][y]) {
    depth_buffer[x][y] = z;
    color_buffer[x][y] = color;
  }
}

void clearBuffers() {
  for (x = 0; x < image_width; x++) {
    for (y = 0; y < image_height; y++) {
      depth_buffer[x][y]   = maximum_z_value;
      color_buffer[x][y]   = background_color;
      stencil_buffer[x][y] = stencil_value;
    }
  }
}

void renderPixel(x, y, z, color) {
  if (passes_stencil_test(x, y)) {
    if (z < depth_buffer[x][y]) {
      depth_buffer[x][y] = z;
      color_buffer[x][y] = color;
    }
  }
}

The “stencil test” is a comparison between a “reference value” and the current value stored in the stencil buffer at a pixel’s (x,y) location. A programmer does not implement this logic, but rather specifies the type of comparison, the reference value, and a mask using a call to gl.stencilFunc( enum func, int ref, uint mask ). The following pseudocode demonstrates the functionality.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

// -------------------------------------------------------------
// STENCIL TEST - does a pixel pass the stencil test?
int stencil_func = ALWAYS;  // DEFAULT
int reference    = 0;       // DEFAULT
int bit_mask     = 0xFF;    // DEFAULT - all bits are ones

// Sets the STENCIL TEST parameters:
void gl.stencilFunc( enum func, int ref, uint mask ) {
  stencil_func = func;
  reference    = ref;
  bit_mask     = mask;
}

// Performs the STENCIL TEST:
boolean passes_stencil_test(x, y) {
  condition = TRUE;
  if (STENCIL_TEST_IS_ENABLED) {

    stencil_value   = stencil_buffer[x][y] & bit_mask;  // bit-wise AND
    reference_value = reference            & bit_mask;  // bit-wise AND

    switch (stencil_func) {
      case NEVER:    condition = false;
      case ALWAYS:   condition = true;  // DEFAULT
      case LESS:     condition = (reference_value <  stencil_value);
      case LEQUAL:   condition = (reference_value <= stencil_value);
      case EQUAL:    condition = (reference_value == stencil_value);
      case GREATER:  condition = (reference_value >  stencil_value);
      case GEQUAL:   condition = (reference_value >= stencil_value);
      case NOTEQUAL: condition = (reference_value != stencil_value);
    }
  }
  return condition;
}

For example, gl.stencilFunc( gl.EQUAL, 2, 0x02 ) would configure the stencil test to be true for a pixel at (x,y), if the value at stencil_buffer[x][y] has its low order 2nd bit set to 1. Note that performing a bit-wise AND operation using a mask of 0x02 will produce either a value of 2 or 0.

For another example, gl.stencilFunc( gl.GREATER, 15, 0xF0 ) would configure the stencil test to be true for a pixel at (x,y), if the value at stencil_buffer[x][y] has any of its four high order bits set to one. Note that performing a bit-wise AND operation using a mask of 0xF0 will produce one of the following 16 values: 0, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240.

As the above pseudocode indicates, the stencil test can be configured to perform one of eight possible comparisons. In addition, it makes sure that only certain bits are used in the comparison.

Creating a Stencil¶

Each pixel in a stencil buffer is assigned a value by clearing the buffer with a specific value and then rendering a scene. As with the “stencil test,” the work of defining a stencil is hard-coded into the graphics pipeline. The programmer’s responsibilities is to assign appropriate parameters to the stencil operation parameters before a rendering is performed.

Please study the following pseudocode which describes the internal workings of the graphics pipeline and shows when a stencil operation is performed and the data it uses. Please notice the following:

• To simplify the pseudocode, tests to determine if the gl.STENCIL_TEST is enabled have been left out. However, stencil operations are only performed when the gl.STENCIL_TEST has been enabled.
• The stencil buffer is updated once by each invocation of a fragment shader. Stencil operation parameters define a separate operation for fragments that failed the “stencil test,” or that passed the “stencil test” but either passed or failed the “depth test.”
• Updating the stencil buffer is performed after the stencil test and the depth test have been completed. This takes some contemplation! A stencil buffer’s value is used, and then updated to possibly a different value! In typical usage this rarely happens. The “stencil test” parameters and the “stencil operation” parameters are typically configured to perform one or the other, but not both at the same time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

void renderPixel(x, y, z, color) {
  if (passes_stencil_test(x,y)) {
    if (z < z_buffer[x][y]) {
      depth_buffer[x][y] = z;
      color_buffer[x][y] = color;
      status = DEPTH_TEST_PASSED;
    } else {
      status = DEPTH_TEST_FAILED;
    }
  } else {
    status = STENCIL_TEST_FAILED;
  }
  update_stencil_buffer(x, y, status);
}

Updating the stencil buffer is described by the following pseudocode. Please study the pseudocode carefully. (Note that the gl.REPLACE operation uses the reference value set by gl.stencilFunc( func, reference, mask ). In addition, the mask limits the bits that can be modified in the stencil buffer’s value.

// -------------------------------------------------------------
// STENCIL OPERATION - sets a pixel's value in a stencil buffer.
int stencil_test_failed_func = KEEP;  // DEFAULT
int depth_test_failed_func   = KEEP;  // DEFAULT
int depth_test_passed_func   = KEEP;  // DEFAULT

int MAX_VALUE = 255; // for an 8-bit stencil buffer
int MIN_VALUE = 0;

void gl.stencilOp( enum sfail, enum dpfail, enum dppass ) {
  stencil_test_failed_func = sfail;
  depth_test_failed_func   = dpfail;
  depth_test_passed_func   = dppass;
}

void update_stencil_buffer(x, y, status) {
  if (STENCIL_TEST_IS_ENABLED) {

    switch (status) {
      case DEPTH_TEST_PASSED:   stencil_func = depth_test_passed_func;
      case DEPTH_TEST_FAILED:   stencil_func = depth_test_failed_func;
      case STENCIL_TEST_FAILED: stencil_func = stencil_test_failed_func;
    }

    stencil_value   = stencil_buffer[x][y] & bit_mask;  // bit-wise AND
    reference_value = reference            & bit_mask;  // bit-wise AND

    switch (stencil_func) {
      case KEEP:      value = stencil_value;
      case ZERO:      value = 0;
      case REPLACE:   value = reference_value;
      case INCR:      value = stencil_value + 1;
                      if (value > MAX_VALUE) value = MAX_VALUE;
      case DECR:      value = stencil_value - 1;
                      if (value < MIN_VALUE) value = MIN_VALUE;
      case INVERT:    value = ~ stencil_value;
      case INCR_WRAP: value = stencil_value + 1;
                      if (value > MAX_VALUE) value = MIN_VALUE;
      case DECR_WRAP: value = stencil_value - 1;
                      if (value < MIN_VALUE) value = MAX_VALUE;
    }
    stencil_buffer[x][y] = (value & bit_mask) |
                           (stencil_buffer[x][y] & ~bit_mask);
  }
}

// -------------------------------------------------------------
// STENCIL OPERATION - sets a pixel's value in a stencil buffer.
int stencil_test_failed_func = KEEP;       // DEFAULT
int depth_test_failed_func   = KEEP;       // DEFAULT
int depth_test_passed_func   = KEEP;       // DEFAULT

int back_stencil_test_failed_func = KEEP;  // DEFAULT
int back_depth_test_failed_func   = KEEP;  // DEFAULT
int back_depth_test_passed_func   = KEEP;  // DEFAULT

int MAX_VALUE = 255; // for an 8-bit stencil buffer
int MIN_VALUE = 0;

void gl.stencilOp( enum sfail, enum dpfail, enum dppass ) {
  stencil_test_failed_func = sfail;
  depth_test_failed_func   = dpfail;
  depth_test_passed_func   = dppass;
}

void gl.stencilOpSeparate( enum face, enum sfail, enum dpfail, enum dppass ) {
  if (face == gl.FRONT || face == gl.FRONT_AND_BACK) {
    stencil_test_failed_func = sfail;
    depth_test_failed_func   = dpfail;
    depth_test_passed_func   = dppass;
  }

  if (face == gl.BACK || face == gl.FRONT_AND_BACK) {
    back_stencil_test_failed_func = sfail;
    back_depth_test_failed_func   = dpfail;
    back_depth_test_passed_func   = dppass;
  }
}

void update_stencil_buffer(x, y, status, gl_FrontFacing) {
  if (STENCIL_TEST_IS_ENABLED) {

    if (gl_FrontFacing) {
      switch (status) {
        case DEPTH_TEST_PASSED:   stencil_func = depth_test_passed_func;
        case DEPTH_TEST_FAILED:   stencil_func = depth_test_failed_func;
        case STENCIL_TEST_FAILED: stencil_func = stencil_test_failed_func;
      }
    } else { // ! gl_FrontFacing --> back-facing
      switch (status) {
        case DEPTH_TEST_PASSED:   stencil_func = back_depth_test_passed_func;
        case DEPTH_TEST_FAILED:   stencil_func = back_depth_test_failed_func;
        case STENCIL_TEST_FAILED: stencil_func = back_stencil_test_failed_func;
      }
    }

    stencil_value   = stencil_buffer[x][y] & bit_mask;  // bit-wise AND
    reference_value = reference            & bit_mask;  // bit-wise AND

    switch (stencil_func) {
      case KEEP:      value = stencil_value;
      case ZERO:      value = 0;
      case REPLACE:   value = reference_value;
      case INCR:      value = stencil_value + 1;
                      if (value > MAX_VALUE) value = MAX_VALUE;
      case DECR:      value = stencil_value - 1;
                      if (value < MIN_VALUE) value = MIN_VALUE;
      case INVERT:    value = ~ stencil_value;
      case INCR_WRAP: value = stencil_value + 1;
                      if (value > MAX_VALUE) value = MIN_VALUE;
      case DECR_WRAP: value = stencil_value - 1;
                      if (value < MIN_VALUE) value = MAX_VALUE;
    }
    stencil_buffer[x][y] = (value & bit_mask) |
                           (stencil_buffer[x][y] & ~bit_mask);
  }
}

Multi-pass Rendering¶

When using a stencil buffer, a rendering is typically performed by a series of “rendering passes.” The first rendering pass creates a desired stencil buffer while follow-on renderings use the stencil buffer to control which pixels in a color buffer are modified. WebGL provides fine-grain control of the draw buffers to allow a scene to be “rendered” but only change specific buffers. The gl.colorMask(bool red, bool green, bool blue, bool alpha) function determines whether individual components of a color buffer can be updated. The gl.depthMask(bool flag) function determines whether the depth buffer can be modified. And the stencilMask(unit mask) function determines which bits in a stencil buffer value can be modified.

Our final pseudocode to describe the internal logic of the graphics pipeline demonstrates this fine-grain control.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

boolean write_red   = TRUE;       // Default
boolean write_green = TRUE;       // Default
boolean write_blue  = TRUE;       // Default
boolean write_alpha = TRUE;       // Default
boolean write_depth = TRUE;       // Default
unsigned_int stencil_mask = 0xFF; // Default

void gl.colorMask(bool red, bool green, bool blue, bool alpha) {
  write_red   = red;
  write_green = green;
  write_blue  = blue;
  write_alpha = alpha;
}

void gl.depthMask(bool flag) {
  write_depth = flag;
}

void stencilMask(unit mask) {
  stencil_mask = mask;
}

void renderPixel(x, y, z, color) {
  if (passes_stencil_test(x,y)) {
    if (z < z_buffer[x][y]) {
      if (write_depth) depth_buffer[x][y] = z;
      if (write_red)   color_buffer[x][y].r = color.r;
      if (write_green) color_buffer[x][y].g = color.g;
      if (write_blue)  color_buffer[x][y].b = color.b;
      if (write_alpha) color_buffer[x][y].a = color.a;
      status = DEPTH_TEST_PASSED;
    } else {
      status = DEPTH_TEST_FAILED;
    }
  } else {
    status = STENCIL_TEST_FAILED;
  }
  update_stencil_buffer(x, y, status, stencil_mask);
}

When a stencil buffer value is modified, which bits are changed is controlled by the stencil_mask. Pseudocode that simulates this might look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

void update_stencil_buffer(x, y, status, gl_FrontFacing) {
  if (STENCIL_TEST_IS_ENABLED) {
    ...
    new_value = (value & bit_mask) | (stencil_buffer[x][y] & ~bit_mask);

    if (stencil_mask.bit[0] == 1) stencil_buffer[x][y].bit[0] = new_value.bit[0];
    if (stencil_mask.bit[1] == 1) stencil_buffer[x][y].bit[1] = new_value.bit[1];
    if (stencil_mask.bit[2] == 1) stencil_buffer[x][y].bit[2] = new_value.bit[2];
    if (stencil_mask.bit[3] == 1) stencil_buffer[x][y].bit[3] = new_value.bit[3];
    if (stencil_mask.bit[4] == 1) stencil_buffer[x][y].bit[4] = new_value.bit[4];
    if (stencil_mask.bit[5] == 1) stencil_buffer[x][y].bit[5] = new_value.bit[5];
    if (stencil_mask.bit[6] == 1) stencil_buffer[x][y].bit[6] = new_value.bit[6];
    if (stencil_mask.bit[7] == 1) stencil_buffer[x][y].bit[7] = new_value.bit[7];
  }
}

Using a Stencil Buffer¶

The following two WebGL programs provide examples of tasks that use a stencil buffer.

Summary¶

In summary, the stencil buffer can be used to control which pixels can possibly have their color modified. Using the stencil buffer typically requires multiple renderings. One rendering will set the stencil; another rendering will update the color buffer based on the stencil.

To create a stencil, make the “stencil test” always true, enable writing to the stencil buffer, and set how the stencil value is updated based on the status of the stencil and depth tests.

// Rendering pass to create a stencil.
gl.stencilFunc(gl.ALWAYS, reference_value, mask);
gl.stencilMask(0xFF);  // Stencil buffer can be changed
gl.stencilOp(stencil_failed_func, depth_test_failed_func, depth_test_passed_func);

To use a stencil, set an appropriate “stencil test”, disable writing to the stencil buffer:

// Rendering pass to use a stencil.
gl.stencilFunc(compare_func, reference_value, mask);
gl.stencilMask(0x00);  // Stencil buffer can't be changed

The stencil buffer has many possible uses, as you will discover as you investigate more advanced rendering techniques.

Glossary¶

stencil

An image or pattern that allows some areas of a surface to receive color while preventing other areas from receiving color. It is also referred to as a “stencil mask” because a facial “mask” covers parts of a face while allowing other parts to be visible.

color buffer

A 2D array of color values. Each color value is typically a RGBA color.

depth buffer

A 2D array of “distance from the camera” values. The dimensions of the array must be identical to the dimensions of its associated color buffer.

stencil buffer

A 2D array of “mask” values. The dimensions of the array must be identical to the dimensions of its associated color buffer.

stencil test

Logic that produces a true or false result. If true, a fragment is allowed further processing by the graphics pipeline. If false, the fragment is discarded from the graphics pipeline.

stencil operation

Logic that sets the value of a single pixel in a stencil buffer.

bit-mask

An integer number with specific bits set to one. Performing a bit-wise logical AND operation between a number, n, and a bit-mask guarantees that all bits in n where there is a 0 in the bit-mask there is a 0 in the result.

Self Assessment¶