Sub-Pixel Anti-Aliasing Unmixing: How Vectra2 Eliminates Stair-Stepping in Bitmap Vectorization

The Anti-Aliasing Conundrum

When a smooth circular curve or diagonal line is rendered onto a discrete grid of square pixels, the Nyquist-Shannon sampling theorem dictates that spatial frequencies higher than half the sampling rate fold into aliasing artifacts—visible "staircase" jaggedness.

To fool the human eye into perceiving a smooth edge, raster renderers place anti-aliased transitional pixels along the boundary. These pixels are semi-transparent blends between the foreground shape color and the background canvas.

1. The Fatal Flaw of Binary Thresholding

Traditional autotrace algorithms (such as Potrace) convert images into pure binary 1-bit black-and-white silhouettes using a luminance threshold:

This binary decision discards all the sub-pixel geometric information encoded in the anti-aliased edge:

• A pixel that was 48% covered by the shape is abruptly rounded to 0 (completely outside).
• A pixel that was 52% covered is abruptly rounded to 1 (completely inside).

The result is a jagged boundary that zigzags back and forth across pixel coordinates, distorting circles into lumpy octagons and rounding off acute corners.

2. The Vectra2 Linear Mixture Model

Rather than discarding intermediate pixels, Vectra2 treats every border pixel as an optical mixture of two dominant palette anchors $C_1$ and $C_2$ in perceptual Oklab color space:

By projecting the observed pixel onto the vector connecting $C_1$ and $C_2$ in Oklab space, we calculate the exact geometric area fraction $\alpha$:

Knowing $\alpha$ for each pixel in a $2 \times 2$ neighborhood allows the solver to find the exact sub-pixel zero-crossing line with mathematical precision down to $\pm 0.05$ pixels.

3. Preserving Acute Corners Without Filleting

A common defect in curve smoothing is that sharp acute corners (like the tip of a star or the serif of a letter) get rounded off into dull fillets.

Vectra2 evaluates second-order difference tangents across adjacent sub-pixel boundary segments. When the interior angle drops below a critical threshold (typically $120^\circ$), the engine terminates the current cubic Bézier segment, places an explicit $G^0$ sharp corner anchor point, and begins a new tangent segment. This ensures that needle-sharp spikes remain needle-sharp.