Computer vision filters from sensor to screen

A camera effect begins long before a sepia matrix or a blur slider. A conventional image sensor records one color sample at each photosite, an image-signal processor reconstructs the missing colors, and only then do most application filters receive an RGB or YUV frame. This note follows that path from a Bayer mosaic to a display image, derives the filters that matter, and implements the core operations in dependency-free C99 and OpenCV C++.

What a camera actually captures

A photosite measures charge accumulated from incident light. It does not directly report an sRGB triplet. A color filter array (CFA) placed over the sensor restricts each photosite to a broad spectral band. In the common Bayer arrangement, half of the sites carry green filters, one quarter red, and one quarter blue. The two green positions form a denser quincunx sampling lattice. This gives the luminance-like part of the signal more spatial samples, following the design described in Bryce Bayer's 1976 patent.

The four canonical 2 by 2 patterns are named by reading the top row and then the bottom row. An RGGB sensor begins with red at coordinate (0, 0), green at (0, 1) and (1, 0), and blue at (1, 1). Shifting the crop origin by one pixel changes the apparent pattern, which is why raw metadata and row parity are part of the image format rather than incidental bookkeeping.

RGGB                 BGGR                 GRBG                 GBRG

R G R G              B G B G              G R G R              G B G B
G B G B              G R G R              B G B G              R G R G
R G R G              B G B G              G R G R              G B G B
G B G B              G R G R              B G B G              R G R G

The raw sample is normally linear with respect to scene exposure only after black-level subtraction and sensor linearization. White balance compensates for the illuminant and the unequal sensitivity of the color channels. Lens shading corrects spatial falloff. Defective-pixel repair prevents one bad photosite from contaminating several reconstructed pixels. These operations are often performed before or jointly with demosaicing because later interpolation spreads every error.

Bilinear demosaicing, worked by parity

Demosaicing estimates the two missing color values at every location. Bilinear demosaicing makes the local assumption that each color plane changes smoothly. It keeps the measured value and averages the nearest samples of each missing color. For an RGGB mosaic \(M\), the four parity cases are enough to specify the entire algorithm.

At an interior red site \((y,x)\), for example, the reconstruction is

Borders require an explicit policy. Blind coordinate clamping is wrong for a mosaic because clamping a missing north green sample at a red corner can return the red center value. The C implementation below averages only valid neighbors, preserving the expected color at the boundary. Mirror extension is another valid policy when it preserves CFA parity.

Why bilinear interpolation fails at edges

A red-blue boundary violates the smooth-plane assumption. Averaging across it creates false color, while periodic detail near the CFA sampling frequency creates zippering and moiré. The problem is underdetermined because two thirds of every RGB pixel were never measured. A better method must use correlation between channels and local edge direction, but no method can recover detail that aliases before sampling.

Malvar, He, and Cutler start from bilinear estimates and add Laplacian correction terms from the channel that was actually sampled. Their method remains a linear operation and can be expressed with compact 5 by 5 kernels. It uses cross-channel correlation to correct interpolation error and is a useful middle ground between bilinear previews and more expensive adaptive or learned reconstruction. OpenCV also exposes Variable Number of Gradients (VNG) and edge-aware demosaicing modes. Those names describe different algorithms, not quality levels that can be selected without testing the target sensor.

The image-signal-processing pipeline

A practical image-signal processor (ISP) is a sequence of transforms with different mathematical domains. Vendors reorder, fuse, and replace stages, especially when multiple frames are available, but the following map is a useful contract.

sensor mosaic
    │
    ├─ black level, linearization, defective pixels, lens shading
    ├─ white-balance gains
    ├─ demosaic into three device-color channels
    ├─ camera color matrix into a defined working space
    ├─ spatial and temporal denoise
    ├─ local and global tone mapping
    ├─ sharpening and detail control
    ├─ transfer function and gamut mapping
    ▼
display RGB or an encoded YUV image

Operations that model light, including exposure and physically meaningful channel mixing, belong in a linear-light working space. A display transfer function such as sRGB is nonlinear. A numerical average of two sRGB code values is therefore not the same as the average of their light. Blurs and resampling in gamma-encoded space can produce dark fringes around bright features. Some creative looks intentionally accept that behavior, but it should be a choice rather than an accident.

A normal mobile camera preview has already passed through much of this pipeline and commonly arrives as YUV. It is not a live Bayer buffer. Raw photo capture is a separate path with sensor metadata, higher bit depth, and a different latency budget. An application should not expose a “Bayer” control on an ordinary processed frame as though it were changing the sensor reconstruction.

Linear spatial filters

A finite impulse response image filter combines a neighborhood with a kernel. For source image \(I\) and kernel \(K\), discrete convolution flips the kernel in both axes,

Cross-correlation uses \(I(y+j,x+i)\) instead. Many computer vision APIs call correlation “convolution.” The distinction disappears for symmetric kernels such as a Gaussian, but changes the sign or orientation of derivative kernels. The C implementation below performs the mathematical convolution and names its replicated border policy.

Box and Gaussian blur

A normalized box filter assigns every sample in a window weight \(1/(wh)\). It is cheap with an integral image or running sums, but its sharp frequency cutoff produces a less natural point-spread function. A Gaussian gives nearby pixels more weight,

The Gaussian is separable because \(G_\sigma(x,y)=g_\sigma(x)g_\sigma(y)\). A \(k\) by \(k\) convolution therefore becomes one horizontal and one vertical pass, reducing work per pixel from \(O(k^2)\) to \(O(k)\). Kernel coefficients should sum to one when the goal is to preserve a constant image.

3 × 3 box                    3 × 3 Gaussian approximation

1 1 1                        1 2 1
1 1 1  × 1/9                 2 4 2  × 1/16
1 1 1                        1 2 1

Border behavior is part of the algorithm

Zero padding darkens the frame because the filter sees invented black pixels. Replication extends the nearest edge value. Reflect padding mirrors the image and usually introduces a smaller derivative discontinuity. Wrap padding is correct for periodic textures and surprising for photographs. OpenCV exposes these as border modes. Tests should name the selected mode because two correct implementations with different borders will disagree around the perimeter.

Gradients, edges, and sharpening

An image gradient estimates spatial change. The Sobel operator combines a central difference in one direction with smoothing in the other.

Sobel Gx                     Sobel Gy

-1  0  1                     -1 -2 -1
-2  0  2                      0  0  0
-1  0  1                      1  2  1

The gradient magnitude is \(\sqrt{G_x^2+G_y^2}\), while \(\operatorname{atan2}(G_y,G_x)\) gives orientation. Scharr uses coefficients 3 and 10 in place of Sobel's 1 and 2 to improve rotational symmetry for a 3 by 3 derivative. The Laplacian estimates the sum of second derivatives and responds on both sides of a transition, which makes it useful for focus measures and sharpening but more sensitive to noise.

Canny is a pipeline, not one kernel

1. Suppress high-frequency noise with a Gaussian.
2. Estimate gradient magnitude and direction.
3. Apply non-maximum suppression along the gradient direction to thin each ridge.
4. Classify pixels with low and high thresholds.
5. Keep weak pixels only when hysteresis connects them to a strong edge.

The low threshold controls continuity and the high threshold controls confidence. A useful interface can expose one threshold and derive the other by a documented ratio, but the underlying values still depend on bit depth, normalization, blur scale, and scene contrast.

Unsharp masking adds high frequencies back

Despite its name, unsharp masking is a sharpening method. A low pass image \(B=G_\sigma * I\) defines the high-frequency residual \(H=I-B\), then

The radius controls the spatial scale through \(\sigma\), while the amount \(a\) controls gain. A threshold can suppress small residuals so sensor noise is not sharpened with real structure. Large radii and amounts create halos because the method cannot distinguish an edge from a desired overshoot.

Nonlinear and edge-aware filters

A median filter replaces a pixel with the median of its neighborhood. It rejects isolated impulse noise without averaging the outlier into nearby pixels, but repeated use rounds corners and removes thin structures. It is nonlinear because the output cannot be written as one fixed weighted sum of the inputs.

A bilateral filter makes the weights depend on both spatial distance and intensity difference,

The spatial scale \(\sigma_s\) controls reach. The range scale \(\sigma_r\) controls how different two values may be before they stop influencing each other. This preserves high-contrast edges, though it can flatten texture into cartoon-like regions and is substantially more expensive than a separable Gaussian. Applying the range distance independently to red, green, and blue can also shift color. A perceptual or luminance-chrominance space is often a better metric.

The guided filter assumes a local linear relationship between a guidance image and the output. Box-filter statistics make an exact linear-time implementation possible independent of the nominal radius. It is useful for edge-aware smoothing, detail decomposition, joint upsampling, and feathering masks. Unlike the bilateral filter, its local linear model avoids some gradient reversal artifacts near strong edges.

Morphology operates on shapes

For a bright-foreground grayscale image, dilation takes a local maximum and erosion takes a local minimum over a structuring element. Opening, erosion followed by dilation, removes small bright components. Closing, dilation followed by erosion, fills small dark gaps. These operations are appropriate for masks, segmentation cleanup, and graphic effects. They are not substitutes for denoising a continuous-tone photograph.

Color and stylistic filters

Many camera looks need no spatial neighborhood. Exposure in linear light multiplies RGB by \(2^{e}\), where \(e\) is measured in exposure values. White balance applies channel gains. A color matrix rotates and scales the three-channel vector. Saturation can interpolate between a luminance estimate and the original color. Curves and lookup tables (LUTs) provide nonlinear tone and color mappings.

A one-dimensional LUT maps each channel independently and is suitable for tone curves. A three-dimensional LUT maps an RGB point to another RGB point and can encode coupled hue changes. Trilinear or tetrahedral interpolation avoids hard boundaries between lattice cells. Blending the transformed image with the original gives an intuitive strength control, \(O=(1-t)I+tF(I)\).

Posterization quantizes each channel to a small number of levels. A halftone compares local intensity against a periodic screen or distributes quantization error through neighboring pixels. Chromatic aberration shifts channels by slightly different transforms. Bloom extracts values above a threshold, blurs them, and adds the result back in a linear high-dynamic-range space. Each effect is simple in isolation. The visual result depends more on ordering, color space, and clipping policy than on the name in the filter tray.

Mapping algorithms to camera controls

User-facing controls should be resolution-aware and expressed in a stable domain. A blur radius measured in full-resolution pixels cannot be reused unchanged on a quarter-size preview. Thresholds stated in 8-bit code values must be rescaled for 10-bit, 16-bit, or normalized floating-point input.

Runnable C99 and OpenCV C++

The C tab implements RGGB bilinear demosaicing for unpacked 16-bit samples and a generic floating-point convolution with replicated borders. It carries explicit width and row stride, does not assume tightly packed rows, avoids in-place convolution, and includes deterministic self-tests. The constant-color CFA test is especially useful because every reconstructed pixel should return the same RGB triplet, including borders.

The C++ tab uses OpenCV 4.11.0. It reads a deliberately narrow raw interchange format, one native-endian unpacked unsigned 16-bit sample per pixel, then calls the explicit COLOR_BayerRGGB2BGR_EA conversion. Real camera files may pack 10, 12, or 14 bits and carry black levels, crop origins, orientation, CFA layout, and color matrices in metadata. A raw decoder must resolve those before this example begins.

filters.c#include <assert.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>

typedef struct {
    const uint16_t *data;
    size_t width;
    size_t height;
    size_t stride;  /* uint16_t samples per row */
} Raw16View;

typedef struct {
    uint16_t *data;
    size_t width;
    size_t height;
    size_t stride;  /* uint16_t values per row, interleaved RGB */
} Rgb16View;

typedef struct {
    int dx;
    int dy;
} Offset;

static uint16_t average_valid(
    Raw16View raw,
    size_t x,
    size_t y,
    const Offset *offsets,
    size_t count
) {
    uint64_t sum = 0;
    size_t used = 0;

    for (size_t i = 0; i < count; ++i) {
        const ptrdiff_t sx = (ptrdiff_t)x + offsets[i].dx;
        const ptrdiff_t sy = (ptrdiff_t)y + offsets[i].dy;
        if (sx >= 0 && sy >= 0 &&
            (size_t)sx < raw.width && (size_t)sy < raw.height) {
            sum += raw.data[(size_t)sy * raw.stride + (size_t)sx];
            ++used;
        }
    }

    assert(used != 0);
    return (uint16_t)((sum + used / 2) / used);
}

bool demosaic_rggb_bilinear(Raw16View raw, Rgb16View rgb) {
    static const Offset cross[4] = {
        { 0, -1}, { 0, 1}, {-1, 0}, {1, 0}
    };
    static const Offset diagonal[4] = {
        {-1, -1}, {1, -1}, {-1, 1}, {1, 1}
    };
    static const Offset horizontal[2] = {
        {-1, 0}, {1, 0}
    };
    static const Offset vertical[2] = {
        {0, -1}, {0, 1}
    };

    if (raw.data == NULL || rgb.data == NULL ||
        raw.width < 2 || raw.height < 2 ||
        raw.width != rgb.width || raw.height != rgb.height ||
        raw.stride < raw.width ||
        rgb.width > SIZE_MAX / 3 ||
        rgb.stride < rgb.width * 3) {
        return false;
    }

    for (size_t y = 0; y < raw.height; ++y) {
        for (size_t x = 0; x < raw.width; ++x) {
            const bool even_y = (y & 1U) == 0;
            const bool even_x = (x & 1U) == 0;
            const uint16_t measured = raw.data[y * raw.stride + x];
            uint16_t r;
            uint16_t g;
            uint16_t b;

            if (even_y && even_x) {
                /* R G
                   G B */
                r = measured;
                g = average_valid(raw, x, y, cross, 4);
                b = average_valid(raw, x, y, diagonal, 4);
            } else if (even_y) {
                /* Green on a red row. */
                r = average_valid(raw, x, y, horizontal, 2);
                g = measured;
                b = average_valid(raw, x, y, vertical, 2);
            } else if (even_x) {
                /* Green on a blue row. */
                r = average_valid(raw, x, y, vertical, 2);
                g = measured;
                b = average_valid(raw, x, y, horizontal, 2);
            } else {
                b = measured;
                g = average_valid(raw, x, y, cross, 4);
                r = average_valid(raw, x, y, diagonal, 4);
            }

            uint16_t *pixel = rgb.data + y * rgb.stride + 3 * x;
            pixel[0] = r;
            pixel[1] = g;
            pixel[2] = b;
        }
    }
    return true;
}

typedef struct {
    const float *data;
    size_t width;
    size_t height;
    size_t stride;  /* float values per row */
} GrayF32View;

typedef struct {
    float *data;
    size_t width;
    size_t height;
    size_t stride;
} GrayF32Output;

static size_t clamp_coordinate(ptrdiff_t value, size_t limit) {
    if (value < 0) {
        return 0;
    }
    if ((size_t)value >= limit) {
        return limit - 1;
    }
    return (size_t)value;
}

bool convolve_f32(
    GrayF32View src,
    GrayF32Output dst,
    const float *kernel,
    size_t kernel_width,
    size_t kernel_height
) {
    if (src.data == NULL || dst.data == NULL || kernel == NULL ||
        src.data == dst.data ||
        src.width == 0 || src.height == 0 ||
        src.width != dst.width || src.height != dst.height ||
        src.stride < src.width || dst.stride < dst.width ||
        kernel_width == 0 || kernel_height == 0 ||
        (kernel_width & 1U) == 0 || (kernel_height & 1U) == 0) {
        return false;
    }

    const ptrdiff_t radius_x = (ptrdiff_t)(kernel_width / 2);
    const ptrdiff_t radius_y = (ptrdiff_t)(kernel_height / 2);

    for (size_t y = 0; y < src.height; ++y) {
        for (size_t x = 0; x < src.width; ++x) {
            double sum = 0.0;
            for (size_t ky = 0; ky < kernel_height; ++ky) {
                for (size_t kx = 0; kx < kernel_width; ++kx) {
                    const ptrdiff_t offset_y = (ptrdiff_t)ky - radius_y;
                    const ptrdiff_t offset_x = (ptrdiff_t)kx - radius_x;
                    const size_t sy =
                        clamp_coordinate((ptrdiff_t)y + offset_y, src.height);
                    const size_t sx =
                        clamp_coordinate((ptrdiff_t)x + offset_x, src.width);

                    /* Flip both kernel axes for mathematical convolution. */
                    const size_t flipped_y = kernel_height - 1 - ky;
                    const size_t flipped_x = kernel_width - 1 - kx;
                    const float weight =
                        kernel[flipped_y * kernel_width + flipped_x];
                    sum += (double)src.data[sy * src.stride + sx] * weight;
                }
            }
            dst.data[y * dst.stride + x] = (float)sum;
        }
    }
    return true;
}

static void test_constant_rggb(void) {
    enum { WIDTH = 4, HEIGHT = 4 };
    uint16_t raw[WIDTH * HEIGHT];
    uint16_t rgb[WIDTH * HEIGHT * 3] = {0};

    for (size_t y = 0; y < HEIGHT; ++y) {
        for (size_t x = 0; x < WIDTH; ++x) {
            if ((y & 1U) == 0 && (x & 1U) == 0) {
                raw[y * WIDTH + x] = 1000;  /* red */
            } else if ((y & 1U) != 0 && (x & 1U) != 0) {
                raw[y * WIDTH + x] = 100;   /* blue */
            } else {
                raw[y * WIDTH + x] = 500;   /* green */
            }
        }
    }

    const Raw16View input = {raw, WIDTH, HEIGHT, WIDTH};
    const Rgb16View output = {rgb, WIDTH, HEIGHT, WIDTH * 3};
    assert(demosaic_rggb_bilinear(input, output));

    for (size_t i = 0; i < WIDTH * HEIGHT; ++i) {
        assert(rgb[3 * i + 0] == 1000);
        assert(rgb[3 * i + 1] == 500);
        assert(rgb[3 * i + 2] == 100);
    }
}

static void test_box_convolution(void) {
    static const float box[9] = {
        1.0f / 9.0f, 1.0f / 9.0f, 1.0f / 9.0f,
        1.0f / 9.0f, 1.0f / 9.0f, 1.0f / 9.0f,
        1.0f / 9.0f, 1.0f / 9.0f, 1.0f / 9.0f
    };
    float impulse[25] = {0};
    float blurred[25] = {0};
    impulse[2 * 5 + 2] = 1.0f;

    const GrayF32View input = {impulse, 5, 5, 5};
    const GrayF32Output output = {blurred, 5, 5, 5};
    assert(convolve_f32(input, output, box, 3, 3));

    double total = 0.0;
    for (size_t i = 0; i < 25; ++i) {
        total += blurred[i];
    }
    assert(total > 0.999999 && total < 1.000001);
    assert(blurred[2 * 5 + 2] > 0.111110f);
    assert(blurred[2 * 5 + 2] < 0.111112f);
}

int main(void) {
    test_constant_rggb();
    test_box_convolution();
    puts("C99 demosaic and convolution tests passed");
    return 0;
}

opencv_filters.cpp#include <algorithm>
#include <cmath>
#include <cstdint>
#include <fstream>
#include <iostream>
#include <limits>
#include <stdexcept>
#include <string>
#include <vector>

#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>

cv::Mat read_unpacked_raw16(
    const std::string& path,
    int width,
    int height
) {
    if (width <= 0 || height <= 0) {
        throw std::invalid_argument("width and height must be positive");
    }

    cv::Mat raw(height, width, CV_16UC1);
    const auto bytes = static_cast<std::streamsize>(
        raw.total() * raw.elemSize()
    );
    std::ifstream input(path, std::ios::binary);
    if (!input.read(reinterpret_cast<char *>(raw.data), bytes)) {
        throw std::runtime_error(
            "raw file is missing or does not contain width*height uint16 samples"
        );
    }

    char extra;
    if (input.get(extra)) {
        throw std::runtime_error("raw file contains trailing bytes");
    }
    return raw;
}

cv::Mat gamma_lut(float gamma) {
    if (!(gamma > 0.0f)) {
        throw std::invalid_argument("gamma must be positive");
    }

    cv::Mat lut(1, 256, CV_8UC1);
    for (int i = 0; i < 256; ++i) {
        const float x = static_cast<float>(i) / 255.0f;
        const float y = std::pow(x, 1.0f / gamma);
        lut.at<std::uint8_t>(i) =
            cv::saturate_cast<std::uint8_t>(255.0f * y);
    }
    return lut;
}

void write_pgm(const std::string& path, const cv::Mat& gray8) {
    if (gray8.type() != CV_8UC1) {
        throw std::invalid_argument("PGM output requires one-channel uint8");
    }

    std::ofstream output(path, std::ios::binary);
    output << "P5\n" << gray8.cols << ' ' << gray8.rows << "\n255\n";
    for (int y = 0; y < gray8.rows; ++y) {
        output.write(
            reinterpret_cast<const char *>(gray8.ptr(y)),
            static_cast<std::streamsize>(gray8.cols)
        );
    }
    if (!output) {
        throw std::runtime_error("failed to write " + path);
    }
}

void write_ppm_from_bgr(
    const std::string& path,
    const cv::Mat& bgr8
) {
    if (bgr8.type() != CV_8UC3) {
        throw std::invalid_argument("PPM output requires three-channel uint8");
    }

    std::ofstream output(path, std::ios::binary);
    output << "P6\n" << bgr8.cols << ' ' << bgr8.rows << "\n255\n";
    std::vector<std::uint8_t> row(
        static_cast<size_t>(bgr8.cols) * 3
    );
    for (int y = 0; y < bgr8.rows; ++y) {
        const cv::Vec3b *source = bgr8.ptr<cv::Vec3b>(y);
        for (int x = 0; x < bgr8.cols; ++x) {
            const size_t offset = static_cast<size_t>(x) * 3;
            row[offset + 0] = source[x][2];
            row[offset + 1] = source[x][1];
            row[offset + 2] = source[x][0];
        }
        output.write(
            reinterpret_cast<const char *>(row.data()),
            static_cast<std::streamsize>(row.size())
        );
    }
    if (!output) {
        throw std::runtime_error("failed to write " + path);
    }
}

struct FilterResult {
    cv::Mat graded_bgr8;
    cv::Mat canny8;
    cv::Mat scharr8;
};

FilterResult process_rggb(const cv::Mat& raw16) {
    if (raw16.type() != CV_16UC1) {
        throw std::invalid_argument("expected one-channel uint16 Bayer data");
    }

    cv::Mat bgr16;
    cv::demosaicing(raw16, bgr16, cv::COLOR_BayerRGGB2BGR_EA);

    cv::Mat bgr32;
    bgr16.convertTo(bgr32, CV_32FC3, 1.0 / 65535.0);

    /* Demonstration gains only. Real values come from raw metadata. */
    std::vector<cv::Mat> channels;
    cv::split(bgr32, channels);
    channels[0] *= 1.20f;  /* blue */
    channels[1] *= 1.00f;  /* green */
    channels[2] *= 1.65f;  /* red */
    cv::merge(channels, bgr32);

    cv::Mat denoised;
    cv::bilateralFilter(
        bgr32,
        denoised,
        7,                  /* pixel diameter */
        0.08,               /* range sigma in normalized color units */
        3.0,                /* spatial sigma in pixels */
        cv::BORDER_REFLECT_101
    );

    cv::Mat lowpass;
    cv::GaussianBlur(
        denoised,
        lowpass,
        cv::Size(),
        1.2,
        1.2,
        cv::BORDER_REFLECT_101
    );

    cv::Mat sharpened;
    constexpr double amount = 0.65;
    cv::addWeighted(
        denoised,
        1.0 + amount,
        lowpass,
        -amount,
        0.0,
        sharpened
    );
    cv::max(sharpened, 0.0, sharpened);
    cv::min(sharpened, 1.0, sharpened);

    cv::Mat gray32;
    cv::cvtColor(denoised, gray32, cv::COLOR_BGR2GRAY);

    cv::Mat gx;
    cv::Mat gy;
    cv::Scharr(gray32, gx, CV_32F, 1, 0);
    cv::Scharr(gray32, gy, CV_32F, 0, 1);

    cv::Mat magnitude;
    cv::magnitude(gx, gy, magnitude);
    double max_magnitude = 0.0;
    cv::minMaxLoc(magnitude, nullptr, &max_magnitude);
    cv::Mat scharr8 = cv::Mat::zeros(magnitude.size(), CV_8UC1);
    if (max_magnitude > 0.0) {
        magnitude.convertTo(scharr8, CV_8UC1, 255.0 / max_magnitude);
    }

    cv::Mat gray8;
    gray32.convertTo(gray8, CV_8UC1, 255.0);
    cv::Mat canny_input;
    cv::GaussianBlur(
        gray8,
        canny_input,
        cv::Size(5, 5),
        1.0,
        1.0,
        cv::BORDER_REFLECT_101
    );
    cv::Mat canny8;
    cv::Canny(
        canny_input,
        canny8,
        60.0,   /* low threshold */
        150.0,  /* high threshold */
        3,
        true    /* L2 gradient magnitude */
    );

    cv::Mat display8;
    sharpened.convertTo(display8, CV_8UC3, 255.0);
    cv::Mat graded8;
    cv::LUT(display8, gamma_lut(1.08f), graded8);
    return {graded8, canny8, scharr8};
}

int parse_dimension(const char *text) {
    const long value = std::stol(text);
    if (value <= 0 || value > std::numeric_limits<int>::max()) {
        throw std::out_of_range("dimension is outside the int range");
    }
    return static_cast<int>(value);
}

int main(int argc, char **argv) {
    if (argc != 6) {
        std::cerr
            << "usage: opencv_filters raw16 width height output.ppm edges.pgm\n";
        return 2;
    }

    try {
        const int width = parse_dimension(argv[2]);
        const int height = parse_dimension(argv[3]);
        const cv::Mat raw = read_unpacked_raw16(argv[1], width, height);
        const FilterResult result = process_rggb(raw);

        write_ppm_from_bgr(argv[4], result.graded_bgr8);
        write_pgm(argv[5], result.canny8);
        write_pgm("scharr-magnitude.pgm", result.scharr8);
    } catch (const std::exception& error) {
        std::cerr << "error: " << error.what() << '\n';
        return 1;
    }
    return 0;
}

The C test builds without external libraries.

build-c.shclang -std=c99 -O2 -Wall -Wextra -Wconversion -pedantic filters.c -o filters
./filters

A Homebrew OpenCV installation can build the C++ example by exposing its include and library directories. The example writes portable PPM and PGM files directly, so it only links the OpenCV modules it uses.

build-opencv.shopencv_prefix="$(brew --prefix opencv)"
clang++ -std=c++17 -O2 -Wall -Wextra opencv_filters.cpp \
  -I"$opencv_prefix/include/opencv4" -L"$opencv_prefix/lib" \
  -lopencv_imgproc -lopencv_core -o opencv_filters

Making filters fast on a phone

A live preview and a saved photograph have different contracts. The preview must meet a frame deadline and can operate at reduced resolution. A full-resolution export may take longer, but should run off the capture callback, expose progress, support cancellation, and preserve the original. Reusing the same parameter model at both resolutions prevents the preview from promising a different look.

• Respect row stride and pixel format. Camera planes are not guaranteed to be tightly packed. A YUV preview may have separate luma and chroma planes, while display buffers may be BGRA. Conversion should happen once at a deliberate boundary.
• Scale the work before optimizing arithmetic. Render the preview near screen resolution, crop to the active region when possible, and avoid full-frame effects behind an opaque overlay.
• Use separability and running sums. Gaussian and box filters should not pay \(k^2\) work per pixel when their structure permits \(O(k)\) or constant work.
• Fuse adjacent point operations. Exposure, channel gains, a color matrix, and a tone curve can often share one pass or one GPU kernel. Each avoidable intermediate costs a full image read and write.
• Pool scratch storage. Repeated allocation in a frame callback produces latency spikes. Preallocate bounded working buffers and keep ownership explicit.
• Prefer production primitives. On Apple platforms, Core Image and Metal fit GPU filter graphs, while Accelerate and vImage provide vectorized CPU image operations. OpenCV is valuable for portable algorithms. A scalar loop is still useful as a correctness oracle.
• Measure the whole frame path. Kernel time alone excludes pixel-format conversion, synchronization, allocation, orientation, and display. Report warm and cold latency at the exact resolution and device tested.

Testing image code without trusting the picture

Visual inspection catches gross artifacts and misses systematic errors that look plausible. Image code needs small exact fixtures beside natural images.

Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) can compare reconstruction against a known reference. Neither proves that a filter looks good, and both are meaningless without a defined dataset, crop, color space, bit depth, and border policy. Color pipelines also need neutral-patch and saturated-patch tests. Performance claims need the same discipline, including device, image size, pixel format, number of warmup frames, and percentile latency rather than one best run.

Common misconceptions

A Bayer image is a low-resolution RGB image. It is one scalar sample per photosite with color identity supplied by position. Treating it as interleaved RGB corrupts both spatial layout and channel meaning.

More blur radius only changes strength. Radius changes scale. A larger Gaussian removes larger structures, while repeated small blurs compose into a different effective sigma unless the parameters are combined mathematically.

Convolution and correlation are always interchangeable. They agree for symmetric kernels and differ for asymmetric derivatives. The operation and sign convention must be tested.

Bilateral filtering preserves every edge. Preservation is controlled by the range sigma. Texture below that contrast scale is smoothed, and large values approach an ordinary spatial blur.

Canny thresholds transfer between cameras. Thresholds depend on normalization, bit depth, denoising, exposure, and scene contrast. Fixed 8-bit constants are not a camera model.

RGB is one universal color space. RGB describes a channel model, not primaries, white point, transfer function, range, or alpha convention. Those properties determine what matrix multiplication and blending mean.

A fast kernel guarantees a smooth preview. Copies, allocation, queueing, GPU synchronization, and display conversion can dominate the measured kernel.

Sources and code to read

1. Bryce E. Bayer, Color imaging array, U.S. Patent 3,971,065, filed 1975 and published 1976.
2. Henrique S. Malvar, Li-wei He, and Ross Cutler, High-Quality Linear Interpolation for Demosaicing of Bayer-Patterned Color Images, IEEE ICASSP, 2004.
3. Pascal Getreuer, Malvar-He-Cutler Linear Image Demosaicking, Image Processing On Line, 2011, with an ANSI C reference implementation.
4. Bahadir K. Gunturk et al., Demosaicking, Color Filter Array Interpolation in Single-Chip Digital Cameras, IEEE Signal Processing Magazine, 2005.
5. Edward Chang, Shiufun Cheung, and Davis Y. Pan, Color Filter Array Recovery Using a Threshold-Based Variable Number of Gradients, SPIE, 1999.
6. John Canny, A Computational Approach to Edge Detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986.
7. Carlo Tomasi and Roberto Manduchi, Bilateral Filtering for Gray and Color Images, IEEE International Conference on Computer Vision, 1998.
8. Kaiming He, Jian Sun, and Xiaoou Tang, Guided Image Filtering, European Conference on Computer Vision, 2010.
9. Hanno Scharr, Optimal Operators in Digital Image Processing, doctoral dissertation, Heidelberg University, 2000.
10. Richard Szeliski, Computer Vision, Algorithms and Applications, second edition, 2022.
11. Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, fourth edition, 2018.
12. OpenCV, Color conversions and Image filtering, API reference checked against the installed 4.11.0 headers in July 2026.
13. OpenCV, imgproc.hpp at tag 4.11.0, the authoritative enumeration and function declarations used by the C++ example.
14. Apple, vImage documentation and Core Image documentation, production primitives for Apple-platform image pipelines.