Image Algorithmics Home Page

Universal Demosaicker

A universal demosaicker allows a single ISP to demosaic all CFA patterns. While the Bayer CFA is currently the pattern of choice, this is likely to change in the near future. As decreasing pixel sizes put pressure on image SNR, inefficiencies of the Bayer CFA pattern are being increasingly questioned, and alternate patterns are being developed.

Some patterns, such as Omnivision/Truesense, Aptina's Clarity+ and Sony's RGBW CFAs employ clear/white/panchromatic and IR pixels to let more light in. Others, such as Fuji's Xtrans are designed to be resistant to aliasing thereby omitting the OLPF and recovering the associated high frequency SNR.

Demosaicking Quality

Several universal demosaickers have been proposed over the years, but their shortcoming has always been quality. CFA specific demosaickers have always generated better quality images than universal demosaickers.

Conventional RGBW Demosaickers

Commercial offerings model RGBW demosaicking on the traditional Bayer method of reconstructing the most densely sampled color plane first and using that to guide the reconstruction of the remaining color planes. The most densely sampled color is W for the RGBW CFA and G for the Bayer. This works well for Bayer that has only 2 colors in each row and column but not for the more complicated RGBW patterns. While these demosaickers can achieve good edge definition in simple pictures, such as charts, they artifact heavily in natural images.

Recent academic approaches demonstrate good simulation results by assuming W = R+G+B. The practicalities of dyes and pigments preclude precise tailoring of R, G, B spectral responses to sum to the W spectral response, severely degrading their performance on real camera raw images.

Noise

Noise further complicates the problem by introducing edge sensing errors. Even when edges/directions are sensed correctly, the one-dimensional nature of the underlying estimators leads to noise streaking. This degrades the texture and micro-detail of the image.

The IA Solution

IA's fresh approach, based on rigorous mathematical analysis, side-steps the minefield of issues with conventional RGBW demosaicking. IA's Universal Demosaicker yields higher image quality on Bayer and Xtrans and on all RGBW CFAs from Omnivision/Truesense, Sony, Aptina than their respective pattern specific algorithms.

Features of the Universal Demosaicker include:

  1. on RGBW patterns,
    • W pixels are only used to improve SNR
    • W = R+G+B is not assumed, resulting in accurate color
  2. structure-less noise amenable to separable noise reduction
  3. tested on real camera images
  4. can support pixel binning
  5. can be extended to extract high bandwidth chrominance
  6. can be extended to correct chromatic aberration
  7. mostly multiply accumulate operations with simple control flow allowing for effective GPU implementation and hardware acceleration

The reconstructed images have few artifacts, accurate color, good resolution and micro-detail as well as fine grained, structure-less noise. These attributes combine to create a photo-realistic look.

RGBW Processing Comparison on a Truesense KAI-01150 Image

Drag the yellow magnifier to zoom into your region of interest

Fine Detail

Observe the lettering on the phone and their improved legibility with the IA demosaicker. The commercial demosaicker, on the other hand, suffers from artifacting.

Color Bleed

Note the clean transition between red and blue patches. Improved demosaicking partly accounts for the reduced color bleed, with noise reduction accounting for the rest.

Noise

RGBW processing requires heavy chrominance denoising but little, if any, luminance denoising since luminance is clean to begin with. The little role demosaicking plays is limited to the texture and and amount of luminance noise. Noise from the commercial demosaicker is slightly lower, but is streaked. The IA demosaicker has fine grained noise that is independent of the image, making it more amenable to denoising.

Download IMAX images here. All images are mosaicked with the Truesense RGBW CFA with noise modeled on the KAI16070 sensor. The optical pipeline consists of a diffraction limited lens with airy disc diameter of 2 pixels and a birefringent OLPF with a 1 pixel shift. Comparable performance has been demonstrated with the Sony RGBW CFA.