LMS CAMERA
ENGINEERED LIKE THE RETINA
The LMS sensor has spectral responses identical to the Long, Medium and Short wavelength cones of the human retina, hence the name LMS. It senses colors exactly as our eyes do.
The L, M, S colors are arranged in a pattern more like the human retina than the Bayer CFA. It has more L and M pixels but fewer S pixels.
Benefits
- Accurate color in any lighting, including mixed lighting
- 5 dB SNR advantage over RGB in low light
- 3.5 dB SNR advantage over RGB in bright light
TECHNICAL DETAILS
The Human Visual System (HVS) uses M and S cones that are similar to G and B, respectively, used by image sensors. However, the L cone is closer to R+G than R. Roughly speaking, the HVS calculates R as the difference L-M.
The HVS removes both the luma and chroma noise. Now, luma denoising is hard but chroma denoising is easy. By following this indirect route of sensing L=R+G, the HVS increases its luma SNR, since L is more sensitive than R, but decreases its chroma SNR, since L-M is noisier than directly sensed R. Thus the HVS trades the hard problem of removing luma noise for the easy problem of removing chroma noise. The LMS camera does the same.
RGB Color Filter Choices of Today's Cameras
Image sensor color filters, for the most part, were designed before denoising was common or even possible with the signal processing hardware of the time. Without denoising, sensing R directly is a rational choice.
However, color errors are inevitable when sensing R directly since part of its spectrum is negative. This is the primary source of color errors in today's cameras. The spectrum of L, on the other hand, is positive at all frequencies and is an easy to manufacture Gaussian function.
Attempts to circumvent R's negative spectrum by using the XYZ color space, which is a linear combination of LMS colorspace, have worked for color meters, but the filters needed have proven to be too complex for CFA implementation. With the availability of denoising the motivation for using XYZ color space, instead of LMS, is not clear since it trades the easy to remove chroma noise for the hard to remove luma noise.
Chroma Denoising
Chroma denoising is very effective and generates very few artifacts. It does not obscure detail or texture. Luma denoising is much harder and is responsible for the loss of texture, and for the plastic and painted look of many image processors.
When applied gently on brightly lit images, chroma denoising is essentially transparent, causing no visible damage. When applied strongly on low light images, chroma denoising desaturates colors of small features but cleans up the image a great deal in the bargain.
Chroma denoising is almost universally used by modern Image Signal Processors.
The New Color Filter Array
The new CFA has more L and fewer S than a hypothetical Bayer LMS pattern for the following reasons:
- L is more sensitive than S, therefore replacing S with L increases luma SNR.
- More L pixels improves the SNR of the otherwise noisy R computed as L-M.
- The human visual system is not sensitive to the resulting loss quality of S/blue because the retina itself has very few S.
Sparse S Pixels
Low density of S/blue, or any other color, does not automatically lead to resolution loss since natural images are compressible and other colors are densely sampled. The Nyquist theorem does not apply in this setting and sampling density can be reduced - up to a point.
The sparsity of S pixels does increase blue noise and requires more denoising. Fortunately, blue denoising artifacts are not very visible because the retina itself has very few S.
Manufacturing Color Filters
The LMS camera requires the new L color filter, which is a shade of yellow, to be manufactured. This is a relatively simple task because the response curve of L is Gaussian. In contrast, several non-RGB color filter arrays proposed in the past had complicated spectra and were hard to manufacture.
COMPARISON WITH OTHER COLOR CAMERA ARCHITECTURES
Cyan-Magenta-Yellow
CMY is similar to LMS in that the high transmissivity of Cyan, Magenta and Yellow color filters increases luma SNR but the color conversion to RGB increases chroma noise. When coupled with chroma denoising, this yields a noise profile similar to LMS.
The problem with CMY is poor color accuracy, poorer than even conventional RGB sensors due to difficulties in manufacturing its color filters. The luma sensitivity of CMY is roughly similar to LMS since L is similar to Yellow and the quantum efficiency of M is between those of Magenta and Cyan. The quantum efficiency of S is lower than all CMY colors, but only 1 in 8 pixels in the LMS CFA is S. This raises the possibility of a small luma SNR advantage of CMY over LMS whose quantification needs careful accounting of quantum efficiencies of the sensor silicon in question. This small possible luma SNR lead of CMY over LMS is almost certainly not worth its color errors.
RYYB
The erstwhite Aptina had proposed and Huawei has implemented a Bayer pattern sensor with G replaced with Yellow. Since L is similar to Y and accounts for half the pixels in both LMS and RYYB, the difference between LMS and RYYB is reduced to how the other half of the pixels are colored.
RYYB obtains G by differencing Y-R while LMS obtains R by differencing L-M which is similar to Y-G. RYYB suffers from poor R and G accuracy; R since accurate R is impossible to sense directly and G because of difficulties in manufacturing Y such that Y-R is high quality green. In contrast, LMS captures high quality G by directly sensing G (M) and its calculated R=L-M is more accurate than any directly sensed R can be. This makes RYYB's colors less accurate than conventional RGB imagers while LMS' colors are more accurate than RGB.
Noise of R=L-M is controlled by the high density of both L and M in the LMS CFA. Noise of G=Y-R is not as well controlled because of the low density of R in the RYYB CFA. Furthermore loss of G quality, either as increased noise or increased chroma denoising artifacts, is very visible since G is the color our eyes see best.
LMS also has a luma SNR advantage over RYYB since it has a lot of G pixels that are more sensitive than either R or B. The only advantage of RYYB over LMS is higher blue SNR since it has twice as many B pixels as LMS has the equivalent S pixels. This is of minor importance since our retinas have very few S and cannot easily sense loss of blue quality. The improved G SNR of LMS alone is more valuable than the improved B SNR of RYYB, with the improved color accuracy and luma SNR being extra benefits of LMS.
RGBW
RGBW CFA patterns with 50% White or Clear pixels admit more light than LMS. However the luma SNR of LMS is surprisingly close to RGBW because of better numerical stability of LMS demosaicking.
It is too early to say if RGBW or LMS has better SNR. For now it is safe to say that LMS promises better colors but requires the new L color filter to be manufactured, while RGBW is available today.
Foveon and other Multi-Layer Image Sensors
The Foveon X3 has 3 stacked photosites at each pixel, with the top layer sensing B, the middle layer G and the bottom layer R. Other similar image sensors have also been proposed with different photo-sensitive layers for R, G and B.
Foveon and other multi-layer image sensors cannot be used for LMS because of the large spectral overlap between L and M. Once a layer senses M, and in so doing removes light in M's spectra, most of L's signal is lost. Placing the L sensing layer over M has the reverse problem.
Random Color Filter Array
The L, M, S cones in the human retina are hexagonal and arranged in a random pattern, while the LMS image sensor pixels are square and arranged in a regular pattern. Both are optimal for their pixel shape.
Square pixels admit higher resolution along the diagonal directions than the horizontal and vertical directions. Since lenses have the same resolution along all directions, this extra diagonal resolution goes unused. Color filter arrays with regular patterns take advantage of this unused diagonal channel capacity to convey color information.
Hexagonal pixels, on the other hand, have almost the same resolution in every direction and are, thus, less wasteful. You need 20% fewer hexagonal pixels than square pixels for the same resolution monochrome image. The simple mechanism of color encoding using regular color patterns does not work with hexagonal pixels; instead more sophisticated compressive sensing techniques with random color patterns must be used. This is what the HVS likely does.
While hexagonal pixels are more efficient, yielding higher frame rates or consuming less power or both, the imaging industry is standardized on square pixels for now. Random color patterns do not outperform regular color patterns on square pixel sensors.