New DSLR with Exmor R CMOS sensor

[David Kilpatrick]

Daniel, I think I know what Agorabasta is on about.

A typical camera lens produces three sizes of images for R, G and B channels - we use the G channel, which represents the middle magnification across the wavelength range for a non-apochromatic lens, for luminance and this also has twice the spatial frequency weighting.

The R and B channels even with the existing sensors cause visible CA (not blooming-related fringes) which can be corrected by changing the scale of the R and B channels relative to the G, around the image axis. CA correction in Photoshop, for this reason of axially centred symmetry, does not function properly on a pre-cropped image. CA correction in ACR is similar to correction in Photoshop. It simply changes the scaling of the R and B channels as required.

Agorabasta is pointing out that if R and B pixels, in a second layer, are used in any way to derive luminance values then a perfectly apochromatic lens would be needed. Red and blue image detail is most strongly separated in terms of magnification when using nearly all current camera lenses, including so-called APO designs (which are really superachromats, in Zeiss's terms). Interpreting the R and B pixel values would require a map of the CA (or of the scaling the R and B components of the image).

In this case, the imperfect performance of real world optics, relative to a theoretical model where RGB is precisely coincident to pixel level, would probably mess up calculations.

I may be wrong, I'm not a scientist, and my understanding is generally limited to things I can observe.

David

[dkloi]

Daniel, I think I know what Agorabasta is on about.

A typical camera lens produces three sizes of images for R, G and B channels - we use the G channel, which represents the middle magnification across the wavelength range for a non-apochromatic lens, for luminance and this also has twice the spatial frequency weighting.

The R and B channels even with the existing sensors cause visible CA (not blooming-related fringes) which can be corrected by changing the scale of the R and B channels relative to the G, around the image axis. CA correction in Photoshop, for this reason of axially centred symmetry, does not function properly on a pre-cropped image. CA correction in ACR is similar to correction in Photoshop. It simply changes the scaling of the R and B channels as required.

Agorabasta is pointing out that if R and B pixels, in a second layer, are used in any way to derive luminance values then a perfectly apochromatic lens would be needed. Red and blue image detail is most strongly separated in terms of magnification when using nearly all current camera lenses, including so-called APO designs (which are really superachromats, in Zeiss's terms). Interpreting the R and B pixel values would require a map of the CA (or of the scaling the R and B components of the image).

In this case, the imperfect performance of real world optics, relative to a theoretical model where RGB is precisely coincident to pixel level, would probably mess up calculations.

I may be wrong, I'm not a scientist, and my understanding is generally limited to things I can observe.

David

Thanks David for interpreting . I think the CA thing is a red herring in trying to understand interpolation and demosaicing. CA is not special, it's just an aspect of the image that the sensor is sampling. The goal of the sensor and image processing chain is to reconstruct in the best way the RGB value of the light at the location of each pixel (up to some Nyquist frequency cut-off). Any way of reconstructing the image better, including its CA, means the better you can compensate for CA after the reconstruction.

The demosaicing actually happens in RGB space, not LAB (I clarified this point in my previous reply). The pixels on the top layer gather R+G or B+G signals, the bottom row B or R. Demosaicing gives a reconstructed RGB image (Note 1). The maths is quite simple for the simplest kind of (linear) interpolation and demosaicing.

Step 1. Interpolate R and B channels. Choose interpolation methods of choice. Bilinear is simple and fast.

Step 2. Derive G channel from (R+G)-R and (B+G)-B. (Note 2)

Step 3. End up with a fully reconstructed RGB image.

Step 4. Now you can do all the fussy business of mapping the reconstructed sensor-referred RGB image to a tri-stimulus model with white-balance etc. It's here where you can convert to LAB if you want.

If the three RGB channels are scaled differently, then the RGB channels in the reconstructed image will reflect this. (Note 3.)

I've actually looked up some interesting papers by Hiragawa and Wolfe
http://www.accidentalmark.com/research/ ... FA_TIP.pdf
http://www.accidentalmark.com/research/ ... A_ICIP.pdf
http://www.accidentalmark.com/research/ ... A_ICIP.pdf
+ Patent WO/2008/067472 (which I can't link to as it exceeds the three links per post limit)
which deal with panochromatic colour filter arrays (having more than one primary colour, as opposed to RGB Bayer) and they show quite convincingly (both through theory and practical comparison) that they are superior to convention RGB Bayer. They more faithfully reconstruct the signal, have less artifacts, and are more robust to noise. I.e., if there is CA in the actual projected image, then they would be able to reconstruct the CA better than RGB Bayer. If you can reconstruct the CA better, you can then invert its effect better.

Now you could easily apply this type of analysis to a two-layer sensor with a suitable panchromatic array, but because you have more information, it must automatically be at least as good as a single layer panchromatic sensor array (in terms of demosaicing reconstruction fidelity), hence be better than an RGB Bayer, irrespective of CA.

The basic point is that (panchromatic) complementary colour arrays are not inherently inferior to primary colour arrays for spectral or spatial resolution, if properly designed. My original suggestion of (R+G) over B and (G+B) over R two-layer sensor may not be precisely optimum but looking at the colour arrays that Hiragawa and Wolfe have come up, they don't look too far off. With a bit of optimisation of the two layers (the lower layer being the complement of the upper layer), I would expect that performance very close to a true RGB per pixel array can be obtained. I may actually suggest this as a signal processing project for a masters.

Cheers,
Daniel.

Note 1: The two-layer sensor, with lower layer being the complement of the upper layer, has a nice feature. Adding upper and lower pixels will give you R+G+B, you automatically get a panchromatic monochrome sensor without the fuss of interpolation or demosaicing . Foveon would be the same.

Note 2: Even with this crudest interpolation and demosaicing algorithm, the spatial resolution of the green channel reconstruction depends on the red and blue sampling frequency, which in this case is the same as for the the green pixels of an RGB Bayer, hence the spatial resolution of the green channel of this two-layer sensor cannot be any worse than the RGB Bayer sensor. Since the R and B channels of the two-layer sensor has twice the density as the RGB Bayer, the reconstructed R and B channels will have as 40% better linear resolution compared to RGB Bayer. Hence, taking this all together, this two-layer sensor should be better able to resolve the scaling of the R G and B channels due to Lateral CA.

Note 3. To see this, you just have to use the (mathematical) linearity of the (very) simple reconstruction algorithm I outline. Imagine, instead of capturing the whole image at once, we first project the R channel image on the sensor and reconstruct the image. We then do the G and B channels similarly. If each of the individual R, G and B images were of different scales, their separately reconstructed images will also be reconstructed at the same scale. Now, because of linearity of the reconstruction, if we were to add these three reconstructed images together, this final image will be the same as if all three channels were exposed to the sensor at once. Working backwards, we see that the reconstructed image, that we obtain by normally exposing the sensor to the projected image, will have the same R, G and B scale factors applied to the relevant channels as if we recorded the three separate images using only a single colour each time.

[agorabasta]

Agorabasta is pointing out that if R and B pixels, in a second layer, are used in any way to derive luminance values then a perfectly apochromatic lens would be needed.

Basically, you are right. The most important thing is to register the green channel with a dedicated sensel array, without having to separate the green info from any composite data with the help of red&blue.

You see, if we gain sensitivity with white pixels, we lose resolution due to CA - nothing comes for free. If we use any kind of mixed-colour sensels, we have exactly the same problem. (Foveon is not of that kind - they have three separate readouts per pixel.)

We could actually win even more resolution from poor lenses with huge lateral CA if more than three primaries were used, i.e. dividing the spectrum into finer bands. But that would mean further sensitivity loss... And then even further losses would be required if we use some sandwich design to fight longitudinal CA.

All comes at a price in our physical reality...

[agorabasta]

[Step 2. Derive G channel from (R+G)-R and (B+G)-B.

Great proposal
You kill your S/N ratio right away. Your derived G's would have about sqrt(3) more noise than Bayer from just the photon noise. Then you pile up systematic errors due to inequality of sensitivities of R and B in pure channels and in the composite channels respectively. Then the two already crippled noisy G's have to be somehow put together - but due to inherited inequalities and high noise levels that won't be possible without some serious convolution...

Hope you get the picture already...

[dkloi]

[Step 2. Derive G channel from (R+G)-R and (B+G)-B.

Great proposal
You kill your S/N ratio right away. Your derived G's would have about sqrt(3) more noise than Bayer from just the photon noise. Then you pile up systematic errors due to inequality of sensitivities of R and B in pure channels and in the composite channels respectively. Then the two already crippled noisy G's have to be somehow put together - but due to inherited inequalities and high noise levels that won't be possible without some serious convolution...

Hope you get the picture already...

It's not the total noise which matters: there's signal too you have to take into account. Signal goes up (factor of 3 compared to Bayer) since all light is turned into photo-current rather than 1/3rd.

Simple example is to work out the R+G+B signal, if wanted to do some monochrome photography. Let the true signal in R+G+B be N. If we had a true monochrome pixel then the standard deviation would be Sqrt(N) and the signal to noise ratio would be N/Sqrt(N)=Sqrt(N). If we were to imagine a pure green pixel (say one site of an RGB Bayer array) from which you are trying to estimate luminance, the signal to noise would be Sqrt(N/3) approximately. Let us work out what the noise looks like for a two-layer sensor. First a top layer (R+G) sub-pixel. The average signal it will get is 2N/3, standard deviation Sqrt(2N/3). The lower sub-pixel B would have average signal N/3 and standard deviation Sqrt(N/3). Adding (R+G)+B gives average signal N as expected and standard deviation Sqrt(N), so the total signal to noise is still Sqrt(N). Hence even though we've added two signals with more noise than the RGB Bayer G pixel, the signal to noise is still better. This two-layer sensor would give you a true monochrome output with the same shot noise as a single layer monochrome sensor, but still gives you colour output should you desire, but with less complexity than a three-layer sensor.

The analysis of generating the G channel is more complicated because of the interplay between spatial sampling density and reconstruction accuracy, and interpolation. When taking into account the interpolation steps, because this is basically an averaging routine, the signal to noise (for the points at which subtraction of channels is performed) is better than your simple analysis would have it. For instance, if we're talking about a (R+G) sub-pixel surrounded by 4 R sub-pixels (left-right-up-down) in the lower level, then the simplest interpolation would be to take the un-weighted mean of the 4 R sub-pixels (bi-linear interpolation). This means that the standard deviation in value of the interpolated R pixel just halved (Sqrt(1/4)). Since the R and B samples are twice as dense as for RGB Bayer, you can use a larger reconstruction kernel and since you are then doing a (weighted) average over more pixels, this drives down noise even further in the interpolated channel. Since we also have information about (R or B) plus G at each site, this has to be taken into account in the reconstruction when comparing with RGB Bayer which has information on G at half the sampling density. For an apples to apples comparison to RGB Bayer, you have to take into account the trade-off between spatial, spectral reconstruction accuracy and the affect noise has on this.

So the noise analysis is not so trivial and I wouldn't be so quick to dismiss it as wholly disastrous given the extra samples afforded by the two layers.

Systematic errors in sensitivities are routinely compensated in all devices so that's not a dramatic issue.

If you use the spatio-spectro optimised CFAs, simple linear reconstruction techniques, which are robust to noise, can be applied. Non-linear RGB Bayer demosaicing algorithms typically aren't as robust. As a bonus, these CFAs are inherently resistant to aliasing (very little cross-contamination between spatial and spectral channels) hence you can dispense with OLPFs.

Is the in-principle ability of the two-layer sensor to reconstruct images, irrespective of CA, still an issue for you? Or is now the crux of your argument noise robustness? Fine if you have your doubts, I won't try to convince you further.

From my research into demosaicing, I'm quite optimistic about these spatio-spectral optimised CFAs and will have to play around with them, first for single layer CFAs (where there is quite good evidence that they outperform conventional approaches to CFA design and reconstruction) and then combine them with two-layer sensors. Adding the second layer can only improve the performance, the question will be to what extent, as well as extending the signal analysis techniques developed for single layer CFAs.

Cheers,
Daniel.

[agorabasta]

Step 2. Derive G channel from (R+G)-R and (B+G)-B.

Great proposal

Is the in-principle ability of the two-layer sensor to reconstruct images, irrespective of CA, still an issue for you?

The 'principle' is no problem obviously.

Yet, you simply do not have neither the 'R' nor 'B' co-posited in your example - those values have to be interpolated. Hence those R&B have no meaningful 1px freq components, only something below 3px turns meaningful. But the noise in those interpolated R&B still has 1px spatial components.
So you can't restore your G's without losses - just as expected from basic principles (trade something valuable for a value; like 'energy vs time' or 'momentum vs position' ...)

All in all, just as said before - you have to trade sensitivity (the S/N) against resolution.
And your particular proposal is worse than Bayer in every aspect. If you don't trust me - it's up to you to try and prove the opposite (shameful failure at every attempt guaranteed).

[Dulaney Ward]

I'm finding it much more difficult than some of you to dismiss Gustav's post, claiming to know that the A500 & the A550 would have Exmor R sensors.

I follow what you are saying--that reliable researchers have failed to turn up any indication of patents for any such sensors, and that an Australian Sony official has denied that any Exmor R would be in a Sony DSLR in September--although, in the latter case, after all, his division had just screwed up royally and he was in full cover-up mode.

Gustav, on the other hand, is a superb nature photographer, taking wonderful shots of kingfishers from blinds (hence his nic), and seems to be a very sober & reliable person--who happened to have predicted the appearance of an inexpensive Sony FF. Moreover, if memory serves, he has worked for Sony,

I wouldn't bet too much against him.

And then there is this, from the editor, Mike, in Sunday's The Online Photographer (August 9):

"But just for the record—in Nostradamus mode, and we all know how painful that can be—I'll venture the guess that the biggest news this coming Fall will be from Sony, and will concern DSLRs. Dum-de-dum. I'm just sayin'."

Dulaney

[agorabasta]

My guess is neither of leaked model no's holds any surprises.

If there really is something big to happen this year, that mysterious something is quite leak-proof for now...

[kingfisher]

about a few weeks we now more

but what i now

both camera`s have EXMOR R and live view
and i think the live-view is more developed then the A230 A330 and A380
no video on board

greetings
gustav

[Birma]

Thanks Gustav . Shame about the video, but it still sounds like an exciting release.

[aster]

Hello Gustav,

You certainly know something about the behind-the-scenes evolutions of the emerging cameras...How about other models? Any good and sound 'leaks' we can depend on coming from your sources?

Thanks
Yildiz

[kingfisher]

Thanks Gustav . Shame about the video, but it still sounds like an exciting release

hi birma

yes it shall be a very exciting release , and personal i am very happy that we dont have video on board

if i want to make a movie , i buy a filmcamera
personal i dont need it , a lot off friends , has also LIVEVIEW , but when i ask , how much did you used LIVEVIEW they say not much , sometimes , it used a lot of power
i think it`s the same as video , in the beginning you try it sometime`s , later not much anymore

[You certainly know something about the behind-the-scenes evolutions of the emerging cameras...How about other models? Any good and sound 'leaks' we can depend on coming from your sources?/quote]
i now how much MP have the A500 and A550 ,
and i think the new A700 , shall be announced around the PMA en/or PHOTOKINA
so that is around april next year

greeetings

gustav

[alphaomega]

So the A500/550 will have new Exmor R sensors and improved LV. That sounds good, but I have already purchased a new A700 at £499.99 plus PP and came across an entry on DPReview quoting this group test http://www.ephotozine.com/article/DSLR-group-test-11857 of 50D, D300, E30, K7 and A700 and the overall winner? Sony A700 including best noise performance, so they must have used a v.4 model. On going through the features again and the quality of images I have achieved I am actually very happy with my A700 so a replacement at £500.00 looks to me like a steal. (Old model has gone to my son). The cheapest A380 I have seen came in at £560 with the new 18-55. With the limited A380 functionality, the A700 at £500 is a real bargain. I hope that Gustav is right that a worthy A700 replacement is due next year. I may purchase one when the price has just about halved compared with launch price. I would consider an A350 at around £345 better value than the A380 at its current price level.
I also purchased a Tamron 10-24 mm zoom. Only time for a few indoor test shots with flash. Looks OK to me for my use with better corner sharpness than the Sigma 10-20 and straight verticals.

[aster]

and i think the new A700 , shall be announced around the PMA en/or PHOTOKINA
so that is around april next year

Hello Gustav,

Thank you for that little 'light' of information ! (Though, next year's April does sound pretty late for me)

Yildiz

[Nico VG]

and i think the new A700 , shall be announced around the PMA en/or PHOTOKINA
so that is around april next year

PMA 2010 will be held from 21 - 23 February in the Disneyworld convention center (Anaheim)

Photokina will be held from Tuesday, September 21 to Sunday, September 26, 2010

So when can we expect the Alpha 700 successor.

In February 2010 or in September 2010 ?????

regards

Nico VG