Nikon D3X DXOMark Numbers Blow the Sony A900 Away!

[alexramos7]

I think she is photographer and doesn´t videographer... She changes her equipment 3 time each year.

[Greg Beetham]

If you were to take those charts as gospel...then looking at the A100 vs KM5D there is "apparently" about 1/2 stop difference in gain at 100ISO, the KM5D is closer to 200ISO when set at 100ISO according to DxO, I just did some tests both in A mode same settings (100ISO Auto WB) same lens and zoom setting (50mm) same f/stop (f8) at the same subject (concrete pad) at the same distance etc. and both cameras selected the same shutter speed ie. 1/200sec..then I crosschecked with S mode, same everything else and set shutters at 1/200sec and both cameras selected f9 in other words same same.. if one camera was closer too ISO200 and the other wasn't then there would be some difference in exposure surely.
The only difference I could see was in the histograms, the A100's was slightly left of center and the KM5D's was slightly right of center.
BTW what happened to the KM5D's ISO Lo80 in their plot???
Greg

[Greg Beetham]

If I hold both cameras up and look at the screens with the histo's showing there is very little difference on average, though the sunlight was just a little variable, I had too wait for small fast low clouds scudding through quite a bit too get the sun in the clear, and there was also some thin high level cloud but that was almost stationary but it still could have interfered I guess, if I see later that there is no cloud I'll repeat the test and I'll also use a tripod and my Minolta MkIV light meter too make sure the ambient light is the same for all shots.
I took four images with the KM5D and two are a tiny bit right, one bang on middle, and one is a little left, the A100's two images are both a little left. At this stage though I don't have much confidence in DxO.
Greg

[David Kilpatrick]

The problem with reading DxO results is that they are not taking pictures conventionally or even processing the raw file for their measurements. They are exposing the naked sensor to a light source and measuring two points only in its sensitivity range - saturation (all three channels clipped - this is not a very useful measurement, it's better to take the point at which only one channel clips but that would reduce all the d-range figures, all round) - and S/N>0DB, i.e. the darkest level at which any useful image can be extracted above noise. This is how they calculate the ISO (using a formula derived from the clipping value) and the dynamic range (using the total of EV steps between one and the other) and the S/N ratio. They measure all this from raw data value without de-Bayering the raw file at all - they just analyse the pixel bit values.

What DxO does not show is the analog to digital gamma or conversion curve. This is what determines whether the camera gives medium grey a value of 128 (normal), 110, 140 or whatever (when the 12 or 14 bit values in raw are converted to a 256 scale). Secondly, it does not take into account the digital post-processing transfer function either. This is what maps these values to a JPEG, and varies depending on how you set the contrast, picture style etc; the camera histogram is always based on the image AFTER this conversion. So two tone curves have been applied which alter the values between shadow threshold and highlight clipping, but DxO ignores these curves. Raw converters use curves which imitate the ones used in the camera for creating JPEGs (programs like IDC 3 actually have the same curves built in, other programs never match them exactly).

Since we actually judge pictures by the mid-tones, not by the shadow or highlight values, exposure index values (ISO) used on the cameras reflect real-world results, not theory. That's why although DxO says that ISO 100 and 200 on the A900 identical in ISO speed (and neither of them accurate), in practice they behave as you would expect ISO 100 and 200 to behave. ISO 100 uses a flatter A to D curve, ISO 200 a more 'humped' curve.

So, in practice the 'ISO' values reported by DxO have nothing at all to do with the actual EI used by the camera metering, or the density of the result you will get. They disregard all the processes built into the camera and raw conversion software to turn the raw data into a useful image, and you could not set their figures on a hand-held meter and get a correct result. They are 'absolute' measurements which may tell you how well a manufacturer has designed the sensor, but not how well they turn the results into a photographic image.

For what it's worth, I think ISO 12233 is a bad standard, it reverses everything which used to apply to film ISO measurements. dPreview's explanation is not as useful as it could be, as it also omits any reference to gamma (or, in film terms, Contrast Index). You can not state EI without also stating CI.

David

[Greg Beetham]

Ok so what is the KM5D doing then? "technically overexposing the sensor" by 2/3 stop that would mean I'd have more burnt out highlights with that camera than the with the A100, especially around this neck of the woods.
Don I only operate with the facts that I can trust and verify....now I'll have to see if I can rig up a highlight test as well.
BTW I took a shot at Lo80 a few minutes ago (L fine + RAW) and the RAW exif in E6 says ISO = 80.
Greg

[David Kilpatrick]

The problem with reading DxO results is that they are not taking pictures conventionally or even processing the raw file for their measurements.

You and I both know they use prints for most of their analysis! For those who still don't get it read the Westlake quotes above and the DXO text below.

Please tell me how they use prints when the tests are made with no lens fitted to the camera! They make prints from images taken outside the context of their sensor scoring tests, but those tests - the ISO, dynamic range, S/N - are all done without using any kind of photographic target, or any camera lens. Read their website, it explains it all very clearly. They do it this way using a calibrated light source and a Gossen luxmeter, with the camera with its lens removed mounted on a rail to allow it to be slide into position (replacing the Gossen meter).

They do the colour test only, using a lens. This is the only test which resembles the kind of test you or I might do. What I find surprising is that they use a Macbeth 24-patch checker. Even the various camera calibration kits I've looked at, like InCamera, only recommend this as the most basic startpoint. Macbeth themselves make a far more accurate colour target:

http://shop.colourconfidence.com/sectio ... 3god9xldnA" onclick="window.open(this.href);return false;

the Color Checker SG. Why does DxO use the 24-patch checker and not this one, I wonder?

David

[Greg Beetham]

That digital colour checker SG looks interesting, but the price might slow one down a tad..kaf kaf
Good video presentation of how to use them as well.
Greg

[Greg Beetham]

To say I don't understand is correct, you can take that to the bank.
Question: When DxO does a "print" what printing process/printer are they using, and what are they printing? I get the impression it's a theoretical "print" as I can't see any discussion of their printing methods, printer, paper, inks, printer colour profiles, method of measuring the printed output, and with what measuring equipment?
Greg

[Greg Beetham]

Back to the Lo80 for a second, just for interest, Ok the idea is a special jpg tone curve for jpeg's, but what I'm getting at is is the sensor gain actually "set" at ISO80? I just did three dummy exposures out the window each one aimed at the same spot on the ground with the same lighting and the shutter speed dropped with each reseting of the ISO, ie. A mode f5.6 (ISO200 1/60sec), (ISO100 1/30sec), (Lo80 1/25sec). It would seem from that that the camera is actually setting the sensor @ ISO80...agreed?
If so then why did DxO not include that range in the plot? a reasonable question I would think, seeing as how they claim too measure directly off the sensor what the camera is setting as regards actual settings, and I'm still not exactly sure just how they intercept/get the sensor data from the sensor either, and still have a working camera while they are doing it, with only the lens removed.
Greg

ps I caught a preview of your next post while I was submitting this one, I think the key in the sentence is "calculates"

"If a higher-resolution and lower-resolution camera have the same noise levels, when you print their images at the same size the noise from the higher-resolution will be less visually apparent. Recognizing this, DxOMark also calculates a number of its measures based on the appearance of images printed at a standard 8x12 inch output size."

[Greg Beetham]

Yes I've seen that half face graphic in action for real, it sure is an outstanding piece of work alright.
Greg

[stevecim]

manufactures agree with them
-Sonolta

Where does this come from? I've looked on Canon, Nikion and kodak, I did not find any thing supporting Dxomark results?

Just because at some time they purchased dxo test suite, does not mean we know how, where, if or when they use it.

just my 2 sense worth.

[stevecim]

thanks Don.

[Greg Beetham]

Yes, well point taken, Carl Zeiss one would imagine are an optics company that you would not easily pull the wool on, so DxO must have some substance. I'm just not sure where the real world ends and DxO's simulations begin when it comes to camera sensor testing, David K has stated that he's not keen on their methodology at all, so there is a need too get some clarification on their test results, if at all possible.
Greg

[David Kilpatrick]

DO Labs also manufactures a whole range of test targets for lens/sensor resolution etc - which are the only things Carl Zeiss will be bothered about. The DxO Mark sensor evaluation, which we are discussing here, would be of no interest at all to them as it says nothing about lenses. The full range of test targets, lighting rigs, optical benches and studio-lab setup you can get from DO Labs (to use with the full DxO Analyser suite which does far more than the Sensor Mark stuff) is considerable. This is where a manufacturer would go to buy component parts to make their own test lab.

I found that I have a room large enough to fit the entire DxO test gear, my old studio would be just about the exact size they reckon you need. But it would cost several thousand for the installation, not to mention a dedicated PC system and a hefty annual charge for the software, and unless you are an organisation serving clients (testing every single compact camera as they appear, etc) it's overkill. I looked into this some time ago when DO Labs contacted me, as a publisher, to see whether I wanted to buy their turnkey test lab solution. I guess they did not realise I was a one-man publisher and my turnover for the year was about the same as the cost of their system!

But, I reckon that with Imatest targets, off the shelf standard illuminant sources, Manfrotto and Climpex engineering, and some basic calibration tools I could build a test room for £2000 or so. Things like colour accuracy from a Macbeth colour checker are easily mapped, Shirley spent a whole year doing exactly that, measuring Delta-E shifts. Sad thing is that the Mac programs she set up for presenting the results are all defunct now, we would need to buy new Math and Graphing progs or dig out Gnu licensed open source stuff. Here's one of Shirley's several hundred worksheets:

This is her sort of thing, much missed I guess as there was no ultimate commercial use for the skills she acquired, and it's almost 15 years ago now.

The Sensor Mark 'print' thing appears to be a matter of wording. A screen view is a file which can be analysed. A 'normalised' print appears to be a file resampled in size for 8 x 12 inches at 300dpi. This file would be a kind of grey rectangle (pure monochrome) with a pattern of pixel noise. I do not think they would need to make an actual print, after all that introduces a whole stack of variables depending on the imaging method used, and requires elaborate closed-loop calibration. I know of only two processes which use a spectrophotometry closed loop calibration, the HP series of large format printer with installed ceramic calibration target, and the Fujix Pictography machines with a similar reader. I don't think either of these would be suitable for evaluation prints. So I think they are using the word 'print' to mean a resampled ('normalised' in their language) file which is then analysed much the same way the screen (native) resolution raw file is analysed.

Their use of the term 'normalised' will confuse anyone used to the same term in audio signal processing. Very different process from resampling an image for a fixed size.

I may, for interest, get the Sekonic kit for camera ISO calibration for a second time (I have tested this once already, but they have introduced a better target and new software since then).

David

[David Kilpatrick]

DxO does a lot much more than just test targets...

But those aspects are more like Apical's business. Not what we are talking about here - embedded DSP firmware and stuff. Entirely different aspect of DO Labs business and not in any way connected with sensor testing, lens testing or anything like that. Therefore, irrelevant. Rolls Royce and Saab make aerospace technology, but there would be little point in mentioning it when discussing Rolls Royce and Saab cars.

David