Difference between RAW and cRAW?

[maratrs13]

What is the difference between RAW and cRAW other than size? Is there any quality downgrade in cRAW compared to RAW? What are the benefits of shooting RAW if the answer for the previous question is 'No'? Writing (processing) speed?

A bit off-topic: As someone had already noticed in another thread I also noticed difference in reported images to be taken with an empty card (increased cRAW/RAW+JPEG) when I updated the firmware to v4. That's nice but there should be definitely a sacrifice down the road. Is it quality of RAW files or JPEGs or both? Or is it just improved processing/compression?

Too many questions... Thanks for patience!

[Dr. Harout]

There has been much discussion on this topic in this forum as well as other forums too with no final result. Some say there is while some insist the opposite.

[Dr. Harout]

Look here http://www.photoclubalpha.com/forum/vie ... f=32&t=246

[David Kilpatrick]

Apparently the Sony raw format arranges strings of R, G and B data grouped by line (concatenated) instead of interspersed in logical Bayer pattern readout order. This places similar or identical values adjacent to each other, which enables a simple lossless compression algorithm to work efficiently on the file. Similar compression does not work well with normal Bayer raw data order:

RGRGRGRGRGR
GBGBGBGBGBG

but Sony's order is like

RRRRRRRRRRRR
GGGGGGGGGG
BBBBBBBBBBBB
GGGGGGGGGG

(reinterpreted by the raw conversion of course to created RGB 24-bit pixel locations, aka three layer channels of 8-bit R, G, B)

I have never examined or analysed a raw or a craw file, and I don't have the tools or know-how to do so. But raw file experts who have analysed the files explained this in the first few days of the A700's existence. It makes sense.

David

[Dr. Harout]

Whenever i explain to my patients about their illness, I try to do that in a mundane way, in a very simple way, so they 'get the picture' and not in Latin or in big and long medical terminology which is absolutely Chinese to them as is those explanations about cRAW and associates to me.
Could anyone put this in a very simple language? Not everyone here does understand physics or mathematics, and they are not supposed to.
I understand nothing of those but I'll try to explain this as I see it (if I am wrong, try not to hang me, nor "geesh" me, nor "LOL" me, as I'm doing it in good intentions. One of the easiest and cheapest things in life is to criticize.)
Suppose you are in a market. The owner has found a way to arrange 12 kg of oranges in a box, where normally 9 kg would fit. The oranges are arranged a bit tight (or K-bit tight if you want ) invading more air space than in standard arrangement. Round oranges become a bit squared or "rectangled"... And maybe 1 or 2 oranges get squeezed much to ooze a bit in the box. The question is: are you loosing from the 12kg oranges? The answer is no. Did you loose something? the answer is yes, it's the air and a little bit (not K-bit) from the orange peel's liquid. Would you have the same orange juice from the tight box and the standard one? The answer is yes. But if you are a taster you might feel that peel's sour taste mixed in the wonderful taste of the orange juice, but that my friends is one in a ten million who might feel it.
So my 2 lumas (Armenian cents) in this topic. Feel free to use RAW or cRAW as you like. If you want to get deep in the bits and bytes, RGB arrangements or Bayer pattern, do it. I'm interested in what I see as a shot, what I perceive in it.
Hope I've been helpful.

[Dr. Harout]

Haha, your Highness Don Sonolta of Illinois, congratulations with your new degree.

[pakodominguez]

I thought that, other the algorithm, RAW was 14 bits and cRAW was 12 bits -I don't recall anymore where I did read about that...

[pakodominguez]

a700 is 12 bits only

Read the links I presented and some are talking Craw may be 11bit, but it's not conclusive as compared to raw either.

-Sonolta

you kidding, right?
reading ALL THE LINKS you post?
impossible!
:-/
I wish I had the time...

[David Kilpatrick]

a700 is 12 bits only

Read the links I presented and some are talking Craw may only be 11bit in practice, but it's not conclusive as compared to raw either.

-Sonolta

The A700 is 12-bit, but the cRAW files are a compressed 8-bit format - they express 12-bit data, but in an 8-bit form (by measuring differences in values, not the values themselves). This has been confirmed by some of the guys who write raw converters and it shows up as a file information figure in Raw Photo Processor. I guess the A900 cRAW format will be similar:

Here is some info from an uncompressed A900 .ARW2 file:

FileSize: 36 MB
FileModifyDate: 2008:09:30 11:57:46+01:00
FileType: ARW
MIMEType: image/x-raw
ExifByteOrder: Little-endian (Intel, II)
ImageDescription: SONY DSC
Orientation: Rotate 90 CW
Software: DSLR-A900 v1.00
BitsPerSample: 12
Compression: Sony ARW Compressed
PhotometricInterpretation: Color Filter Array
SamplesPerPixel: 1
PlanarConfiguration: Chunky

and here is the same info block from a cRAW A900 file on the same camera:

FileSize: 24 MB
FileModifyDate: 2008:09:22 15:51:02+01:00
FileType: ARW
MIMEType: image/x-raw
ExifByteOrder: Little-endian (Intel, II)
ImageDescription: SONY DSC
Orientation: Horizontal (normal)
Software: DSLR-A900 v1.00
BitsPerSample: 8
Compression: Sony ARW Compressed
PhotometricInterpretation: Color Filter Array
SamplesPerPixel: 1
PlanarConfiguration: Chunky

As you will seem, the bits are reported as being only 8 per sample in the cRAW format and 12 per sample in the RAW format, and both are described as being compressed. Raw Photo Processor is an amazing tool for looking at file data. Here is the complete information panel:

DateTimeOriginal: 2008:09:22 15:51:01
ISO: 200
Aperture: 8.0
ShutterSpeed: 1/320
FocalLength: 28.0 mm
Make: SONY
Model: DSLR-A900
ImageSize: 6048x4032
ExposureCompensation: 0
ExifToolVersion: 7.43
FileName: DSC00265.ARW
Directory: /Volumes/MediaCenter/10080922
FileSize: 24 MB
FileModifyDate: 2008:09:22 15:51:02+01:00
FileType: ARW
MIMEType: image/x-raw
ExifByteOrder: Little-endian (Intel, II)
ImageDescription: SONY DSC
Orientation: Horizontal (normal)
Software: DSLR-A900 v1.00
BitsPerSample: 8
Compression: Sony ARW Compressed
PhotometricInterpretation: Color Filter Array
SamplesPerPixel: 1
PlanarConfiguration: Chunky
XResolution: 350
YResolution: 350
ResolutionUnit: inches
CFARepeatPatternDim: 2 2
CFAPattern2: 0 1 1 2
StripOffsets: 622592
RowsPerStrip: 4048
StripByteCounts: 24611840
OtherImageStart: 155683
OtherImageLength: 453177
ExposureTime: 1/320
FNumber: 8.0
ExposureProgram: Aperture-priority AE
ISO: 200
ExifVersion: 0221
DateTimeOriginal: 2008:09:22 15:51:01
CreateDate: 2008:09:22 15:51:01
ComponentsConfiguration: YCbCr
CompressedBitsPerPixel: 8
BrightnessValue: 8.37
ExposureCompensation: 0
MaxApertureValue: 2.8
MeteringMode: Multi-segment
LightSource: Unknown
Flash: Off
FocalLength: 28.0 mm
Quality: Compressed RAW
FlashExposureComp: 0
Teleconverter: None
ColorReproduction: Standard
SceneMode: Standard
DynamicRangeOptimizer: Off
ImageStabilization: On
LensType: Minolta AF 28-75mm F2.8 (D)
UserComment:
FlashpixVersion: 0100
ColorSpace: sRGB
ExifImageWidth: 6080
ExifImageHeight: 4048
InteropIndex: R98 - DCF basic file (sRGB)
InteropVersion: 0100
FileSource: Digital Camera
SceneType: Directly photographed
CustomRendered: Normal
ExposureMode: Auto
WhiteBalance: Auto
FocalLengthIn35mmFormat: 28 mm
SceneCaptureType: Standard
SR2SubIFDOffset: 36400
SR2SubIFDLength: 53248
SR2SubIFDKey: 1144201745
FirmwareID: 21870002
SensorHeight: 4048
SensorWidth: 6080
ImageHeight: 4032
ImageWidth: 6048
RawDepth: 12
BitDepth: 12
StorageMethod: Linear
BayerPattern: Unknown (0)
WBScale: 2 2 2 2
WBLevels: 713 256 256 313
Saturation: 0
Contrast: 0
Sharpness: 0
WBMode: Auto
ProgramMode: Unknown (128)
ISOSetting: 6153909962
ColorMode: Unknown (0x38)
ColorFilter: 75
BWFilter: 0
ZoneMatching: Unknown (212)
Hue: -1
ColorTemperature: 21400
SubfileType: Reduced-resolution image
Make: SONY
Model: DSLR-A900
ModifyDate: 2008:09:22 15:51:01
ThumbnailOffset: 32770
ThumbnailLength: 3438
Aperture: 8.0
CFAPattern: [Red,Green][Green,Blue]
ImageSize: 6048x4032
LensID: Minolta AF 28-75mm F2.8 (D)
OtherImage: (Binary data 453177 bytes, use -b option to extract)
ScaleFactor35efl: 1.0
ShutterSpeed: 1/320
ThumbnailImage: (Binary data 3438 bytes, use -b option to extract)
CircleOfConfusion: 0.030 mm
FOV: 65.5 deg
FocalLength35efl: 28.0 mm (35 mm equivalent: 28.0 mm)
HyperfocalDistance: 3.26 m
LightValue: 13.3

David

[David Kilpatrick]

Everyone has known about the 8 bit compression format since the beginning!

-Sonolta

Yup, as Sony documented it themselves without revealing how it was done, but it's interesting to see the full file info from the A900, and to note that the cRAW format in the 900 is identical to the A700.

In practical terms you will get about 50% more cRAW files per card than RAW, and copy or write/read the images 50% faster. Last week I was shooting cRAW only. This week I've been shooting RAW+JPEG. I'm not seeing any benefits either in assessing the image or in final quality, so I will go back to cRAW permanently on the Alpha 900. I may also go up to 8GB cards.

Would you say, Don, that it would make more sense to use uncompressed raw for high ISO shots, or to use compressed raw? What happens if you get a hot or dead pixel - or the many salt and pepper hi/lo pixels associated with pushing the gain for high ISO - and the cRAW algorithm then hits these values as part of a pixel subgroup? I'm not really mathematically minded enough to work out the implications of noise on the cRAW scheme. My gut feeling is that anomalous high or low pixel values, associated with high ISO noise, might degrade the efficiency of cRAW and have the effect of dragging overall noise levels up by their presence. Maybe uncompressed raw is safer with high ISO?

David

[01af]

What happens if you get a hot or dead pixel - or the many salt and pepper hi/lo pixels associated with pushing the gain for high ISO - and the cRAW algorithm then hits these values as part of a pixel subgroup? I'm not really mathematically minded enough to work out the implications of noise on the cRAW scheme. My gut feeling is that anomalous high or low pixel values, associated with high ISO noise, might degrade the efficiency of cRAW and have the effect of dragging overall noise levels up by their presence. Maybe uncompressed raw is safer with high ISO?

I'd say if Raw vs. cRaw makes a difference in the first place then it will be with low-ISO files.

But my bet is it won't make any visible difference at all, no matter what the ISO is. The 12-bit-to-8-bit compression is carried out via a (possibly virtual) look-up table which re-maps the levels in a non-linear way, and in the raw conversion process the compression will get undone before the demosaicing. As far as I understand it, this kind of compression is not affected by the presence of noise. The compression is not lossless ... but you'll lose only data which will get lost in the raw conversion process anyway. In the linear raw file, there are twice as many levels in each f-stop than in the previous one. In the upper f-stops, this leads to ridiculously high numbers of levels per f-stop---much higher than required for a proper RGB image. When applying the gamma correction in order to get an RGB image that looks 'right' to our eyes then many of those superfluous levels are getting lost ... and cRaw throws them away beforehand so the number of remaining levels is about equal (more or less) across all (or most of) the f-stops of the dynamic range.

It helps understanding how cRaw works when looking at how the Leica M8 camera implements its own cRaw format. The sensor's A-to-D converter creates 12-bit values as usual (i. e. values from 0 .. 4,095). These values get multiplied by 16 which spreads them across a 16-bit range (i. e. 0 .. 65,535). Then the square root is calculated from the 16-bit values which maps them in a non-linear way to the final 8-bit range (i. e. 0 .. 255). To restore the 'original' Raw level value, simply square the cRaw value and then divide by 16. The data loss occurs when the square root values get rounded off to integer numbers. However don't be mistaken to believe it was pointless to create 16-bit RGB images from cRaw files. It's not! Due to the complex nature of the demosaicing process, a 16-bit RGB image created from a cRaw file still will contain more information than an 8-bit RGB image, even though the cRaw data was 8 bit only.

The square root function here acts as a transfer function which implements a virtual look-up table in an algorithmically easy way. It spreads the lowest values and compresses the high values. I have no idea how Sony has implemented their cRaw format ... they may use the same, or a similar, transfer function, or they may use a real look-up table. But the basic principle is the same for sure. By the way, Adobe's DNG format provides means to store remapped raw values instead of the actual raw values so it supports, and always has supported, this kind of raw compression.

-- Olaf

[David Kilpatrick]

Olaf, if I understood all that I would have done sciences not arts! I probably do understand it - sounds as if a log/linear transform is involved which makes better use of the redundant integer steps in high bit levels. But I'm almost dysnumerate in the same way my very articulate daughter, who just got accepted as a committee report editorial supervisor for the Scottish Parliament, was diagnosed as dyslexic at 18. I never got diagnosed as anything except 'dopey' because they didn't understand different cognitive abilities back then... it's more the language of math which screws me than the visualisation of mathematical precepts.

David

[01af]

I probably do understand it - sounds as if a log/linear transform is involved which makes better use of the redundant integer steps in high bit levels.

Yes, that's the principle exactly. Rather than using the square root as the transfer function, one could also use a logarithm. I guess Leica chose the square root because it's easier to compute ... and it elegantly matches the requirements. The point in using a transfer function (as opposed to an actual look-up table) is to save resources---i. e. memory space and computing power---in the camera's on-board data processing chain.

-- Olaf

[agorabasta]

Strangely wrong opinions I find here...

Anyway, I repost myself from dpr - http://forums.dpreview.com/forums/read. ... e=29228773 -

cRAW does reduce DR at higher spatial frequencies - means it extends/smooths extreme transitions.

Yet it does not introduce any loss if your image's spatial frequency density distribution falls within the limits introduced by cRAW compression. I.e. if your image does not contain extremely sharp transitions of high magnitude - there's no loss.

In practice, it limits the difference between any two adjacent RAW pixels by 7 bits. And 99.99% of real-world images fall here.

On the other hand, it's pretty easy to see the smoothing effect of cRAW compression trying to shoot your monitor screen, especially if it's lit pure fullscreen green or displays thin horizontal green lines over black background. The RAW then clearly shows black gaps between monitor's green stripes, while cRAW smooths them out considerably.

And quite consistently with its nature, the cRAW compression smoothing artifacts would depend on the exposure - if the histogram maximum falls in the middle then all's sharp; if exposed to the right then there's more smoothing.