Maybe DNG isn't so bad afterall

[bfitzgerald]

I never much liked the idea of Adobe running the show with DNG, but since they have applied to have it adopted to ISO standards (I would think it will get that), I am a bit more comfortable with it.

I never bothered with the 5d raw files, converting to DNG, simply because the size isn't really that big anyway, being a 6mp camera. However, playing around with some A900 raw files, I converted some to DNG, compressed, and with original sizes or 35mb+, the DNG compressed ones were vastly smaller, some just over 18mb. Might be a logical space saving solution for A900 users to convert to DNG.

[David Kilpatrick]

It also sort of future-proofs the raw archive. I can see the point. I will have to test my processed A900 files to see if ACR/LRoom still retain the settings, in the DNG made from them after being processed. One reason I value the original raws is that all my corrections are saved alongside the file.

David

[01af]

I will have to test my processed A900 files to see if ACR/LRoom still retain the settings, in the DNG made from them after being processed. One reason I value the original raws is that all my corrections are saved alongside the file.

So they are in the DNG files---provided the XMP files (which store the ACR settings, among other metadata) are there alongside the raw files at DNG conversion time. The DNG files don't use XMP sidecar files but store the metadata including the ACR settings 'under the hood,' just like JPEG or TIFF files would. Of course, if XMP sidecar files are present at DNG conversion time, their contents will get read and digested and thus, included in the DNG files.

-- Olaf

[Mike-Photos]

When I first moved to Bibble, I was shocked to find that they didn't support DNG.
Then I found out that DNG is not really what I thought it was. It's not some kind of magic bullet that removes the need for camera-specific versions. Each camera format still requires specific support even in DNG format. So, the DNG converter has to support the camera, and even the RAW software has to sometimes support the camera.
This being the case, what is the real archival value of DNG? It doesn't make it any easier for RAW software to support a specific camera.
The DNG compression, on the other hand, is of course a plus, so do we lose anything by converting to DNG?
I think we may. If each camera format requires specific support, it can easily be that the original RAW files can contain information that doesn't get transferred into DNG format, because the converter software doesn't know what this information is. This being the case, I would say that you risk losing data by converting to DNG and throwing the RAW away. It could be that future software from the camera manufacturer, or a new generation of third-party RAW software, can extract more out of the native RAW file than the converted DNG file.
Any thoughts?

[01af]

[DNG is] not some kind of magic bullet that removes the need for camera-specific versions.

But it is, and it does!

Each camera format still requires specific support even in DNG format.

Definitely not.

So, the DNG converter has to support the camera ...

Yes sure, the DNG converter ... but not the DNG file format.

... what is the real archival value of DNG?

Even ACR 2.4, originally released almost four years ago, still can read and process DNG files created from the latest digital cameras' raw files tpday.

It doesn't make it any easier for raw software to support a specific camera.

Of course it does! ACR 2.4 is raw software, and through DNG it does support cameras that where unthinkable back when it was released. The DNG format moves the hassles of keeping up with the specific cameras' proprietary raw formats from the image-processing software into the DNG converter---which is (1) free and (2) could be re-programmed by anybody who is able to read a formal file format specification.

The DNG compression, on the other hand, is of course a plus, so do we lose anything by converting to DNG?

In most cases, you'll lose the ability to use the camera vendor's original raw converter.

It could be that future software from the camera manufacturer, or a new generation of third-party raw software, can extract more out of the native raw file than the converted DNG file.

This is possible indeed ... but only when the camera manufacturer keeps refusing to support the DNG format, not because there was something in the original raw files which wasn't in the DNG files. However I guess independent raw converters (which do support DNG) are more likely to extract more out of the image files in the future than proprietary raw converters (which usually don't support DNG).

-- Olaf

[Mike-Photos]

Hi Olaf,

RawShooter died eventually precisely because of the DNG issue. Although it supported DNG files, when the Sony ARW format replaced the Minolta format, then, although you could convert the Sony files to DNG, RawShooter could not read them. I know, I tried myself.

In addition, when new Sony cameras arrive, such as the A900, everyone is always waiting for each RAW program to support its format. If DNG is fully accurate each time, surely no-one would be so excited as and when each software product supports the new cameras? It would be a non-issue. In fact, all any RAW software product has to do then is support DNG, and they would not have to build any native camera support at all. I'm sure we would have seen start-up or freeware products built on this basis if it was so simple.

I'm not sure about ACR, and I'm sure you are correct, but maybe Adobe has some other mechanism for keeping DNG files current.

I don't know the answers, or where I am getting confused, but confused I am!

[01af]

... although you could convert the Sony files to DNG, RawShooter could not read them.

Well ... then RawShooter obviously didn't support the DNG format properly. As I said---ACR 2.4 (and all ACR versions after that, actually) proves that old raw processing software, through DNG, can handle new cameras' raw files. However, many software products that pretend to support DNG actually do so only half-heartedly.

If DNG is fully accurate each time, surely no-one would be so excited as and when each software product supports the new cameras?

You're still confusing the DNG Converter (which converts proprietary raw files to DNG raw files) with raw converters (which convert raw files into images). Of course, you do need a new DNG converter when a new camera comes out ... and Adobe DNG Converter always comes along with the latest Camera Raw and Lightroom versions. The point is: the latest DNG files can be processed with four-year-old raw converters---if they do support the DNG format properly.

-- Olaf

[Mike-Photos]

... although you could convert the Sony files to DNG, RawShooter could not read them.

Well ... then RawShooter obviously didn't support the DNG format properly. As I said---ACR 2.4 (and all ACR versions after that, actually) proves that old raw processing software, through DNG, can handle new cameras' raw files. However, many software products that pretend to support DNG actually do so only half-heartedly.

Are there any other products besides ACR and LR that fully support DNG to this level? I've seen no claims by any manufacturer other than Adobe, and I would have imagined that they would have made that a key selling feature.

If DNG is fully accurate each time, surely no-one would be so excited as and when each software product supports the new cameras?

You're still confusing the DNG Converter (which converts proprietary raw files to DNG raw files) with raw converters (which convert raw files into images). Of course, you do need a new DNG converter when a new camera comes out ... and Adobe DNG Converter always comes along with the latest Camera Raw and Lightroom versions. The point is: the latest DNG files can be processed with four-year-old raw converters---if they do support the DNG format properly.

I do understand this, but my question was that if this is the case, why do smaller and newer RAW software developers bother to support camera-specific formats at all? Since Adobe supply the DNG converter for free, and since they are almost always the first company to support most cameras, All you need to do as a software developer is support the one format. It's notoriously slow, tedious, and expensive to decode each camera manufacturer's RAW formats, and users are always desperate for new versions of software that support the new cameras, so why don't the software developers just use DNG?

More questions:

1) Does the converted DNG file not embody Adobe's particular interpretation of the RAW file? We've seen how different RAW software produces different results from the same RAW file. If you use the DNG file, are you not stuck with Adobe's method? If this is the case, what you are archiving is not optimal.

2) What if Adobe themselves improve their rendering of a particular format in a future release? To get this improved rendering, wouldn't you have to re-convert the original RAW file? If this is the case, then you have to keep your RAW files anyway, once again making little case for using DNG as an archival format.

3) There's an option when converting to DNG to embed the original RAW file as well. Surely this points to the fact that the DNG file is not as raw as the RAW file itself?

Make no mistake, I'd love there to be a way to properly archive and compress all my older and my current RAW files in such a way as to guarantee that I can reprocess any of these files for eternity. But I'm not 100% certain in the case of DNG that what you are keeping is actually this.

[01af]

... my question was that if this is the case, why do smaller and newer raw software developers bother to support camera-specific formats at all?

Because people don't want to be forced into using the DNG format.

Since Adobe supply the DNG converter for free, and since they are almost always the first company to support most cameras, all you need to do as a software developer is support the one format.

Basically, yes.

It's notoriously slow, tedious, and expensive to decode each camera manufacturer's raw formats ...

Actually, it's not. Many raw converter developers simply rely on dcraw.

Does the converted DNG file not embody Adobe's particular interpretation of the raw file?

No. DNG raw is just as raw as proprietary raw. There is nothing inside a DNG file which got 'interpreted.'

If you use the DNG file, are you not stuck with Adobe's method?

No. You are still confusing the DNG Converter (which converts proprietary raw files to DNG raw files) with raw converters (which convert raw files into images).

What if Adobe themselves improve their rendering of a particular format in a future release?

Actually, they're doing this all the time. Virtually each ACR version does include a few improvements over the previous versions ... or at least, some do.

To get this improved rendering, wouldn't you have to re-convert the original raw file?

No.

There's an option when converting to DNG to embed the original raw file as well. Surely this points to the fact that the DNG file is not as raw as the raw file itself?

No. It's there to (1) maintain the ability to use the original manufacturer's raw converter (which usually does not support DNG), and (2) put the minds of notoriously paranoid users at rest. However if you really want to archive both raw formats, DNG and proprietary, then I'd recommend to archive the files separately rather than embedding one into the other ... but that's just a matter of personal preference.

But I'm not 100 % certain in the case of DNG that what you are keeping is actually this.

Currently, it hardly makes any difference whether you're archiving the proprietary raw files or the DNG files created from those. As of now, the danger of older raw file formats becoming obsolete is purely academical. Once it actually happens one fine day, you can always catch up on converting your archive to DNG---which then is going to be a tedious work but doable.

The biggest real-world advantage of the DNG format right now is the lossless compression which will significantly reduce the memory space required for your archive, compared to most proprietary raw formats.

-- Olaf

[Mike-Photos]

Hi Olaf,

Thanks for taking the time, this is an interesting an important discussion!

I went Googling a bit, and found this one:

http://www.apertureprofessional.com/sho ... hp?t=11870" onclick="window.open(this.href);return false;

Quotes from the start and end of the link:

"DNG hasn’t been as much of a global salvation as planned, but it is a very handy file format. The standard’s not quite as “open” as promised and not every imaging program can work with a DNG file. In Aperture 1.5 for example it was only possible to import a DNG file that belonged to a camera format that could already be converted. Handy if someone had sent you a DNG file from something like a D2X, not so handy if you were getting a file off of a non-supported Hasselblad back.

Aperture 2 has embraced the DNG file format and that’s big news for photographers for a few reasons. The most obvious reason is that it allows Aperture to work with a vastly greater range of images. If a file format’s not supported by Aperture but can be converted by Adobe DNG converter, it’s now fair game in Aperture. This is great for people with legacy images and files from DNG based cameras.
.
.
.

Once files are imported as DNG, a new option will be available in the “RAW Fine Tuning” section of the Adjustment inspector, which indicates that the file is being adjusted using the DNG decoder. This box reads “2.0 DNG” and changes to the image using the 2.0 DNG converter are made based on the DNG specification of the file.

If the DNG is for a file that Aperture can already support (or if the file is viewed after support for a format is added) there will be the standard choice of processing the image using the Aperture 2.0 conversion, by selecting “2.0” from the drop down menu. The idea here is that the specific Aperture conversion does a better camera-specific job than the generic DNG does. (So in short, use the 2.0 choice if it’s available, rather than the 2.0 DNG, which is only the default choice if the camera format is not supported by Aperture directly.)"

If what you are saying is 100% accurate, that the files are essentially identical, how can this be?

[01af]

I suggest exploring this site and its links:

http://www.barrypearson.co.uk/articles/dng/index.htm" onclick="window.open(this.href);return false;

-- Olaf

[Mike-Photos]

I suggest exploring this site and its links:

http://www.barrypearson.co.uk/articles/dng/index.htm" onclick="window.open(this.href);return false;

-- Olaf

Barry Pearson is a somewhat fanatical DNG evangelist, who has been banned on many forums for his extreme views. I've had a few conversations on forums with him myself.

I appreciate what you are saying about DNG, but the fact is that it's not that clear cut, and many experts in the field feel that using DNG for archiving is not the right way to go.
From the last bit of the discussion about Aperture, and even the fact that Aperture distinguishes between native RAW support for a camera as opposed to indirect support via DNG, I think it's clear that all is not well as regards 100% compatibility between native RAW and it's DNG.

From my point of view, I'll live with the extra space usage in the meantime. If and when:
* Adobe and/or the experts clearly and unambiguously state that DNG converted RAW is identical to native RAW
* Native RAW files and their DNG equivalents render identically in non-Adobe products
* I can see the same DNG image looking different in each RAW software product, as the native RAWs are
Then I'll switch to DNG and batch convert my old files. Until then, DNG is an Adobe-specific archival and compression format, not a general one.

Of course, if anyone feels comfortable committing to Adobe alone for RAW / DNG processing, then DNG makes admirable sense.

[UrsaMajor]

RawShooter died eventually precisely because of the DNG issue. Although it supported DNG files, when the Sony ARW format replaced the Minolta format, then, although you could convert the Sony files to DNG, RawShooter could not read them. I know, I tried myself.

It is not really of consequence to this discussion, but RawShooter did not die because of the DNG issue. They had some teething problems with it, but I'm confident that these would have been worked out eventually. What killed RawShooter was that Adobe bought the company and hired the principals to work on ACR.

With best wishes,
- Tom -

[Dr. Harout]

Tom, welcome to the forum.
And presuming from your nickname you probably love astrophotography.