FILM - lossless raw to dng converter

[martinus]

@rvietor

So what is a reliable medium to store digital data? The ones I know of all seem to have rather limited lifespans, and every copy cycle risks introducing errors.

So far, the best option for images and text seems to be print (if well executed)… But that takes up a lot of space.

Using multiple backups and media. I have 2 hard-drives, 1 ssd and cloud storage. And all the really important images are with 2 cloud providers. I have not yet started to build a check-sum for every file, but I may do so.

And of course, check from time to time… and I replace my harddrive at the first sign of trouble or after 7 years.

[martinus]

@Mike_Bing

I think it is YOU who doesn’t understand. I have raw files dating back to 2002, 23 years ago and my current totally updated software can read and interpret those without any issues. The DNG format was launched in 2004 and there has not been a single year I heard people say proprietary raws would become unreadable “soon”.

I will not say that they will not be unreadable ‘soon’. I also don’t have converted everything in DNG. Yet, If you look long term, DNG is a clear winner, for a long shot. I don’t care if you convert or not, who am I. But you claim there is not difference. But there is. DNG will survive any other vendors RAW format, no question about that, because it is an open standard used by the big archival parties in this world. If that is with your / my lifespan, that is a total different discussion.

@Mike_Bing

Do you understand the notion of a raw file at all? It is usually a bland representation of naked sensor data which even if demosaiced represents a rather dull and flat version of a scene. I don’t give a sh!t whether the raw data is compressed one way or the other.

I very much understand that notion. But do you understand that a DNG also supports having that bland representation of naked sensor data which even if demosaiced represents a rather dull and flat version of a scene.. A DNG stores the raw, demosaiced data, just in a different format. It represents the same data. That Apple stores there data demosaiced, does mean that every DNG is

@Mike_Bing

I care about the raw file being the ONLY data representation which can be led back to my camera and mine only and to me as the author and rightful owner.

But, this also doesn’t hold for any raw file. Unless you have a content credentials capable camera. You cannot proof that your camera took a certain raw file. Given, it is unlikely that a Nikon camera will have taken ARF file. But any RAW file in the wild says absolutely NOTHING about intellectual property and copyright. Even a RAW file can be altered, the timestamp, the copyright everything. And there is no way you can proof that a given RAW file is taken by a given camera.

@Mike_Bing

And that seems to be something else which has escaped you: the notion and need of backups and backup rotation as well as the need to keep a physical copy at a separate location.

That is an assumption on your side. I have spread my backups over multiple media, multiple locations, yes actually countries. See my post above.

@Mike_Bing

The drive gets mirrored annually with the image validated and stored at my bank in a locker.

And that is a smart move. I do something similar. Just not in a bank.

@Mike_Bing

Here’s a screenshot from my backup drives connected to a raspberry pi hanging off the router.

Okay, I personally would not do that… I have found raspberry pi’s to be unreliable in the long run… But with multiple backups and locations, you should be okay.

[martinus]

@Mike_Bing

? It. did. not. happen! Why would you assume that would change?

Lots of reasons that can lead to these kind of changes… just a few suggestions:

1. Companies can go bankrupt. Some of vendor specific raw files contain features that features that open source is not able to read. (I am looking at you Nikon). If Nikon does not release new software, in the long run I will be able to read those files, or will have to jumps through hoops.

2. Patent issues. It will not affect you directly, now. But if tomorrow a big player find that his rights are violated… wow we are in for a potential surprise.

3. Not applicable for current cameras, but if camera companies starts to lock down and encrypt their raw files (it has been tried before). Then we will get into trouble. And if the camera companies give Adobe a license for Lightroom the world does not care

Against all these potential issues a DNG is protected

[Bert (Bert Blockx)]

I’m more interested in the possibility of using DNG as an intermediate format.
At the moment, I use RawForge to remove noise, which produces a DNG file.
The same applies to my Magic Lantern Dual-ISO files from my Canon 5D Mark III, which need to be converted from Canon CR2 format to DNG in order to be edited in darktable.
Those files are bigger than the original RAW files.

DXO PureRaw does the same as RawForge but with the newer, compressed version of the DNG format.
Apparently, Lightroom uses DNG to create panoramas and HDR merged files, amongst other things.

I certainly wouldn’t mind if these temporary files were a lot smaller.
For this reason I would love to see the latest DNG version 1.7 supported in darktable.

[rvietor]

@martinus

Against all these potential issues a DNG is protected

And those are all future issues, with little bearing on the current situation.

What has bearing on the current situation is:

@martinus

A DNG stores the raw, demosaiced data, just in a different format.

And that’s my issue with storing my base data converted to DNG: I do not want to lose the raw data, as the demosaicing process has an effect on the resulting image.

For the rest, I feel you are not always strictly separating the issues, e.g. between the current situation and the future, and between archival media, and an archival process (see my question about reliable media, where you replied with a process).

That gives the impression that you want to push something… And in that case, I prefer leaving the discussion.

[mino]

@Mike_Bing I am a bit confused by your harsh tone. If keeping your original RAWs works for you that’s great.

I personally would be glad to be able to store all my RAWs as JXL compressed DNGs because storage is growing slimmer and don’t mind the possible downsides. But it’s okay to take another route.

[martinus]

@rvietor

And that’s my issue with storing my base data converted to DNG: I do not want to lose the raw data, as the demosaicing process has an effect on the resulting image.

Sorry, I miss wrote… DNG can store raw undemosaiced data, it will store it in a standardized way, but it is raw undemosaiced data, again sorry for the miss information.

@rvietor

And those are all future issues, with little bearing on the current situation.

Not all of them… there are features in vendor raw files that open source processing is not supporting because of the closed source. I am a Nikon shooter so the high efficiency NEFs are out for me, and pixel shift images is also an issue.

And that the problem with future issues, you don’t know if it will happen… but are risks

[martbetz (ᴍᴀʀᴛ ʙᴇᴛᴢ)]

I see there’s an AppImage, which is extremely useful for me. As an Arch user, I like to keep my use of the AUR to a minimum (and I’m far too lazy to compile from source). Now I can swap out my explicite AUR install of DNGLab and use this instead.

I’m a very experienced Linux user, but I’ll always welcome a well put together GUI as an additional option.

Nice work, @Otherone !