Is there an open-source RAW format?

I’ve been revisiting my file storage solutions and pondering about whether I ought to convert everything to dng format for “compatibility”. This git me to thinking about why I would want to use dng over my camera raw format (orf in my case). What advantages do I greatly gain? Dng is, itself, a proprietary format being pushed by Adobe, and is only “universal” in the sense that a couple camera manufacturers have adopted it (Ricoh and Leica, mainly), and any software that wants to emulate Adobe products in some way will try to offer dng support. Of course we, in the Foss community have the awesome dcraw and other libraries, si we can open pretty much any raw format we want to. But, this got me to thinking, have we tried to make our own raw format that is fully open source, and that could be used as a “lingua franca” moving forward? Something that could be used by open source hardware like that new mapir camera or by open smart phones and other fun hackable? I tried to search, but came up empty. Anyone know of something like this? Off the top of my head, high bit depth TIFF files are the closest thing I think we have to this, but obviously there are drawbacks to that format (as well as advantages). What else is out there that could be thought of as a truly open source raw format?

I don’t think you gain anything by converting, and you’ll most likely loose some information depending on what the source raw format is.

Tiff really isn’t close, as lots of decisions about the image have been made when it is in tiff.

What you really want for a file format is an open specification. This means no reverse engineer of the file format, instead just code to the spec and test.

I don’t think there would be much adoption of a new file format, as you can see the lack of adoption with DNG and that is with an industry giant like adobe pushing it.

Pentax aslo does dng I believe.

Yes, I agree that converting to dng doesn’t give any real advantages, other than having all your files in one format. You will potentially loose some metadata too, depending. Tiff is also unsatisfactory, for similar reasons. I pretty much realized that an open raw format would have no advantage for storage purposes, but mainly, I’m interested in a truly open raw format to use with open source hardware cameras. I have an inkling that open hardware cameras are going to take off soon, and it would be really cool to have an open raw format to use with them.

2 Likes

Actually quite a few raw formats are based on tiff. In the end it’s just a container that you can fill with data. Are there any real problems with DNG and/or it’s licensing that would justify coming up with yet another format? Reminds me of this:

8 Likes

For a format to be of any use to users it needs to have wide support in software, so that you can be certain that you will be able to find some program which can open it or convert it to a more modern format 10, 50, 100 years from now. Otherwise you end up with formats like PGF and BPG which are better than JPEG but generally not supported therefore generally not used.

DNG is open, supported by image viewer on Mars, and it’s here to stay.

There are benefits of converting to DNG, they depend on what raw format you’re converting from and what program you’re opening the raws with.
See How to convert raw formats to DNG - RawPedia

– update
Haha @Jonas_Wagner I didn’t scroll down and didn’t see your reply until I posted this. I will leave the duplicate comic to stress the point :wink:

1 Like

Yes, I have seen that meme, and fully get the point it makes. There is, however, some issue with relying only on a standard that is being pushed by a single company, even if they’ve made it “open”. One big drawback to dng is that you are limited only to record the kinds if metadata that Adobe has thought important. However, as open source hardware cameras catch in and meld with other open hardware like teensy, arduino, raspberry pi, etc, I can see the need to record a lot of non standard meta data. I guess it’s neither here nor there, however. Dng or some variation of TIFF is what we’ve got, so that’s what we will likely use moving forward. For example, geoTIFF is the default open standard for raster files in gis, despite some “better” data containers used by certain projects (GRASS’s ASCII and binary formats, for example).

Doesn’t XMP stand for Extensible Metadata Platform?

I thought dng doesn’t take or need xmp sidecars. Isn’t that one of the “selling points” of dng? That it is self contained? And, what raw format does make an external xmp in camera for metadata? None that I’m aware of?

Can you not write xmp into the file header?

Of course you can.

DNG is a variation of TIFF, too. To me it seems you want to fix problems that are not there in the first place.

Oh, no, I’m not suggesting to do anything at the moment… I’m just merely bringing up the questions. I simply have a feeling that pretty soon there might be a need for something like this. I quite clearly can see that as of right now, however, there isn’t a compelling need, as it would add absolutely zero practical advantage to what’s out there for current uses. Add in that open hardware, however, and then I’m not so sure…

I’m sure you can. And then we are back at a “special” variation of TIFF. This is the current situation for geoTIFF, by the way. Some programs require a sidecar file (.tfw) with the geographic data, whereas most modern software can read these data direct from the file header. This is a source of constant mix up among my students, who never know what creation options to choose when exporting to geoTIFF. The stand alone geoTIFF is all well and good for archiving and portability in theory, but it comes down to the actual file size of TIFF in the end. There are more efficient containers than TIFF(netCDF, ECW, various binary formats), but these are not as widespread, so geoTIFF remains the standard, at least for now. NetCDF is perhaps the biggest new contender, and is starting to take off in popularity now that raster data files regularly can exceed 1gb in size. The main advantage of netCDF is efficient lossless compression for very large file sizes. It is starting to be the standard for data from NCAR, USGS, and NASA.

Yes, there are some benefits. But also pretty major drawbacks. Not least being that only Adobe’s converter guarantees accurate conversion, a D if you want to use that on Linux then you have to use WINE. Sure, it works, but not ideal from a Libre point of view. I’m not making any judgements, just putting it out there.

It is not true that Adobe DNG Converter guarantees accurate conversion - every program has bugs and Adobe DNG Converter is no exception; it is not true that bugs aside Adobe DNG Converter is the only program capable of accurate conversion - what does “accurate” even mean, and it’s an open standard which anyone can follow; using Adobe DNG Converter in Linux does require Wine though I don’t see how that is a problem, let alone a major one.

I would like to hear about these major drawbacks, including sources. If there’s a flaw, I want to know about it. But so far every conversation of this nature starts with the FUD which littered photography forums during the last decade, FUD which was based on facts at some early point but those facts have become obsolete since spec version 1.2.0.0 or even 1.1.0.0, and then the conversation gradually dwindles down to nothing as people fail to find sources to back up their claims and realize the Adobe DNG Converter, like Roy Batty from Blade Runner, is not all that bad.


One issue which has not been mentioned is that the spec can be ambiguous about how certain elements of it are to be interpreted, what order they should take in the processing pipeline. Though this has not much/nothing to do with raw file conversion.

There is only one problem I see which I consider to be a major one: you might lose metadata. EXIF is using absolute offsets, so just copying over the whole blob to the DNG won’t work*. You have to parse it and understand every single setting in there. However, that is not possible in the general case, at least with maker notes. So the probability to miss something is quite high. Should one day such a metadatum become important – either for your workflow or for the program you are using – you are screwed. It’s basically the same argument against editing raw files.

On the other hand I would be happy to see a single argument FOR conversion.

*) Interlude: You could copy the whole blob but that would mean you couldn’t easily do necessary adjustments.

Well, even if open camera hardware will become a thing (I doubt it, besides toy projects) then why not just write DNGs? That is a widely supported standard, free to use (AFAIK) and technically sound. And the code needed to write such files is quite small, too, and there are plenty of places to look for examples.

Until DNG decides to murder it’s maker, you say? :smiley:

1 Like

Sure thing. I will break them down them according to three categories.

Licensing issues
These come straight from the DNG Specification Patent License.

  1. “Adobe may revoke the rights granted above to any individual or organizational licensee in the event that such licensee or its affiliates brings any patent action against Adobe or its affiliates related to the reading or writing of files that comply with the DNG Specification.”
  2. “Subject to the terms below and solely to permit the reading and writing of image files that comply with the DNG Specification”
  3. “Adobe provides the DNG Specification to the public for the purpose of encouraging implementation of this file format in a compliant manner.”
  4. “All rights not expressly granted herein are reserved.”

These may all be “fair” in terms of a general copyright license, but they are things that would not be in a truly libre software license. In fact, according to the patent license linked above, I’m not at all sure that DNG can be included in a truly libre project (although IANA patent lawyer). Adobe remains in express and complete control of the specification, and so is the one driving the ship here, not the community.

The DNG Specification
All of these come from the current version of the DNG Specification you can download from Adobe’s website.

  1. You can use TIFF-EP, EXIF, IPTC, or XMP tags for metadata, but only in the standard tags, or the extra tags that Adobe has added. If you want other metadata, this is what they say: “Camera manufacturers may want to include proprietary data in a raw file for use by their own raw converter. DNG allows proprietary data to be stored using private tags, private IFDs, and/or a private MakerNote.” To me, this means “we will allow you to pigeonhole your non-adobe approved metadata into our container, but we will never make any moves to help you make it easy, or to ever bring those fields into the standard tag set, so if you want these metadata to be read, you have to make a special extra reader to get them in and out. Good luck with that!”

  2. “Lossy JPEG (34892) is allowed for IFDs that use PhotometricInterpretation = 34892
    (LinearRaw) and 8-bit integer data.” So, it is possible to have lossy compression inside DNG in some circumstances. The end user of a program may not know that, and accidentally use lossy compression when they wanted a true “archival” format. This may not commonly happen, but IMO, it would still be better for a camera raw format to never use lossy compression.

General Issues
Adobe is now on version 1.4.0.0 of their DNG specification. Looking at the changes between versions, they are almost all of them to do with adding tags for new features/capabilites in Adobe Lightroom and or Photoshop. Therefore, it is pretty clear to me that, since Adobe maintains control of the specification, they mainly want it to be “open” to the public so as to entice people into using their own software. Indeed, you can only get the most out of the DNG format if you set up your software to record or need the same types of information that Adobe products take. This stifles creativity and freedom.

Case in point what I said above about Adobe Raw Converter. In fact it is in the very same RawPedia article you linked to that I paraphrased from the following quote: “Adobe’s DNG Converter is not the only program that converts raw files to DNG … however it is unknown what matrices this converter uses or where it gets them from, therefore it is safer if you stick to using the official Adobe DNG Converter.” (Emphasis added by me). Since this only runs via WINE, and it is not libre, it not a great solution for Linux users. And, if this truly is the safest way to convert DNG, then it is really a bad solution for Linux users.

Not all companies are bad, and there are reasons to limit data storage formats to certain specifications. But, do we really want a company who’s main business is selling (very expensive) software to be the ones in charge of creating the specifications??

Just remember, however, that it was not very long ago that Linux operating systems were considered the hobby OS’s of a small group of enthusiasts, and would never get large. That has changed. Arduino’s and other open microcontroller kits were just for weird tinkerers and students of electrical engineering. That has changed. Raspberry pi was a fun idea for people who wanted to waste time on a cell phone processors. That has changed. Drones were silly things that RC hobbists built in their garages, and no one else cared about. That has changed. All of these are robust businesses nowadays, generating serious revenues. The maker community is getting larger, not smaller. People want to mount cameras to all kinds of things these days, not just in their phones. The cameras used in those phones are pretty great, and very small, and easy to hack. It seems to me that the conditions are ripe for open camera hardware and software to take off, although who knows if it really will. I’m not proposing to make a preemptive strike and make another new standard where there is no real need for it. I’m merely pointing out that there are some pretty serious problems with what we seem to have as of now, and I was curious if any projects had been started that might mediate some of those issues if and when there is a serious need for it…