Help with a floating point DNG file and processing it in RT

[adrian_bacon]

I’m very new here and very new to RT, since I don’t know what I don’t know, please be kind and just point me in the right direction. I’ve already read RawPedia, and searched the forums.

I’ve been working on some code that takes a raw scan of a color negative film, inverts it into a positive, linearizes it, and conforms it to a color profile, then outputs the image to a floating point DNG file. The DNG file in question is for all intents and purposes a linear HDR image that contains (depending on the film used and contrast of the scene captured) upwards of 20+ stops of DR.

The samples are scaled as such that that a correctly exposed 18% gray card is 4 stops below 0.18 with the baseline exposure tag set to +3. The lower limit where the film base plus fog is roughly 4-5 stops below that, and the the upper limit where the film stops recording density is easily several (or more) stops above the middle gray value.

For all intents and purposes, it’s a linear DNG that has a lot of dynamic range. Here’s where I’m left scratching my head trying to figure out what I’m doing wrong: When I add this DNG to Adobe Lightroom, and look at the preview it generates, it looks exactly how I would expect it to; not really any different than any other image that I may have shot digitally, except that it has way more dynamic range in it than had I shot it with a digital camera. When I export it to a Tiff or jpeg, what I saw in the preview is pretty much what I get with the exported file. Life is good.

Now on the confusing part: when I open this exact same file in RT, no matter what I seem to do, I get a crazy super low contrast image. The colors look OK, and if I place an 18% gray card at 0.18, that renders out to about 50% on the histogram in RT, so it looks like RT is recognizing the DNG color matrix tag that I embedded, but for some reason, it does not seem to think that this file is has a linear gamma. If I’m feeding it linear data, and it is linear internally, then my preview being rendered to my display using my display icc should look correct, but instead it looks super low contrast.

From my own experience with Lightroom with my files, I know that internally it clips floating point data at 256.0, so any values above 256 basically just get clipped off, and it assumes a black level of 0 unless told otherwise in the file, otherwise, it assumes 0.18 is 18%. Knowing this, I set the white level tag in the DNG to 256, and get a completely clipped to white image in RT, even though none of the values in my DNG go that high. Confusing. If I scale the white level tag up it starts to look more normal by 16383, and is back to what I was originally seeing at 65535, however, the blacks are always at least 40% on the histogram. I’m really confused. I would think that if I feed a scene referred floating point DNG to RT, even with the neutral profile, it would treat it as a scene referred linear image and simply clip off anything in the preview that is outside of what is displayable until I go into the processing options and start to muck around with stuff instead of doing what it’s doing.

What am I doing wrong? Should I be assuming that RT is expecting the floating point values to be in a different range? Why does it seem to be crushing all the contrast down to a super flat image?

Please help out a clearly clueless person. Here’s a link to download the DNG file in question as well as the .pp3 file that RT made when I opened it in the editor: Dropbox - File Deleted

Many thanks.

Adrian

[paperdigits (Mica)]

Welcome! I have no answer for you, but I suspect the answer you get will be interesting

[Morgan_Hardwood]

Hi @adrian_bacon

• Please show a screenshot or saved JPEG of what you expect this file to look like. Geeqie, Gwenview and ExifTool are unable to find an embedded preview.
• ColorMatrix1 uses 32-bit signed int precision. This might be fine, but I just wanted to point that out, as I haven’t seen that before and so don’t know whether it’s handled correctly.

  Exif.Image.ColorMatrix1    SRational   9  2147483647/2045590716 0/1 -209346/2147483647 -1064943615/2147483647 2147483647/1563724757 210968863/2147483647 0/1 0/1 2128697471/2147483647
  

  I’m used to seeing one of these:

  Exif.Image.ColorMatrix1    SRational   9  63579/65536 -31490/65536 -5214/65536 -22911/65536 64449/65536 28093/65536 -1355/65536 2831/65536 51795/65536
  Exif.Image.ColorMatrix1    SRational   9  7491/10000 -3027/10000 351/10000 -4231/10000 11579/10000 3031/10000 -818/10000 1721/10000 7466/10000
  Exif.Image.ColorMatrix2    SRational   9  148/128 -37/128 -41/128 -68/128 200/128 8/128 -10/128 36/128 72/128
  

• RawTherapee’s floating-point DNG support was implemented to make HDRMerge’s output compatible with RawTherapee. I don’t recall seeing anyone mention testing it with float DNGs from other sources.
• Your DNG file has no WhiteLevel or BlackLevel tags.
• White level from DNG metadata is ignored · Issue #4472 · Beep6581/RawTherapee · GitHub

[adrian_bacon]

@Morgan_Hardwood thank you. I’m not at my computer right now, but I’ll generate a jpeg out of Lightroom and attach it as soon as I am. For the other points:

Yes, there is no preview embedded in the file as it is straight out of my tool with no Importing into Lightroom, which has a bad habit of writing tags in. I wanted an unadulterated version for people to look at.

The color matrix I embedded is 6 full digits of precision. My tool actually defaults to 32 bit floats for the image data, which LR suppports just fine. I had to downgrade to 16 bit floats to get RT to do anything with them.

Hmm… I’ve not looked closely at what HDRMerge generates, but I suspect it’s probably not generating what I’d call scene referred, and probably not that spec compliant.

That’s absolutely correct that there is no white and black level tags defined in my DNG, nor will there likely be any for one very simple reason: the DNG 1.4 spec very clearly defines what those default values are if none are explicitly defined in a given file or profile. The expectation is that a spec compliant file should be handled per the spec, and my DNG file is spec compliant. If it wasn’t, LR and ACR would puke on it or behave strangely, and they very clearly don’t.

Yes, I saw issue 4472. Again, the spec very clearly defines what those values are supposed to be if not defined anywhere else.

[agriggio (Alberto)]

Hi @adrian_bacon, thanks for the report. I think RT is not very compliant with the DNG specs, I am sure there is a lot of room for improvement in that area. I’ll try to take a look at what is going on with your file, but I can’t promise anything about the turnaround time at the moment, sorry… (but maybe some other dev can – ping @heckflosse

[adrian_bacon]

@agriggio, Thank you.

Here’s the sRGB jpeg of this DNG as rendered by Lightroom. All I did was add it to the LR catalog, then exported it. Like I said before, for the most part, it looks pretty much the same as it would had I pulled my iPhone out of my pocket and shot the exact same photo, except that this photo was shot on FujiColor 200 color negative film.

The min and max floating point image data values for this photo being written into the DNG are:

max RGB: 0.859318, 0.635701, 0.281081
min RGB: 0.000901, 0.000957, 0.000846

For this particular scene, not particularly high dynamic range, and well within 1.0-0.0.

Feel free to use the DNG I provided as a reference for a scene referred floating point DNG that is spec compliant.

[ilias_giarimis]

Are you sure that your data are linear ?
Looks like either LR applies a TRC curve (gamma correction ??) or RT applies a wrong TRC (inverse gamma ??)

Add upon this the fact that RT does not recognize the “baseline exposure” tag nor the embedded color profile.

Can you upload the (embedded in DNG) color profile in dcp or icc format ?.

[adrian_bacon]

Yes, it is linear. I own and operate a film lab and have the equipment at my disposal to measure the gamma of the straight line portion of any film emulsion. From there, it is trivial to make it linear. For this particular emulsion, the red channel is 0.46, the green channel is 0.52, and the blue channel is 0.56. Once it’s linear, proceed with whatever else you’re going to do with it.

Can you upload the (embedded in DNG) color profile in dcp or icc format ?.

This request tells me that you have a very basic mis-understanding of the DNG format. There is no ICC or DCP embedded in my DNG because you don’t need them unless you are deviating from the defaults that are specified in the 1.4 spec.

You can certainly embed an ICC profile in the DNG with the AsShotICCProfile tag or the CurrentICCProfile tag, but either one of those requires a correct AsShotPreProfileMatrix or CurrentPreProfileMatrix tags. Going that route, then requires the DNG processor to take the DNG image data, apply the pre-profile matrix to get the image data from whatever colorspace it is in to the color space of the ICC profile embedded, then convert it from that ICC color space to the native ProPhoto color space. If you really want to get convoluted, you can certainly do that, however, I don’t recommend it as it is totally unnecessary.

OR, you can simply not do any of that and just include a correct ColorMatrix1 tag, which is defined as the transformation matrix that converts XYZ values to the native color space of the image data. Going that route, then becomes nothing more than apply the inverse of the ColorMatrix1 tag to the image data to get to XYZ, then apply the XYZ to ProPhoto matrix to get to ProPhoto, or if you’re exporting to a jpeg or tiff that is a different color space, go from XYZ to whatever other colorspace. Much simpler. This is basically what Lightroom is doing to render the sRGB jpeg I uploaded. All that being said, the samples in the DNG file I provided are natively in the ACES 2065-1 color space and the ColorMatrix1 tag is in fact the XYZ to AP0 matrix that is published on the internet.

It would probably be useful to read chapter 6 of the DNG 1.4 spec as it covers this with sufficient detail.

As for the other things, the DNG spec specifies default values for all things that are not explicitly enumerated in the file itself or a ICC/DCP etc. So in this case, the white and black levels should be at their default values of 1.0 and 0.0 (since we’re dealing with floating point data) and the gamma for the image data should be at the default value of 1.0.

In order to correctly render my DNG to any color space and have correct looking output in terms of color, all you need is the ColorMatrix1 tag. The image data is already linear and middle grey is 0.18 minus the baseline exposure offset, which from what I’m gathering is not in a range that RT is particularly well equipped to deal with as the image data does not exactly look like raw data that you would get from a real camera, but is instead centered around middle grey (again, the baseline exposure offset comes into play, but once it’s applied, then yes, everything is centered around middle grey).

Adrian

[adrian_bacon]

OK, so now I’m pretty certain that RT is doing something unexpected when processing floating point DNG files because here’s a screen shot of the exact same DNG as I provided, except generated where the image sample data is between 65535 and 0, then output as linear 16 bit unsigned integer values.

[adrian_bacon]

…And here’s a screenshot of the exact same file as the previous post, except I also had my tool apply a reverse 2.4 gamma (0.416666666666667) to the output before scaling it to 65535-0. That looks an awful lot like what RT is doing to the linear floating point DNG that I provided before. It would appear that RT somewhere in the chain is applying an inverse gamma function where it should be treating things as linear.

[agriggio (Alberto)]

@adrian_bacon, as was written before currently the support for float DNGs in RT is only partial, and only aimed at what hdrmerge produces. I’m looking into this, I’ll post an update as soon as I have more news. Thanks for your tests btw! If you have also a 32-bit float that you can share, that would also be appreciated.

[agriggio (Alberto)]

Hi @adrian_bacon, I managed to hack something together to support your DNGs. First, you need to apply this patch on the current dev version of RawTherapee. Then, apply also the following patch:

diff -r 6fce4eda8251 rtengine/dcraw.cc
--- a/rtengine/dcraw.cc	Wed Oct 24 12:03:15 2018 +0200
+++ b/rtengine/dcraw.cc	Sat Oct 27 13:53:10 2018 +0200
@@ -30,6 +30,10 @@
 //#define BENCHMARK
 #include "StopWatch.h"
 
+#include <zlib.h>
+#include <stdint.h>
+
+
 /*
    dcraw.c -- Dave Coffin's raw photo decoder
    Copyright 1997-2018 by Dave Coffin, dcoffin a cybercom o net
@@ -1170,23 +1174,37 @@
   }
 }
 
+static uint32_t DNG_HalfToFloat(uint16_t halfValue);
 void CLASS packed_dng_load_raw()
 {
   ushort *pixel, *rp;
   int row, col;
+  int isfloat = (tiff_nifds == 1 && tiff_ifd[0].sample_format == 3 && tiff_bps == 16);
+  if (isfloat) {
+    float_raw_image = new float[raw_width * raw_height];
+  }      
 
   pixel = (ushort *) calloc (raw_width, tiff_samples*sizeof *pixel);
   merror (pixel, "packed_dng_load_raw()");
   for (row=0; row < raw_height; row++) {
-    if (tiff_bps == 16)
+    if (tiff_bps == 16) {
       read_shorts (pixel, raw_width * tiff_samples);
-    else {
+      if (isfloat) {
+          uint32_t *dst = reinterpret_cast<uint32_t *>(&float_raw_image[row*raw_width]);
+          for (col = 0; col < raw_width; col++) {
+              uint32_t f = DNG_HalfToFloat(pixel[col]);
+              dst[col] = f;
+          }
+      }
+    } else {
       getbits(-1);
       for (col=0; col < raw_width * tiff_samples; col++)
 	pixel[col] = getbits(tiff_bps);
     }
-    for (rp=pixel, col=0; col < raw_width; col++)
-      adobe_copy_pixel (row, col, &rp);
+    if (!isfloat) {
+        for (rp=pixel, col=0; col < raw_width; col++)
+            adobe_copy_pixel (row, col, &rp);
+    }
   }
   free (pixel);
 }
@@ -10086,7 +10104,14 @@
     if (raw_width  < width ) raw_width  = width;
   }
   if (!tiff_bps) tiff_bps = 12;
-  if (!maximum) maximum = ((uint64_t)1 << tiff_bps) - 1; // use uint64_t to avoid overflow if tiff_bps == 32
+  if (!maximum) {
+      if (tiff_nifds == 1 && tiff_ifd[0].sample_format == 3) {
+          // float DNG, default white level is 1.0
+          maximum = 1;
+      } else {
+          maximum = ((uint64_t)1 << tiff_bps) - 1; // use uint64_t to avoid overflow if tiff_bps == 32
+      }
+  }
   if (!load_raw || height < 22 || width < 22 ||
 	tiff_samples > 6 || colors > 4)
     is_raw = 0;
@@ -10169,8 +10194,8 @@
 
 /* RT: DNG Float */
 
-#include <zlib.h>
-#include <stdint.h>
+// #include <zlib.h>
+// #include <stdint.h>
 
 static void decodeFPDeltaRow(Bytef * src, Bytef * dst, size_t tileWidth, size_t realTileWidth, int bytesps, int factor) {
   // DecodeDeltaBytes
@@ -10202,9 +10227,10 @@
 
 }
 
-#ifndef __F16C__
+#if 1 //ndef __F16C__
 // From DNG SDK dng_utils.h
-static inline uint32_t DNG_HalfToFloat(uint16_t halfValue) {
+static //inline
+uint32_t DNG_HalfToFloat(uint16_t halfValue) {
   int32_t sign     = (halfValue >> 15) & 0x00000001;
   int32_t exponent = (halfValue >> 10) & 0x0000001f;
   int32_t mantissa =  halfValue        & 0x000003ff;

And now you should get a rendering that’s reasonable. The BaselineExposure tag is still ignored, but that’s easy to correct manually. If you can post also a 32-bit version, I will try to take a look at that too.
Note that the code is totally unpolished, it would need a bit of cleanup before getting into RT (including proper error checking and more testing).

Hope this helps anyway.

[adrian_bacon]

@agriggio many thanks. I’m not at my computer right now, but will post 32 bit floating point versions as soon as I can as well as monochrome versions of scans of black and white film, which we haven’t even gotten to yet. If we can get this improved processing into the next official release of RT that would be wonderful.

If possible, adding support for honoring the baseline exposure tag would be extremely useful as many of my clients that prefer to use RT instead of LR/ACR may not be as experienced at dealing with HDR class images and not understand that the DNG spec expects the important parts of the image to be between 1.0 and 0.0 if it is floating point, which results in really dark images if the baseline exposure tag is ignored.

Stopping to think about it, if you scale your image data so that all of the image DR is between 1.0 and 0.0 in floating point, how do you know where middle gray is? Many image operations are best between 1.0 and 0.0, so there’s a lot to be said for having everything between 1.0 and 0.0 as a normal state. Adobe’s solution prescribed in the DNG spec is to provide the baseline exposure tag to indicate where middle gray is. It’s not perfect, but it works. After converting the EV value to a multiplier and applying that, you essentially end up with an “unbounded” floating point image, where middle gray is 0.18 and your whites and blacks end up where they do, the whites many times, way over 1.0.

At any rate, I know that unbounded Floating point image stuff has been the subject of some pretty contested discussion in the past and my intent is not to reignite that, but instead point out that Adobe’s way of dealing with it in floating point DNG files could be a reasonable compromise, at least if dealing with DNG files.

Adrian

[adrian_bacon]

@agriggio Here is the link to the files you asked for: http://m.avcdn.com/sfl/adobe_floating_point_dng_reference_images.zip

That file will stay at that link until I manually do something about it, once again, feel free to use these images as reference/test images.

Included in the zip file is both 16 and 32 bit floating point images, a 16 bit floating point image with the Adobe Deflate compression that RT seems to have some trouble with, and a 16 bit floating point monochrome image that RT also has some issue with.

I’ve not yet had a chance to try your patch out since I’ve not ever built RT, but instead downloaded the OS X pre-built image, so I’ll have to go through that process to try it out.

At any rate, thank you for your effort here as this will benefit not only me, but all my clients, and anybody else who uses floating point DNG files.

Incidentally, I’ve made the discovery that Affinity Photo behaves exactly the same way with my files as RT does before your patches, so I’ll also be reaching out to them as well as probably Dave Coffin with these same reference images in an effort to try to get improved handling of floating point DNG files. If there are other raw processing libraries that support floating point DNG files, please feel free to forward the above link to them so that they can test and fix any issues that come up.

Adrian

[gaaned92]

if ever you can use a Windows PC,
https://keybase.pub/gaaned92/RTW64NightlyBuilds/RawTherapee_dev_5.4-996-gb651cdd40_WinVista_64.zip
is patched

[agriggio (Alberto)]

I’ve created a floatdng branch to work on this. See also:
https://github.com/Beep6581/RawTherapee/issues/4900

[agriggio (Alberto)]

Thanks @HIRAM, but I wasn’t done yet
I’ve just pushed a few more changes, with support for the BaselineExposure tag (as everything in this branch, this should be considered experimental). Now all the DNGs provided by @adrian_bacon should render correctly, with the exception of the 16-bit one with deflate compression. The problem with that is that it is using (as far as I can tell) a “strips” layout, whereas RT currently supports only a tiled layout for deflate compressed float DNGs. This is however something can I won’t be able to address at the moment, but maybe someone else can pick up from here. Or just be aware of this limitation and do not generate this kind of files in the first place

[adrian_bacon]

Many thanks.

Yes, my files are not tiled, but rather striped. I can look into generating tiled DNGs, however, the simplest thing so far has been to just do the stripes, one row per stripe.

Doing the compression is a nice to have, though not totally necessary as the savings are typically 30% or less with the floats.

[HIRAM (Richard E Barber)]

Ok, here is an up to date floatdng build:
https://filebin.net/olar3dimkj1c55nb/RawTherapee_OSX_10.9_64_5.4-1013-g632c1362c.zip?t=caxddn57