Problems with RT and ON1 NoNoise DNG files

I use ON1 NoNoise as an initial step in my workflow to remove noise. I save the output as a DNG file and then do the rest of my processing using RT. This worked fine until a couple of months ago when ON1 released a new version of NoNoise. Now when I import the DNG into RT it looks terrible - way over exposed and some colour casts (even when using a Neutral profile). I contacted ON1 and sent them some sample files - they say it is a problem with RT as the DNG files look fine in Lightroom, and I need to change some of the parameters in RT.

After much tweaking I can get the image in RT looking somewhat similar to the raw file. The white balance is way off in the DNG file (eg 6490 vs 5202, even though both use ‘camera’ mode). If I manually change the settings in the white balance to match the raw file and then apply an ‘Auto-matched tone curve’ I can get the two images similar but still with some colour differences. I feel I am fishing in the dark a little, as I don’t really understand how RT processes DNG files. Does anyone have any ideas on what might be causing the problem and how best to fix it?

The following shows a typical image in RT with a neutral profile - Raw first then DNG version.


These are the same files with manually adjusted white balance and an auto-matched tone curve applied to each one.


Hi, thanks for the effort and sorry that things don’t immediately work in RT. It is interesting that it worked before… so something changed in the way ON1 exports its DNGs.
Would you be willing to share a DNG from before and after the change in ON1? I.e. one that does look good in RT and one that doesn’t look good anymore.

1 Like

Noooo clue if this is the same or even related …

But DNG files created by DxO Optics Pro work fine in dcraw and darktable , but in Rawtherapee they always opened with color casts.

After comparing files created by Adobe DNG converter i started trying stuff. By accident discovered that the color cast could be fixed by writing BlackLevel tags with values 0 in the DNG .

It could also be that Rawtherapee uses white levels and black levels from its camconst.json file , because it recognizes the camera make/model . But those values might not be correct anymore in a DNG file if the levels have been normalized or changed in the DNG writer .
I believe (but don’t quote me on that) that Rawtherapee uses BlackLevel and whitelevel if they are present in the DNG metadata , but if they are not there it searches camconst.json to get defaults for camera make/model . While some DNG apps expect ‘no BlackLevel metadata means 0’.

So, as a test: try removing camconst.json (only temporary) to see if that changes things.

And/or, compare a previous DNG file with a newer dng file by using exiftool and listing all the metadata inside of it (exif, xmp and ifd particularly). There might be a tag missing in the new one or changed that could be filled in to get the file working .

So you are saying that the DNGs have a black level of 0 but there is no black level tag at all?

I can’t remember if the black level is mandatory.

I don’t know if that’s always the case, i know it worked for my Sony a7m2 files which come from DNG from DxO photolab .

I know the same DNG files worked fine in filmulator, until it was changed to use the RT camconst.json than it got the same issues. Removing camconst.json brought it back.

I don’t know what the spec says … Dng comes in many flavours.

Attached are 2 sample DNG files. ON1v1 is from the previous version of ON1 NoNoise that works fine in RT. ON1v2 is from the new version of ON1 which doesn’t. Not sure whether this helps, but I’ve also noticed that if I create multiple layers in the previous version of NoNoise I get similar issues to the new version of NoNoise. If I export images as TIFF in either version they work fine.

P1210712_ON1v1.dng (63.4 MB)
P1210712_ON1v2.dng (77.4 MB)

v1 white/black points:

Black Level Repeat Dim          : 1 1
Black Level                     : 0 0 0
White Level                     : 63248 63248 63248

v2:

Black Level Repeat Dim          : 1 1
Black Level                     : 0 0 0
White Level                     : 65535 65535 65535

v2 has a linearization table present too

v1 has both ColorMatrix1 and ColorMatrix2, v2 has a VASTLY different ColorMatrix1

v2’s CM1 tag looks familiar - I think that’s an sRGB matrix???

It is not - if not present, the default to be assumed is 0.

1 Like

I removed camconst.json but this didn’t seem to make a difference. I then used exfitool to analyse the differences between the 2 DNG files, and where possible I changed the values in the non-working DNG file to match the working one (things I changed included WhiteLevel, ColorMatrix1 (and 2) and AsShotNeutral. None of these improved matters (some of the changes made it worse with a strong pink cast).

I am quite a novice in this area and have no previous understanding of DNG formats and use of exiftool, so it is possible that I may have made some mistakes in doing this. It does, however, seem clear that the new version of NoNoise is making quite a few changes to the DNG metadata from the previous one. I cannot see how I can configure these changes within the software so am assuming that these are hard-coded in some way.

I can get a working solution if I export as TIFF rather than DNG. NoNoise recommends the use of DNG as maintaining more of the raw data, however, from the limited tests I have done I seem to get very similar results in RT when I work on the TIFF file rather than the DNG one. Are there likely to be any drawbacks using TIFF in this way?

I bet the dng coming from ON1 is barely more than a container for a tiff…I vote you are not likely using much…NNoise has already modified your file so it’s not like you still have the raw data

TIFF is also just a container. DNG is a TIFF with a few more rules when it comes to content and metadata. You can study in detail what is contained in either using e.g. exiftool. It could even be the same content, but you can’t know until you analyze.

Since I was digging in RT dcraw.cc code for another RT pull request I tried to take a look at this.

The image is a linear dng with 16bit rgb (65536). The dcraw.cc linear_table function limits the parsing to 4096 values. So the linearization isn’t done for values greater than 4096 and you get exploded highlights.

I Don’t know if it’s a typo (just a missing ending 0 since the array is already defined to contain 65536 entries) but I extended the limit to 0x10000 instead of 0x1000 and I get a better image.
It’s just more exposed and more reddish, I think it’s related to the strange single ColorMatrix1 (probably on1 v2 is doing some more adjustments or there’s something else missing in RT).

I think this deserves a github issue to better discuss this. In the meantime, if you’re interested the patch is here: GitHub - sgotti/RawTherapee at dcraw_linear_table_increase_size

Pre patch (neutral profile):

After patch (neutral profile):

5 Likes

Many thanks for looking at this and producing a fix so quickly. I have to say I am impressed with RT as a piece of software, but also the helpful community around it

1 Like

Please make a PR for this. LGTM :+1:

2 Likes

@heckflosse Thanks. Done: dcraw: increase linear table parsing to 65536 values by sgotti ¡ Pull Request #6448 ¡ Beep6581/RawTherapee ¡ GitHub

Regarding the more reddish on1 v2 image looks like a different default white balance. If I use the same white balance of the on1 v1 image (and increare the v1 exposure) they looks the same (with more sharpening introduced in the dng by on1 v2). I’ll take a look also at this.

Can you run ON1 as an external editor in RT and do the direct return back to RT after using O1 ??

I don’t think you can connect RT and ON1 in a particularly seamless way. You can invoke ON1 as an external editor in RT. If you do this RT converts the file to TIFF and then passes this to ON1. ON1 can work on this TIFF file and save it (as TIFF or DNG). However, it doesn’t automatically pass control back to RT. You then have 2 files available in RT - the original raw and the denoised TIFF. I think there is better integration with Lightroom or Capture1 through plugins. The ON1 documentation doesn’t specifically mention RT, but does say that if using Lightroom without the plugin and instead use ‘Edit In’ the process appears similar to that described above with RT.

The ON1 documentation also describes an automated way of passing images between Photoshop and NoNoise, but also recommends that for optimum results you invoke NoNoise in standalone mode and work on the Raw file, saving as DNG, before opening this file in Photoshop.

In general, ON1 recommend you denoise Raw files rather than TIFF ones for best results.

In [4]: colormatrix = np.array([[3.1339, -1.6169, -0.4906], [-0.9788, 1.9161, 0.0335], [0.0719, -0.229, 1.4052]])

In [5]: invmatrix = np.linalg.inv(colormatrix)

In [6]: print(colour.primaries_whitepoint(invmatrix))
(array([[ 0.64839926,  0.33085869],
       [ 0.3211397 ,  0.597861  ],
       [ 0.15587208,  0.06602003]]), array([ 0.34566183,  0.3584914 ]))

It’s not quite sRGB, but it’s pretty close other than the whitepoint being way off from sRGB

So the v2 outputs are a quite restricted gamut. Since it’s using the LinearizationTable it has no way to represent out-of-gamut colors with negative numbers.

In DT if you switch the input profile to linear rec709 and turn off the WB module then you get this…no other edits…

image

1 Like

That’s pretty close. Rec709 and this DNG (Edit: Previously said rec2020 which was REALLY WRONG - what a typo!) have nearly the same primaries. I’m guessing that under the hood the DNG linearizationTable is still used despite declaring it “linear”, unless that LinearizationTable is, itself, linear, in which case my thoughts are “WTF?”

The closest match to the primaries appears to be NTSC, but again - not quite, and it has a whitepoint of D50.

I would consider this a significant regression in behavior for the “new” ON1 behavior - it’s now gamut restricted to approximately the sRGB gamut. Not sure if ON1 has any options to disable this undesirable behavior.