Problems with RT and ON1 NoNoise DNG files

Software RawTherapee
9

I removed camconst.json but this didnā€™t seem to make a difference. I then used exfitool to analyse the differences between the 2 DNG files, and where possible I changed the values in the non-working DNG file to match the working one (things I changed included WhiteLevel, ColorMatrix1 (and 2) and AsShotNeutral. None of these improved matters (some of the changes made it worse with a strong pink cast).

I am quite a novice in this area and have no previous understanding of DNG formats and use of exiftool, so it is possible that I may have made some mistakes in doing this. It does, however, seem clear that the new version of NoNoise is making quite a few changes to the DNG metadata from the previous one. I cannot see how I can configure these changes within the software so am assuming that these are hard-coded in some way.

I can get a working solution if I export as TIFF rather than DNG. NoNoise recommends the use of DNG as maintaining more of the raw data, however, from the limited tests I have done I seem to get very similar results in RT when I work on the TIFF file rather than the DNG one. Are there likely to be any drawbacks using TIFF in this way?

10

I bet the dng coming from ON1 is barely more than a container for a tiffā€¦I vote you are not likely using muchā€¦NNoise has already modified your file so itā€™s not like you still have the raw data

11

TIFF is also just a container. DNG is a TIFF with a few more rules when it comes to content and metadata. You can study in detail what is contained in either using e.g. exiftool. It could even be the same content, but you canā€™t know until you analyze.

12

Since I was digging in RT dcraw.cc code for another RT pull request I tried to take a look at this.

The image is a linear dng with 16bit rgb (65536). The dcraw.cc linear_table function limits the parsing to 4096 values. So the linearization isnā€™t done for values greater than 4096 and you get exploded highlights.

I Donā€™t know if itā€™s a typo (just a missing ending 0 since the array is already defined to contain 65536 entries) but I extended the limit to 0x10000 instead of 0x1000 and I get a better image.
Itā€™s just more exposed and more reddish, I think itā€™s related to the strange single ColorMatrix1 (probably on1 v2 is doing some more adjustments or thereā€™s something else missing in RT).

I think this deserves a github issue to better discuss this. In the meantime, if youā€™re interested the patch is here: GitHub - sgotti/RawTherapee at dcraw_linear_table_increase_size

Pre patch (neutral profile):

P1210712_ON1v2-pre-patch
P1210712_ON1v2-pre-patch4592Ɨ3448 5.4 MB

After patch (neutral profile):

P1210712_ON1v2-after-patch
P1210712_ON1v2-after-patch4592Ɨ3448 5.92 MB

5 Likes
13

Many thanks for looking at this and producing a fix so quickly. I have to say I am impressed with RT as a piece of software, but also the helpful community around it

1 Like
14

Please make a PR for this. LGTM :+1:

2 Likes
15

@heckflosse Thanks. Done: dcraw: increase linear table parsing to 65536 values by sgotti Ā· Pull Request #6448 Ā· Beep6581/RawTherapee Ā· GitHub

Regarding the more reddish on1 v2 image looks like a different default white balance. If I use the same white balance of the on1 v1 image (and increare the v1 exposure) they looks the same (with more sharpening introduced in the dng by on1 v2). Iā€™ll take a look also at this.

16

Can you run ON1 as an external editor in RT and do the direct return back to RT after using O1 ??

17

I donā€™t think you can connect RT and ON1 in a particularly seamless way. You can invoke ON1 as an external editor in RT. If you do this RT converts the file to TIFF and then passes this to ON1. ON1 can work on this TIFF file and save it (as TIFF or DNG). However, it doesnā€™t automatically pass control back to RT. You then have 2 files available in RT - the original raw and the denoised TIFF. I think there is better integration with Lightroom or Capture1 through plugins. The ON1 documentation doesnā€™t specifically mention RT, but does say that if using Lightroom without the plugin and instead use ā€˜Edit Inā€™ the process appears similar to that described above with RT.

The ON1 documentation also describes an automated way of passing images between Photoshop and NoNoise, but also recommends that for optimum results you invoke NoNoise in standalone mode and work on the Raw file, saving as DNG, before opening this file in Photoshop.

In general, ON1 recommend you denoise Raw files rather than TIFF ones for best results.

18 
In [4]: colormatrix = np.array([[3.1339, -1.6169, -0.4906], [-0.9788, 1.9161, 0.0335], [0.0719, -0.229, 1.4052]])

In [5]: invmatrix = np.linalg.inv(colormatrix)

In [6]: print(colour.primaries_whitepoint(invmatrix))
(array([[ 0.64839926,  0.33085869],
       [ 0.3211397 ,  0.597861  ],
       [ 0.15587208,  0.06602003]]), array([ 0.34566183,  0.3584914 ]))

Itā€™s not quite sRGB, but itā€™s pretty close other than the whitepoint being way off from sRGB

So the v2 outputs are a quite restricted gamut. Since itā€™s using the LinearizationTable it has no way to represent out-of-gamut colors with negative numbers.

20

In DT if you switch the input profile to linear rec709 and turn off the WB module then you get thisā€¦no other editsā€¦

P1210712_ON1v2
P1210712_ON1v24592Ɨ3448 4.76 MB

image

1 Like
21

Thatā€™s pretty close. Rec709 and this DNG (Edit: Previously said rec2020 which was REALLY WRONG - what a typo!) have nearly the same primaries. Iā€™m guessing that under the hood the DNG linearizationTable is still used despite declaring it ā€œlinearā€, unless that LinearizationTable is, itself, linear, in which case my thoughts are ā€œWTF?ā€

The closest match to the primaries appears to be NTSC, but again - not quite, and it has a whitepoint of D50.

I would consider this a significant regression in behavior for the ā€œnewā€ ON1 behavior - itā€™s now gamut restricted to approximately the sRGB gamut. Not sure if ON1 has any options to disable this undesirable behavior.

22

Ya I have ON1 but donā€™t use it that muchā€¦

The only places that you can choose color profiles are in the preferences for default transfer format and colorspace of which DNG is not a type.

image

Or when you export

image
image702Ɨ633 70.2 KB

Here if selecting DNG no optionsā€¦

image

I wonder if the result would be any different if the file was opened in ON1 and not using one of their profilesā€¦they have a linear option which is what I use to avoid their color profiles??

image

I would think this would be the best way to do it anyway if you are passing along to RT for further editing??

23

I did some experimenting as I have ON1ā€¦so if you load your image in RT and turn off the WB module and set the abstract profile to linearā€¦you will get a nice looking imageā€¦this is easy to implement and no patch need be usedā€¦at least for now

image
image1920Ɨ1080 553 KB

EDIT as loaded if you donā€™t use those tweaks

image
image1920Ɨ1080 545 KB

24

The DNG generated by on1 v2 is indeed strange.

It should be a linear DNG (the linearization table, as the defined in the DNG spec, is used to make the data linear as the base data has been ā€œreducedā€ to improve file size compression) so there should be no notion of color space but the use of color matrix(es) as input color profile.

The strange thing is that it provides only ColorMatrix1 with an unknown CalibrationIlluminant and thereā€™s no AsShotNeutral tag so RT canā€™t know the camera suggested white balance.

Instead the on1 v1 version has the two typical ColorMatrix1/2 tags, standard CalibrationIlluminant1/2 and AsShotNeutral tag.

I think that RT should be made able to handle it without any user workaround and the linear_table patch is a step in that direction.

Now Iā€™m not sure how to detect the camera white balance as looks like it isnā€™t provided by any known standard tag (Iā€™m not sure how other closed source softwares handle this kind of dng).

25

From what I saw the WB is somehow messed up for sureā€¦in both DT and RT the best result was to turn it offā€¦ So maybe in RT as this is your toolā€¦ Turn it off, add the linear abstract profile and then you have a pretty good start. Now maybe you can use the vectorscope and if you need to tweak wb do it with curves???

26

It looks like ON1 is nuking any white balance metadata - and in fact is also performing white balance internally, since the white point of the colorspace defined by ColorMatrix1 is consistent with D50 - which no real camera has as the camera-native whitepoint. Is there a setting in the ON1 version that generates these ā€œnewā€ DNGs to perform white balance?

Theyā€™ve basically done white balance prescaling and some sort of gamut transform into what is an approximation of sRGB with an altered whitepoint. (Normal sRGB has a D65 whitepoint, this has a D50 whitepoint and slightly altered primaries too.)

1 Like
27

Looks the same to me. So this isnā€™t the ā€œoriginalā€ (after demosaic+denoise+sharpen) linear data anymore. i.e. If you try to use an adobe DCP camera profile for the camera everything is messed up.

Iā€™m also curious if this is the default behavior, or the user just generated a dng of an already worked file. The on1 v1 behavior look like a standard demosaic+denoise+sharpen like other softwares like DxoPureRaw do and I think that this should also be a default available also for v2.

28

Based on the discussion, I suspect someone at ON1 messed up. sRGB is also a spaghetti western to mix a few puns. No two sRGB profiles are the same. Best to avoid them in intermediary files. Thus,

RT maybe shouldnā€™t be made to do anything to cater to this special breed of DNG.

In the meantime, @priort says he has ON1. Perhaps he could tell us more about it and the types of DNGs it could produce. Perhaps @JamesMorgan could find a way to make them play nice.

PS - another option is to reach out to ON1. I donā€™t know how open they would be about their softwareā€¦

29

I canā€™t see any way to tweak things in ON1 but I think it would be a bugā€¦

I am not at the pc with that on it now but I guess I could export a DNG with and without NoNoise to see if that is handled the same or they are differentā€¦

For now using the abstract profile in RT seems to do the trickā€¦set it to linear and turn off WB and you get a pretty good imageā€¦