Raw and DNG files do not look the same

I think we don’t know what DNG converter is doing to the data behind the scene. Perhaps nothing! Only Adobe knows. In peculiar, all exif data are shuffled, some modified. e.g. Lens cannot be recognised in the DNG file.
So I think the only way to be sure to obtain the very same output from ORF and DNG is to use an Adobe product.

Hi all,
I already posted this on github, but since probably not many people are actively reading RT bug reports, I’ll repost here :slight_smile:.
I’m not sure I fully understand what is the problem experienced by the OP, so I tried this little experiment:

$ rawtherapee-cli -j -p DNGAuto.pp3 -o orf.jpg -c P1015545.ORF
$ rawtherapee-cli -j -p DNGAuto.pp3 -o dng.jpg -c P1015545.dng

Then I loaded orf.jpg and dng.jpg as layers in GIMP, and set layer mode to “Difference”. Here is what I get:

Is this the difference in output we are talking about? If not, could someone please clarify? Thanks!

And just to put this in context:

$ rawtherapee-cli -j93 -p DNGAuto.pp3 -o orf-93.jpg -c P1015545.ORF
$ rawtherapee-cli -j90 -p DNGAuto.pp3 -o orf-90.jpg -c P1015545.ORF

Gives this:

Yes! That occurred to me as well :slight_smile:.

Hi,

IMHO your problem is the DNG file - everybody assumes that converting to DNG is a lossless conversion but it isn’t. They do normalize some of the data contained in the original RAW file (most notable the black point data has often been mangled and even at times falsely interpreted and some of the masked rows and columns of the sensor were previously discarded) and transform them to a common denominator by embedding a camera profile which must be adhered to by the development software. The original RAW file OTOH will be interpreted completely by the RAW converter which will apply it’s own camera profile.

If you were to extract the camera profile from the Adobe LR or RAW converter library and instruct RT to take that one to convert the image I’m willing to bet that the results will be practically identical.

I made three test using the color picker in photoshop:

test1- rawtherapee: orf and dng are always different ( even without colormatrix or the same colormatrix)
test2-cameraraw: orf and dng match perfectly
test3-darktable for windows: orf and dng match perfectly

My camera can save both DNG and a vendor specific raw format PEF I’m not sure it’s useful but if you want to research if it’s a general bug you can find files here: Filebin | 005fe0bioakmcuhp

DNG, PEF processed twice once using Neutral and once with the standard Pentax dcp applied. It’s really sunny here and i’m doing this whilst caring for a sick child (nothing serious) so I’ve not analysed the results. The light changes a bit unfortunately. I can do static shots with off-camera flash if it’s useful.

Edit: after looking closer at the images I see the light (and background due to wind) changes to much for the images to be useful.

Edit2: Looks like Neutral profile does some things different with PEF and DNG after all.

It’s conceivable that the PEF is a bit brighter pushing it over the edge but I doubt it as the DNG is also blown out.

1 Like

@nosle (and all) thanks for the help. We are already digging into this, please check RAW files and DNGs obtained from them give different results at identical settings. · Issue #4129 · Beep6581/RawTherapee · GitHub if you are interested

Sorry I don’t have a github account but following the discussion I’d like to add that I would find it useful if RT could make full use of all DNG colour settings like matrices WB etc. In Darktable the DNG embedded matrix becomes an option if available. It’s sometimes useful to be able to replicate the camera settings as closely as possible. In RT you have to get hold of the settings from the Adobe software when it’s already embedded in the DNG.

1 Like

I’m seeing this too with RawTherapee 5.4 and Adobe DNG converter 10.4 using a Nikon D600. The DNG files appear darker and their highlights are pink. For this example I’m using neutral settings (built-in preset in RT). I’ve chosen cropped compressed raw format in order to reduce file sizes for everyone and thus hopefully speed up debugging.

Here I’ve shot the same photo with longer and longer exposure to provoke over exposure so that you can gradually see the problem getting more and more obvious. All other camera settings are fixed. I hope this helps.

http://www.robotplanet.dk/t2/rawtherapee5.4_dngconv10.4_pink_highlights.7z

Now, I noticed that the difference in brightness is exactly 2 EV on the exposure compensation slider, so I tried upping all the .dng files to +2 and this makes things far more even. Except, a bit of pink is still seeping through. So they’re still not quite the same. (I’ve left out the raw files this time, as they’re already provided in the above file.)

http://www.robotplanet.dk/t2/compensation_2ev.7z

Test using the latest dev build.

Hmm, that turned out not to be super easy to locate. I tried both the official site and google.

The official web site has a drop down where you can select developer versions. The latest is from 2013-06-15. Then there’s this thread also listing old versions: Download RawTherapee development builds
This page lists even older versions: Release Continuous build · aferrero2707/RawTherapee · GitHub or this link which lists versions from 2012: RawTherapee Nightly Builds : “RawTherapee PPA Packagers” team … and there’s also this github page which links back to some of the above pages: GitHub - Beep6581/RawTherapee: A powerful cross-platform raw photo processing program

Where do I get the latest developer binaries for windows?

Developer binaries for windows are often linked in this forum by @gaaned92

2 Likes

Ahh… the filename “RawTherapee_5.4_WinVista_64.zip” lead me to belive it was just the same 5.4 that I already have, but I can see the date on the .exe file is much later. My bad! Thanks for clarifying.

I just tried a beta which turned out to be from 20-mar-2018 (I don’t understand the filename convention, so I’ll have to try different ones): Same result on that one…

Are you using the dev version? I.e., RawTherapee-dev-5.4-524-g3bfd903d.AppImage.

Apparently not. The links points to “nightly builds” which I thought were dev builds. This time I tried “RawTherapee_newlocallab_5.4-772-ge65c6786f_WinVista_64.zip” which also gives the same result. These are the choices I seem to have on that nightly builds page:

RawTherapee_32b-tiff-output-cli_5.4-481-gd217dcd2c_WinVista_64.zip
RawTherapee_5.4_WinVista_64.zip
RawTherapee_autowblocal_5.4-637-gd40ddf5c4_WinVista_64.zip
RawTherapee_blackwhiteauto_5.4-301-g69dd5fbf5_WinVista_64.zip
RawTherapee_cat02wb_5.4-521-g64e899a71_WinVista_64.zip
RawTherapee_dev_5.4-488-gc30f03ea0_WinVista_64.zip
RawTherapee_newlocallab_5.4-772-ge65c6786f_WinVista_64.zip
RawTherapee_testoutputprofile_5.4-572-g8862f0fb2_WinVista_64.zip
RawTherapee_waveletnew_5.4-384-gca8598fb5_WinVista_64.zip

The other links offers…

RawTherapee-git-dev-20171217_1337-glibc2.14.glibc2.14-x86_64.AppImage
RawTherapee-git-dev-20180115_1806-glibc2.14.glibc2.14-x86_64.AppImage
rawtherapee-w64-20171120_2059-git-dev.zip
Source code (zip)
Source code (tar.gz)

I can’t help noticing “2017” there. And “20180115” which could be “2018-01-15” which is even older than what I have. Perhaps you could send me a direct link to the specific file you want me to download?

is what you want for Windows

Wait, got it! … but … RawTherapee_dev_5.4-488-gc30f03ea0_WinVista_64.exe gives me the same result again.

It did fix another issue that I had though (the folder tree now sees new folders automatically, which is super convenient, so thanks for fixing that guys!)