Black borders around files from Adobe DNG converter

Software RawTherapee
1

Hi, I recently made a post about Sony Lossless Compressed ARW files apparently not being supported by either the main release, or the dev build of RT.

I see that this is a known issue and is a work in progress, but probably not a high priority.

I also did a search of the forum and found this post, How to remove border around image? - #7 by Morgan_Hardwood but it dosen’t seem to help my issue

I am looking for a work around to be able able to edit photos that I shot previously with lossless compressed raw.

I have tried manually passing the photos through the Adobe DNG converter, and this appears to be fine for Lossy raw files, but when doing this with lossless raw files, RT (both 5.8 and dev) is adding a black border along the bottom and right edges of the image.

This is occuring with both the default settings in Adobe DNG converter, and with custom settings, making sure that liner (demosaiaced) setting is turned OFF.

Obviously the DNGs are working perfectly in adobe software (lightroom and ps).

Something probably related, but not specific to RT, is that when these DNG files are opened in FastRawViewer (powered by libraw) the resolution is being displayed as 53mpx, but after loading another image and returning, it is displaying as 49mpx, this leads me to believe that this is either an issue with the RAW format causing the DNGs to be weird, causing problems with both libraw and RT, or something weird with how the Adobe DNG converter is treating these raw files.

Its also worth noting that when opening ARW files in FastRawViewer the file size is initially displayed as 50.2pmx before dropping down to 49.8mpx.

I don’t think that the default DNG settings are demosaicing the file, as I still have full access to the demosaiacing tab in RT, but included it just for completeness sake

I understand that this is likely a problem with the DNG format, and not necessarily with RT, but maybe someone might know a better workaround?

Original Raw File: _DSC6237.ARW (82.1 MB)
Default Adobe DNG settings: 20220329_001072_DSC6237.dng (67.5 MB)
Turned off Liner DNG: Custom_DSC6237.dng (66.3 MB)

2

RT has a border parameter under the raw tab where you could shrink the image to crop the border. Usually this is automagical but when we have unusual or not yet defined use cases, then you need to do it manually.

image

However, you can’t adjust only one side of the border using this method.

3

Sony lossless ARWs are somewhat strange in that sense.

I would’ve thought the DNG Converter would take care of that completely unnecessary cruft in a better way (i.e. to make them look exactly like uncompressed or lossy), but it just updates the DefaultCrop settings - not incorrect, but you have to handle it differently obviously.

ImageWidth                      : 8704
ImageHeight                     : 6144
DefaultCropOrigin               : 12 12
DefaultCropSize                 : 8640 5760
ActiveArea                      : 0 0 6144 8704

Seems RT does nothing OOTB w/ DefaultCrop tags?

4

For completeness, compare to DNG conversion of uncompressed/lossy A1 ARW:

ImageWidth                      : 8672
ImageHeight                     : 5784
DefaultCropOrigin               : 12 12
DefaultCropSize                 : 8640 5760
ActiveArea                      : 0 0 5784 8672

WTH Sony?! (and Adobe, but less so…)

5

@kmilos wow, and it only took Sony 10 years and an 8 thousand dollar camera to give us this broken lossless compressed option. Great cameras, shitty software.

Do you have any suggestions about what I could do until a fix is added? I know that shooting lossy or uncompressed will avoid this in the future, but is there a manual / semi automated work around to work on files shot retroactively? I’m thinking to setup a profile that would automatically crop the black borders?

1 Like
6

I guess this is the way to go, but I have no idea how to do that in RT, sorry. I’m sure someone will chime in soon now that the root of the problem is clear.

An not only that, their lossless compression scheme (and resulting ratio) is really not that fundamentally different than the DNG lossless compression scheme (4 channel lossless JPEG SOF3 instead of 2 channel one), so they could’ve just gone with that years ago, and saved a lot of developer hairs among other things…

7

https://github.com/Beep6581/RawTherapee/blob/59a36d8f8a79918749a9da298bf559a0cedc4cf8/rtengine/camconst.json#L4

Looks like the forum isn’t giving a preview. Here is the top of the file with some instructions:

DO NOT EDIT THIS FILE!

All changes made here will be lost on software update. If you want to add custom values or change existing ones, create
a "camconst.json" file next to your personal "options" file. Its values will then override and/or complete the ones of
this file.

If you add values for your own camera, consider sharing them with the community for everyone's benefit. You can do so
via our forum or GitHub page:
- https://discuss.pixls.us/c/software/rawtherapee
- https://github.com/Beep6581/RawTherapee/issues

Value priority:
1. If you set the dcraw matrix in your user camconst.json file for a camera entry which already exists in RT's file
    (same camera, same model), your values will replace RT's ones.
2. If you set the black level values in your user camconst.json file for a camera entry which already exists in RT's
    file, your values will replace RT's ones, even if RT's ones are more complete and/or detailed. You might want to
    copy/paste RT's levels first (if provided) to your user's file and complete/modify them.
3. Same for the white levels, independent of the black levels.

This file is in JSON format and contains camera constants which RawTherapee uses when parsing raw files.
8

@Calandracas

I just experimented a bit with the camconst.json file and your Default Adobe DNG (67.5 MB) example. Based on that it seems that if you add one line you should be good to go:

    { // Quality C
        "make_model": "Sony ILCE-1",
        "raw_crop": [ 0, 0, 8672, 5784 ], // ADDED BY HAND
        "dcraw_matrix": [ 8161, -2947, -739, -4811, 12668, 2389, -437, 1229, 6524 ] // DNG v13.2
    },

Look for the Sony ILCE-1 and add the “raw_crop”: … line. I do think you need to restart RT to make it take effect.

I ended up using the numbers that @kmilos posted (ImageWidth: 8672 and ImageHeight: 5784). You might need to tweak these, I only had 1 example to test with. I think these numbers are OK, though. 4 pixels will be removed due to the Demosaicing border settings which should be enough of a margin.

One important thing: If you reinstall RawTherapee there’s a fair chance that the camconst.json will be overwritten again (new data and all that), so you might need to re-apply this change if you do.

3 Likes
9

A user can make a camconst.json file of their own, put it in the right place, and it’ll survive RT updates:

http://rawpedia.rawtherapee.com/File_Paths

Scroll down to “config”

1 Like
10

Yes, that is the reason I posted the link from GitHub. Specifically:

All changes made here will be lost on software update. If you want to add custom values or change existing ones, create a “camconst.json” file next to your personal “options” file. Its values will then override and/or complete the ones of this file.

1 Like
11

This solution worked perfectly, thank you and everyone else for your help

I am currently running a dev build, and will be sure to remember to check the camconst.json file when updating between versions

12

@Calandracas

If you do opt for using a personalized camconst.json file do make sure to also check the main one when rebuilding/installing. The new version might have the correct, full set of settings implemented for this camera type which would make the personal copy unwanted.

So, whichever method you choose, you need to check the file(s)…

13

So from what I can tell by everyone’s input, is that depending on whether you look at the ARW or a DNG different crops are needed? For sure that is going to give problems, as we currently don’t support multiple ‘versions’ in camconst.json for a single camera.

I would be happy to include @Jade_NL’s modification, but only if it makes sense in light of the above caveat.

1 Like
14

Nope. Different pixel array sizes depending whether it is “new” lossless ARW (and its converted DNG) vs traditional uncompressed/lossy ARW (and their converted DNGs), from the same camera (ILCE-1 and ILCE-7M4 so far). The crops are the same for all.

15

And just to add onto what @kmilos has said, I’m working with DNG because RT does not yet support Sony’s new lossless compressed raw format used by the ILCE-1 and ILCE-7M4. Doing a DNG conversion is the only way to open these files in RT

16

Funny thing, this is the second time this week I’ve seen a report of software (FastRawViewer included!) not handling Sony lossless ARW properly: Sony a7IV has broken apsc crop mode with lossless raws: Sony Alpha Full Frame E-mount Talk Forum: Digital Photography Review

One thing to be careful of with that camconst change is what happens if someone enables APS-C mode.

17

That is why using the custom file is better. I do this with G’MIC too: I have a text editor tab with custom files open and read to change as needed. Not everyone’s cup of tea but makes things easier. :wink: