Why? I am following the instructions given here.
What happens when you do what I do in the video and when you add your command before creating the DNG with exiftool? Why doesn’t RT detect this mosaic?
Don’t know! But if the demosaic methods in rt are not available, it’s a clear indicator that the data is already demosaiced.
Actually it is there when I zoom the file generated through Glenn’s method but I suppose it is not treated as mosaiced for some other reason. Is it not necessary to somehow include in DNG (metadata or otherwise, I have no idea) that the data is mosaiced? Perhaps that would tell the raw converter how to approach it properly.
Sorry, my comment was a bit wrong: If the demosaic methods in rt are disabled it doesn’t mean that the image is already demosaiced. But it does mean that the underlying raw decoder (a modified version of drcaw in case of rt) does not detect that image as undemosaiced raw file. There are a lot of possible reasons for this behaviour, but without an example file I can’t answer that.
This is how I generate the files:
[/tmp/download/raw]: rm *
[/tmp/download/raw]: convert hald:12 -depth 16 -colorspace RGB -gravity NorthWest -splice 4x4 -gravity SouthEast -splice 4x4 foo.tif
[/tmp/download/raw]: gmic foo.tif -mul 0.25 -adjust_colors 0,0,-100 -rgb2bayer 0 -o foo.tif
[gmic]-0./ Start G'MIC interpreter.
[gmic]-0./ Input all frames of TIFF file 'foo.tif' at position 0 (1 image 1736x1736x1x3).
[gmic]-1./ Multiply image [0] by 0.25.
[gmic]-1./ Adjust colors of image [0], with brightness 0, contrast 0, gamma -100, hue shift 0, saturation 0 and value range [0,0].
[gmic]-1./ Transform image [0] to a RGB-Bayer monochrome grid, starting from pattern '0'.
[gmic]-1./ Output image [0] as tif file 'foo.tif', with pixel type 'float', no compression and bigtiff support (1 image 1736x1736x1x1).
[gmic]-1./ End G'MIC interpreter.
[/tmp/download/raw]: exiftool -DNGVersion=1.4.0.0 -PhotometricInterpretation='Linear Raw' foo.tif
1 image files updated
[/tmp/download/raw]: mv -v foo.tif Hald_CLUT_Identity_12.dng
'foo.tif' -> 'Hald_CLUT_Identity_12.dng'
Then I open the DNG in RT.
I copy-pasted straight from RawPedia. Using RT5.1 without any custom tweaks - vanilla camconst.json.
morgan@sabayon /tmp $ convert -version
Version: ImageMagick 6.9.8-6 Q16 x86_64 2017-05-31 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib cairo fftw freetype jng jpeg lcms ltdl openexr pangocairo png rsvg tiff wmf x xml zlib
morgan@sabayon /tmp $ convert hald:12 -depth 16 -colorspace RGB -gravity NorthWest -splice 4x4 -gravity SouthEast -splice 4x4 foo.tif
morgan@sabayon /tmp $ exiftool -DNGVersion=1.4.0.0 -PhotometricInterpretation='Linear Raw' foo.tif
1 image files updated
morgan@sabayon /tmp $ mv -v foo.tif Hald_CLUT_Identity_12.dng
'foo.tif' -> 'Hald_CLUT_Identity_12.dng'
morgan@sabayon /tmp $ ~/rt_5.1/rawtherapee
- Why does your RawTherapee look the way it does? Since that is not a look we support, it indicates you’ve been changing things yourself - perhaps you changed more than just the look?
- Have you edited your
camconst.jsonfile? - Are you using a Q16 version of ImageMagick? “
convert -version” to find out. - Your histogram looks different/incorrect. Apply the “(Neutral)” profile - does it look correct now?
- Upload your DNG file. Here’s mine: https://filebin.net/lnwe1um8wkgli7gy/Hald_CLUT_Identity_12.dng
I have created/edited ~/.config/RawTherapee/my-theme-GTK3-20_.css because I couldn’t find a way to use the system colors for the interface (which was an option available in previous version). The default theme - medium gray on dark gray - is too exhausting for the eyes. If you know a way to use the system (Plasma5 desktop) colors, I would be glad to know that.
Never heard of it.
convert -version
Version: ImageMagick 6.8.8-1 Q16 x86_64 2017-06-06 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules OpenMP
Delegates: bzlib cairo djvu fontconfig freetype jng jp2 jpeg lcms ltdl lzma openexr pangocairo png rsvg tiff webp wmf x xml zli
That’s because the DNG image is incorrect, I don’t know why. (Neutral) profile doesn’t change anything.
Yours looks fine when opening in RT (no mosaic/demosaic though). Here is mine:
Is there any problem with my ImageMatick? Looking at the tiff which convert generates I see it fine. Is it possible that exiftool doesn’t do the next step right?
Yours has
-TIFF-IFD0:FillOrder=Reversed, mine has
-TIFF-IFD0:FillOrder=Normal
Maybe that’s the problem, maybe not. I don’t have time to investigate further.
After some googling I think I nailed it:
convert hald:12 -depth 16 -colorspace RGB -gravity NorthWest -splice 4x4 -gravity SouthEast -splice 4x4 -define tiff:fill-order=msb-to-lsb foo.tif
Thanks @Morgan_Hardwood or sharing your findings.
I wish I knew how to add mosaic to the raw data.
Hi,
I understand that you would like to get to a Raw editing default method without variables like lens imperfections. However, this default will always be at least Sensor and In-camera Raw Compression dependent to begin with if you want to include the use of different algorithms in your test. You will never get to the perfect default editing method that works with any sensor and any in-camera compression. This is one of the reason for the existence of the different algorithms. Furthermore, what use is a default Camera-Editing method without including a lens? In real life there will always be at least one lens (not to mention a variety of them) in the game and a default editing method that did not include a lens is obviously not going to give you the best result once you include a lens.
Finally, if you “merely” want to test editing methods that work for any image, you will have to exclude the demosaicing step and just start your test from there on. I that case; why don’t you create a 16bit Tiff in a way that applies to your testing needs. Give it perfect lines, pure colors, build in contrast levels to your needs, etc. and convert it into a .dng. No real life photograph will ever match such perfection.
Hope this was helpful. Cheers!
I understand what you are saying but in my test I still want to avoid the effect of lens and to have the mosaic in the dng. Lens is something which is inevitable but the choice of raw converter is not, so by isolating the effect of lens I can focus on other things without having influence by the variations which different lens models add. As for compression - I am not sure what exactly you mean. Could you please expand? In case you mean some compression applied by the camera software (or hardware) that may be better to exclude too as it is also inevitable. In my test I am willing to focus entirely on the raw conversion without any additional post processing such as color profile correction, lens aberration correction, noise filtering, sharpening etc. Of course if you know something about if certain raw converters use different demosaic or WB algorithms which depend on the camera model, I would be interested to learn more about that too.
Hi George,
As I understand it, you focus on having an as pure as possible file to begin with. No problems here. However, considering that the files that come out of your camera have been processed by the in-camera software that translates the data from your sensor into a Raw file readable by a Raw converter, the camera software is a major factor in determining what you get out of the file at the end. Therefor, what you intend to obtain through tests with an as pure as possible default file will only work with real life out of camera images when the test is done based on images coming out of that particular camera. Thus, if you create a default method that works on sensor A, it does not mean that it works on images shot through sensor B, and I am not even considering lens variables here. That is why I recommended the creation of a as high as possible tiff file that excludes all of these variables, lens and sensor.
As far as I have seen, but I do not know everything, Raw Therapee is a pretty good program, for you can choose between a variety of algorithms for demosaicing,and, furthermore, it offers an infinity of options other Raw editors I know do not include. But, to be honest; Raw Theree, at least on my Windows platform is a pain in the a… It crashes frequently and because I try to sustain myself through my work, I have to produce, and, of course, crashing programs do not encounter my needs. This doesn’t mean your computer can’t necessarily handle the program though. I would say; try it, if you don’t already know it. Raw Therapee is Open Software (that’s probably why it crashes), so; it is free and easy to give it a try.
Whatever you discover, please let me know. I am as interested in the possible results as you are!
Cheers!
Which version of RT do you use (I guess you mean RawTherapee and not Raw Theree 
) ?
well… 
Yeah sure, I meant Raw Therapee, I wrote it wrong of course…
Anyway, I uninstalled it a few weeks ago, but I was using the latest version.
I am interested to test how different raw converters process the same raw image data regardless of camera/lens make/model. That’s why I am not really interested in testing the combination of in-camera software + raw converter but only raw converter. Of course different cameras have different qualities but I don’t think that WB, gamma curve or demosaic are camera dependent algorithms (in case of same bayer pattern). Correct me if I am wrong.
What I am asking is not directly related to RT per se although it will be part of the test of course.
I am using RT mainly on Linux but I have it on Windows too. Never crashed.
Some time ago I made a very simple test, comparing different raw converters:
This was based on using real raw file, from a camera. However it is fairly limited and obviously testing with lens imperfections (vignette etc) or without the possibility to have a clean linear gradient it is somewhat skewed. So my idea is to use a synthetic raw so I can take it further, test not only gradients but also different other patterns.
Hi George,
It is really good to see someone “digging deep”. My sincere congrats on that!
I understood that your goal is to test Raw converters and that in order to do so you intend to exclude as many variables as possible. So far, there are no questions I guess. I might also agree to that for equal sensor principals the proper algorithms to use are probably the same. However, I have my doubts as to what extend WB or any other settings applied by cameras to their Raw files might coincide between files from different brands and/or different cameras (even within a same brand), so, considering that there might be possible differences, the test would not achieve it’s full purpose.
Furthermore, even if the above mentioned possible differences would not exist; in real life you will always have factors like for example different lens behavior in contrast or sharpness due to different lighting circumstances (even with the same lens). On top of this, these real life variables are sometimes more and sometimes less considerable. So; variables on variables here.
Now, I know that you do not want real life Raw’s, so your test would not be influenced by all those variables, How to get your hands on a Raw file without the use of a lens… might be some task. That’s why I said that as far as I see it, testing a file specifically created for the purpose might be the most practical way to test how a program acts on whatever file you feed it. But, even so, I don’t see how such a test might possibly be giving you consistent answers about what to do with real Raw files, using real lenses and shooting real life situations. Personally, after years of struggling, I have long ago come to the conclusion that every shot in different circumstances needs different adjustments if you want to get the most out of it. God, how I wish there were default settings I could use!
Hope you prove my words wrong though and in case you ever reach a verdict (whatever it might be), please let me know. I am very interested!
Cheers and good luck!
The question how to make the synthetic raw mosaiced (using different patterns) remains.
I am not sure I understand correctly. I approach this considering that raw files are not white balanced, i.e. they are in the so called uniwb. Other WB is applied during raw conversion. Of course different cameras may have different uniwb but that has to do the hardware of the sensor (CFA etc) not with the built-in camera software which creates jpg previews. Or what are you talking about?
But this is the process of exposure. I am testing the next step - the processing of the exposed and recorded data. The way the raw converter works has nothing to do with the scene, its dynamic range, lens etc. (unless it is some “smart” software which makes “auto magic” and gives no control). If those are to be tested, one should fix the other thing - the software - which is what camera testing labs do. We just can’t have variables on variables. Only one thing at a time should be tested in order to make it objective.
Well, it is not. Someone already suggested how to do it. I just need the last piece - making it mosaiced.
As I explained - I am not testing lenses or cameras. As you see in my older test it is focused only on the very basics of raw conversion: to check how the program acts on the raster of the image, does it filter it badly, how does it handle demosaic. It clearly shows how the beloved ACR and C1 really suck. But to make it more meaningful and test further I need representative clear data, not just a shot of my monitor. Then I can control it, change it, test with or without noise etc etc…
Thanks. I hope someone can enlighten me on how to add mosaic. Then the rest will follow 