How to create synthetic RAW image?

Okay, try this:

gmic [input_image] -mul 0.25 -rgb2bayer 0 -o [output_image]

-mul seems to do the proper scaling.

Edit: still not right, still need to apply the gamma:

gmic [input_image] -mul 0.25 -adjust_colors 0,0,-100 -rgb2bayer 0 -o [output_image]

There, that produces a reversed image with a histogram approximate to that of the raw data.

How did you do that? Here is a screen recording showing what exactly I do:

https://goo.gl/BvSEY9

Thanks Glenn. Could you just clarify please about input and output, i.e. is the input_image the clean TIFF which I create manually in Photoshop and is output_image a TIFF or DNG?

ETA: I tried both. If I set the output to be a .dng file - RT shows nothing, blank. If it is .TIFF which I convert after that using exiftool - it looks like this:

One more thing which I notice (and seen on the video I posted): there is no way to select demosaic method on any of the DNG files generated this way. RT simply doesn’t have this menu for these files.

That means the data is already demosaiced or is not detected as raw data.

Input is the TIFF created in the drawing program. Output is also TIFF, with the intent being to use that to insert the image array into a DNG or other raw format file.

Others seem to have the chops to help with the DNG insertion; I’ve played with it a bit with no luck.

Why? I am following the instructions given here.

What happens when you do what I do in the video and when you add your command before creating the DNG with exiftool? Why doesn’t RT detect this mosaic?

Don’t know! But if the demosaic methods in rt are not available, it’s a clear indicator that the data is already demosaiced.

Actually it is there when I zoom the file generated through Glenn’s method but I suppose it is not treated as mosaiced for some other reason. Is it not necessary to somehow include in DNG (metadata or otherwise, I have no idea) that the data is mosaiced? Perhaps that would tell the raw converter how to approach it properly.

Sorry, my comment was a bit wrong: If the demosaic methods in rt are disabled it doesn’t mean that the image is already demosaiced. But it does mean that the underlying raw decoder (a modified version of drcaw in case of rt) does not detect that image as undemosaiced raw file. There are a lot of possible reasons for this behaviour, but without an example file I can’t answer that.

This is how I generate the files:

[/tmp/download/raw]: rm *
[/tmp/download/raw]: convert hald:12 -depth 16 -colorspace RGB -gravity NorthWest -splice 4x4 -gravity SouthEast -splice 4x4 foo.tif
[/tmp/download/raw]: gmic foo.tif -mul 0.25 -adjust_colors 0,0,-100 -rgb2bayer 0 -o foo.tif
[gmic]-0./ Start G'MIC interpreter.
[gmic]-0./ Input all frames of TIFF file 'foo.tif' at position 0 (1 image 1736x1736x1x3).
[gmic]-1./ Multiply image [0] by 0.25.
[gmic]-1./ Adjust colors of image [0], with brightness 0, contrast 0, gamma -100, hue shift 0, saturation 0 and value range [0,0].
[gmic]-1./ Transform image [0] to a RGB-Bayer monochrome grid, starting from pattern '0'.
[gmic]-1./ Output image [0] as tif file 'foo.tif', with pixel type 'float', no compression and bigtiff support (1 image 1736x1736x1x1).
[gmic]-1./ End G'MIC interpreter.
[/tmp/download/raw]: exiftool -DNGVersion=1.4.0.0 -PhotometricInterpretation='Linear Raw' foo.tif
    1 image files updated
[/tmp/download/raw]: mv -v foo.tif Hald_CLUT_Identity_12.dng
'foo.tif' -> 'Hald_CLUT_Identity_12.dng'

Then I open the DNG in RT.

I copy-pasted straight from RawPedia. Using RT5.1 without any custom tweaks - vanilla camconst.json.

morgan@sabayon /tmp $ convert -version
Version: ImageMagick 6.9.8-6 Q16 x86_64 2017-05-31 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP 
Delegates (built-in): bzlib cairo fftw freetype jng jpeg lcms ltdl openexr pangocairo png rsvg tiff wmf x xml zlib
morgan@sabayon /tmp $ convert hald:12 -depth 16 -colorspace RGB -gravity NorthWest -splice 4x4 -gravity SouthEast -splice 4x4 foo.tif
morgan@sabayon /tmp $ exiftool -DNGVersion=1.4.0.0 -PhotometricInterpretation='Linear Raw' foo.tif
    1 image files updated
morgan@sabayon /tmp $ mv -v foo.tif Hald_CLUT_Identity_12.dng
'foo.tif' -> 'Hald_CLUT_Identity_12.dng'
morgan@sabayon /tmp $ ~/rt_5.1/rawtherapee
  1. Why does your RawTherapee look the way it does? Since that is not a look we support, it indicates you’ve been changing things yourself - perhaps you changed more than just the look?
  2. Have you edited your camconst.json file?
  3. Are you using a Q16 version of ImageMagick? “convert -version” to find out.
  4. Your histogram looks different/incorrect. Apply the “(Neutral)” profile - does it look correct now?
  5. Upload your DNG file. Here’s mine: https://filebin.net/lnwe1um8wkgli7gy/Hald_CLUT_Identity_12.dng

I have created/edited ~/.config/RawTherapee/my-theme-GTK3-20_.css because I couldn’t find a way to use the system colors for the interface (which was an option available in previous version). The default theme - medium gray on dark gray - is too exhausting for the eyes. If you know a way to use the system (Plasma5 desktop) colors, I would be glad to know that.

Never heard of it.

convert -version                                                                                                                           
Version: ImageMagick 6.8.8-1 Q16 x86_64 2017-06-06 http://www.imagemagick.org                                                                               
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC                                                                                                   
Features: DPC Modules OpenMP                                                                                                                                
Delegates: bzlib cairo djvu fontconfig freetype jng jp2 jpeg lcms ltdl lzma openexr pangocairo png rsvg tiff webp wmf x xml zli

That’s because the DNG image is incorrect, I don’t know why. (Neutral) profile doesn’t change anything.

Yours looks fine when opening in RT (no mosaic/demosaic though). Here is mine:

https://goo.gl/4Rq8Zw

Is there any problem with my ImageMatick? Looking at the tiff which convert generates I see it fine. Is it possible that exiftool doesn’t do the next step right?

Yours has
-TIFF-IFD0:FillOrder=Reversed, mine has
-TIFF-IFD0:FillOrder=Normal
Maybe that’s the problem, maybe not. I don’t have time to investigate further.

After some googling I think I nailed it:

convert hald:12 -depth 16 -colorspace RGB -gravity NorthWest -splice 4x4 -gravity SouthEast -splice 4x4 -define tiff:fill-order=msb-to-lsb foo.tif

Thanks @Morgan_Hardwood or sharing your findings.

I wish I knew how to add mosaic to the raw data.

Hi,
I understand that you would like to get to a Raw editing default method without variables like lens imperfections. However, this default will always be at least Sensor and In-camera Raw Compression dependent to begin with if you want to include the use of different algorithms in your test. You will never get to the perfect default editing method that works with any sensor and any in-camera compression. This is one of the reason for the existence of the different algorithms. Furthermore, what use is a default Camera-Editing method without including a lens? In real life there will always be at least one lens (not to mention a variety of them) in the game and a default editing method that did not include a lens is obviously not going to give you the best result once you include a lens.
Finally, if you “merely” want to test editing methods that work for any image, you will have to exclude the demosaicing step and just start your test from there on. I that case; why don’t you create a 16bit Tiff in a way that applies to your testing needs. Give it perfect lines, pure colors, build in contrast levels to your needs, etc. and convert it into a .dng. No real life photograph will ever match such perfection.
Hope this was helpful. Cheers!

I understand what you are saying but in my test I still want to avoid the effect of lens and to have the mosaic in the dng. Lens is something which is inevitable but the choice of raw converter is not, so by isolating the effect of lens I can focus on other things without having influence by the variations which different lens models add. As for compression - I am not sure what exactly you mean. Could you please expand? In case you mean some compression applied by the camera software (or hardware) that may be better to exclude too as it is also inevitable. In my test I am willing to focus entirely on the raw conversion without any additional post processing such as color profile correction, lens aberration correction, noise filtering, sharpening etc. Of course if you know something about if certain raw converters use different demosaic or WB algorithms which depend on the camera model, I would be interested to learn more about that too.

Hi George,

As I understand it, you focus on having an as pure as possible file to begin with. No problems here. However, considering that the files that come out of your camera have been processed by the in-camera software that translates the data from your sensor into a Raw file readable by a Raw converter, the camera software is a major factor in determining what you get out of the file at the end. Therefor, what you intend to obtain through tests with an as pure as possible default file will only work with real life out of camera images when the test is done based on images coming out of that particular camera. Thus, if you create a default method that works on sensor A, it does not mean that it works on images shot through sensor B, and I am not even considering lens variables here. That is why I recommended the creation of a as high as possible tiff file that excludes all of these variables, lens and sensor.
As far as I have seen, but I do not know everything, Raw Therapee is a pretty good program, for you can choose between a variety of algorithms for demosaicing,and, furthermore, it offers an infinity of options other Raw editors I know do not include. But, to be honest; Raw Theree, at least on my Windows platform is a pain in the a
 It crashes frequently and because I try to sustain myself through my work, I have to produce, and, of course, crashing programs do not encounter my needs. This doesn’t mean your computer can’t necessarily handle the program though. I would say; try it, if you don’t already know it. Raw Therapee is Open Software (that’s probably why it crashes), so; it is free and easy to give it a try.
Whatever you discover, please let me know. I am as interested in the possible results as you are!

Cheers!

Which version of RT do you use (I guess you mean RawTherapee and not Raw Theree :stuck_out_tongue: ) ?