Linear to Log for Film Emulation in Darktable - Are 1D LUTs possible?

[Xavier_Bourque]

Hi,

I’m hitting a wall when trying to setup Darktable to work properly with Film Emulation 3D LUTs that expect LOG inputs.

Here’s an image to explain the workflow I would like to achieve:

It is usually possible to combine the SHAPER 1D LUT and the FILM EMULATION 3D LUT in a single .cube file but when I do that, I get the following error in Darktable’s viewer:

1D cube LUT is not supported

I’ve been trying to workaround that using various ICC profiles but I am hitting a wall.

How is everyone doing film emulation LUTs in Darktable?
Specifically LUTs that expect LOG data on input?
How do you convert from linear to log in Darktable?

Thanks for your help!

Xavier

[mikae1]

@Xavier_Bourque

How is everyone doing film emulation LUTs in Darktable?
Specifically LUTs that expect LOG data on input?

Do you have log input (i.e. stills from log video) or are you working with raw files? In the world of video LUTs are usually either technical or creative. Technical ones convert from log and then you apply a creative LUT after that. I don’t know why you’d like to use technical LUTs intended for log on raw files.

I first apply tone mapping (sigmoid or filmic), then apply creative LUTs.

[black_daveth (Dave)]

@Xavier_Bourque

It is usually possible to combine the SHAPER 1D LUT and the FILM EMULATION 3D LUT in a single .cube file

If you have two LUTs that ought to work you can just duplicate the LUT 3D module and apply them one after the other. Its useful to have a color balance rgb module in between to grade the log image underneath your creative LUT.

I used this tool to create a Rec2020 to LogC LUT: LUTCalc

[kofa (István Kovács)]

filmic rgb is basically a log mapping with a curve on top, I think. You can eliminate the additional curve by:

• setting black/white relative exposure symmetrically (e.g. +5 EV, -5 EV):
  
• the shoulder and toe (contrast in highlights/shadows) set to safe or soft
  
• and contrast reduced to the minimum:
  

If the exposure settings are not symmetrical, but you don’t go crazy, it will still stay close to log:

There is a curve explorer here, but you cannot set all filmic parameters on that page (however, you can also explore the sigmoid (log-logistic)):
https://share.streamlit.io/jandren/tone-curve-explorer

I think this article is about the original filmic (not filmic rgb), but as far as I know the basic idea of the curve is the same:

Filmic module for darktable - HackMD

[agriggio (Alberto)]

Hi,
I don’t think what you want to do is possible in darktable (but I’m happy to be proven wrong).
I’m afraid you have to use another tool, sorry.

[priort (Todd Prior)]

Doesn’t the unbreak profile module allow you to do something with log conversions??

[kofa (István Kovács)]

I think that was what Aurélien tried first, then he added filmic, to keep the pipeline in scene linear right up to the tone mapper.

[Xavier_Bourque]

@mikae1

Do you have log input (i.e. stills from log video) or are you working with raw files?

By the time pixels reach the shaper LUT they are neither ‘raw’ nor ‘log’ since everything gets converted to linear at the start of the pipeline. Input file format makes no difference.

@mikae1

I don’t know why you’d like to use technical LUTs intended for log on raw files.

Most manufacturer display LUTs and commercial film emulation LUTs expect LOG encoded values.
I want to shoot RAW, edit in LINEAR, convert to LOG and apply a LUT to that.

@mikae1

I first apply tone mapping (sigmoid or filmic), then apply creative LUTs.

Display/film emu LUTs are meant to do tonemapping, applying them after filmic/sigmoid is essentially doing double tone mapping.

Commercial film emulation LUTs expect values to be encoded a specific way (i.e. sLog3, AlexaV3LogC, RedLog3G10, ACEScc, etc…). The output of sigmoid or filmic is none of those.

@black_daveth

If you have two LUTs that ought to work you can just duplicate the LUT 3D module and apply them one after the other.

I used this tool to create a Rec2020 to LogC LUT: LUTCalc

I’ve tried LUTCalc and ociobakelut to create various intermediate LUTs.

However, the LUT3D module clamps values above 1.0. The main purpose of the SHAPER 1D LUT is to ‘compress’ our scene linear values without clamping all the highlights above 1.0. In my example, I use Sony sLog3 as the intermediate log format which will preserve hilites up to 30.42 for the 3D LUT to use and tonemap as desired.

@priort

Doesn’t the unbreak profile module allow you to do something with log conversions??

This seems to be the closest thing to a lin2log I have seen yet in Darktable. Thanks for pointing me to this!

However, I would much rather have the LUT3D module support 1DLUTs to avoid having to guess what kind of random Log is coming out of ‘filmic’ or ‘sigmoid’ or ‘unbreak input profile’.

This seems like a bizarre omission from Darktable. There is no 1DLUT support anywhere? Via ICC profiles maybe?

Thanks for everyone’s help!

Xavier

[paperdigits (Mica)]

@Xavier_Bourque

would much rather have the LUT3D module support 1DLUTs to avoid having to guess what kind of random Log is coming out of ‘filmic’ or ‘sigmoid’ or ‘unbreak input profile’.

You can add that functionality or find someone to add it for you. The latter would require a feature request on github, most likely.

[priort (Todd Prior)]

Dumb question but can the Luts be converted rather than modifying DT?? Note I said dumb

[bastibe (Bastian Bechtold)]

Perhaps a dumb question, but I’d assume a 1D LUT operates on lightness only, right? How is it different from a tone curve?

[Xavier_Bourque]

So far I was able to get some kind of reasonable result using the ‘unbreak input profile’ module. It is very tedious to setup though. Since ‘unbreak’ is a lin2log, it is possible to create the reverse log2lin in Resolve or Nuke and bake an appropriate custom 3D LUT that expects the weirdo log coming out of ‘unbreak’… This is very very far from user friendly but I got it to work.

I wonder if it’s possible to get the ‘output color profile’ module to do the lin2log instead of ‘unbreak’?

Does anyone know if the ICC modules in DT (or anywhere else) are able to deal gracefully with values outside of 0-1 on input?

@priort

can the Luts be converted rather than modifying DT?

Converting a 1D LUT to a 3D LUT is often not practical. 3D LUTs do not have enough resolution to take high dynamic range linear values on input. This is why most 3D LUTs expect LOG values on input.

Resolve .cube files include the option to include both a 1D LUT (i.e. shaper) and a 3D LUT in the file to work around this problem.

Even if we wanted to create a super-high-rez 3D LUT from our desired 1D LUT, the ‘LUT 3D’ module in DT clamps all values outside of the 0-1 range on input… destroying all pixels with a value above 1.0 (i.e. all values +2.5EVs above .18 mid gray).

@bastibe

1D LUT operates on lightness only, right? How is it different from a tone curve?

Yes a 1D LUT is essentially a tone curve driven by an array of numbers. As long as the Tone Curve module expects to get values well outside the 0-1 range, than it could be possible to convert a 1D LUT to a DT Tone Curve. However, I’m not sure the Tone Curve module would be happy with tens of thousands of points on the curve.

[123sg (Steven)]

@Xavier_Bourque

As long as the Tone Curve module expects to get values well outside the 0-1 range, than it could be possible to convert a 1D LUT to a DT Tone Curve. However, I’m not sure the Tone Curve module would be happy with tens of thousands of points on the curve.

I don’t know about the latter point, but the rgb curve module operates in unbounded conditions I think…

[priort (Todd Prior)]

Is the problem not the lut module itself as it works in DT… So if I understand or at least from the recent comments by @kofa suggested to me … you might for example normally have DT in a working space…that would be say wide gamut rec2020. Now the lut module in DT seems to have an input setting that is going to take that rec2020 or whatever is presented to it and convert that to what you have selected is needed for the lut input to match your lut table of the associated LUT and therefore yield the expected result… since DT doesn’t have a bypass for this function in the module and has about 5 options none of which are any variety of log I don’t think you can ever use it to apply a lut designed for log input… but I could be wrong… to me it would seem that your lut can be fed only rbg rec709 lin or gamma rec2020 or prophoto…because the module will first put the data in one of those colorspaces before passing it to the lut… Is this what you are trying to circumvent with unbreak and then trying to revert that to a linear space acceptable to the lut module… I would think with all that and any clipping as you note might happen would the lut still provide the same utility or would it end up distorted??

[priort (Todd Prior)]

You might find this post of some interest and perhaps the wider thread that it comes from…

Assistance with LUTs designed for another color space such as Log

Resolve apparently still doesn’t support X-Trans, so that can be problematic for Fuji users. One could probably create linear DNGs in camera-native colorspace and use those - e.g. demosaic it but don’t perform colorspace conversion. “Linear DNG” is the terminology used for a demosaiced DNG, although in reality, whether demosaiced or not, DNGs don’t actually have to be linear… In fact in that S1H thread, it appeared like Atomos was shoving V-Log/V-Gamut data into DNGs. The data was most defin…

[agriggio (Alberto)]

Hi,
I agree with @priort. I think the best way to keep using dt is to extend the lut3d module. You could either make it accept some log space, or simply extend it to support aces clf luts. I would go for the latter, as it gives you more flexibility and can be done in a few lines of code by simply offloading the job to ocio.

HTH

[Donatzsky (Nis Donatzsky)]

@123sg

I don’t know about the latter point, but the rgb curve module operates in unbounded conditions I think…

It does. However, the UI doesn’t expose that and is limited to the 0-1 range.

[Xavier_Bourque]

@priort

since DT doesn’t have a bypass for this function in the module and has about 5 options none of which are any variety of log I don’t think you can ever use it to apply a lut designed for log input…

Yes, the LUT3D module does apply a shaper LUT to the input via the ‘application color space’ option. Simply adding a few LOG options (ACEScc/ACESAP1, sLog3/sGamut3.cine, AlexaV3LogC/AlexaWideGamut, RedLog3G10/RedWideGamut… maybe a couple more for Fuji and Canon…) to this list would make LUT3D way more versatile.

@priort

Is this what you are trying to circumvent with unbreak?

Yes. By having an ‘unbreak’ before the LUT3D, I convert linear/rec2020 pixels to log/rec2020.

Since ‘Unbreak’ doesn’t change the color profile metadata, you can leave LUT3D’s ‘application color space’ to ‘linear/rec2020’… the built-in shaper does nothing since it thinks there is no conversion required.

Another thing to keep in mind is that DT applies a lin to sRGB conversion at the end of the pipe since LUT3D doesn’t change the color profile metadata either.

This means our special homebaked 3D LUT converts from weirdo-log/rec2020 to tonemapped linear (i.e. gamma 0.45-ish), then DT converts that to sRGB.

@priort

I would think with all that and any clipping as you note might happen would the lut still provide the same utility or would it end up distorted

Since DT is 32-bits float, despite fighting useless conversions, the end result looks surprisingly good.

@agriggio

You could either make it accept some log space, or simply extend it to support aces clf luts. I would go for the latter, as it gives you more flexibility and can be done in a few lines of code by simply offloading the job to ocio.

That would be fantastic. Does DT use OCIO anywhere right now?

[priort (Todd Prior)]

@Xavier_Bourque

hat would be fantastic. Does DT use OCIO anywhere right now?

No but @agriggio is doing some cool things in ART… https://bitbucket.org/agriggio/art/wiki/Home

Thanks for the reply and breakdown…

It would be nice to see an example of one worked through when you get it all sorted…

[Xavier_Bourque]

For brave souls that want to replicate my workflow, here’s how I have managed to bake LUTs to make them work with ‘unbreak’ log in Darktable using the Fusion page in DaVinci Resolve.

In theory any display/film emulation/tonemapping LUT could be converted to work in Darktable this way.

This is extremely clunky and hopefully a better workflow could be achieved in the future with a few extra features in Darktable.

In Darktable:

• Make sure to work in a linear workflow (i.e. Input Color Profile → Working Profile → linear rec 2020)
• Develop your image as usual
• DO NOT include ‘filmic’ or ‘sigmoid’ modules (our 3D LUT is going to do the tonemapping for us)
• The last 3 steps of your stack should be:
• Unbreak input profile
  Middle gray = 18%
  Black relative exposure: -8 EV
  Dynamic range: 16 EV
• LUT 3D
  Application colorspace: linear rec2020
  Load the .cube file exported from Resolve/Fusion here.
• Output color profile
  Export profile: sRGB

In Davinci Resolve/Fusion, see this screenshot:

The key is to use the LUTCubeCreator/LUTCubeAnalyzer nodes to ‘bake’ our LUT to a cube file that Darktable can use.

I use the Fusion page instead of the Color page because I need the CineonLog node to convert our weirdo ‘unbreak’-log to linear. This is required to have a standard starting point that we can convert to whatever our film emulation LUT expects.

For a 16 EV ‘unbreak’ log (-8/+8EV centered on 18% mid gray) the following values are required in CineonLog:

Black: 0
White: 670
Soft clip: 0
Film Stock Gamma: 0.42
Conversion gamma: 1.0

I’ve been able to use a LUT designed for Sony sLog3/sGamut3.cine in Darktable this way.

I was also able to use the ACES sRGB display LUT in DT too. This opens great workflow options between DT and Blender/Nuke/Krita/Fusion etc… You can preview your image in DT with the ACES display transform and export non-tonemapped EXR linear/rec2020 files for use in other ACES aware software.

(Linear/AP1 (aka ACEScg) export is also possible by using Elle Stone’s ICC profiles)

Again, this is clunky because you have to remember to disable ‘unbreak’ and ‘LUT3D’ before exporting your EXR files.

Both ACES sRGB display and the sLog3 LUT behave as expected in Darktable once you’ve jumped through all the hoops in Resolve.

I understand that full OCIO/ACES integration might be overkill for casual photographers, but hopefully the benefit of using ‘video’ LUTs in DT might be able to convince developers that adding better ‘shaper’ LUT support in DT is worth their time.

If anyone has figured a more streamlined way to achieve similar results, please share! I think custom ICC profiles might achieve similar results, but I don’t know how to create ICC profiles that describe a ‘known’ log profile.

Thanks to @priort for pointing me to the ‘unbreak profile’ module… this is the key that unlocks this workflow.