Processing RAWs for HDR displays in Darktable

[JLTastet]

@kmilos

I have a few gripes, at least when it comes to JXL support in darktable:

I am not very familiar with the technical details, so I will trust you on that. Hopefully the API issues will be resolved once the reference implementation is officially published.

As far as uptake is concerned, it seemed to me that there was strong interest from some big players like Adobe and Facebook. In particular, the lossless JPEG transcoding (and the associated reduction in bandwidth) could be a big driver for adoption by social media platforms, especially since there is no need to wait until all browsers support JXL for the benefits to start to appear (e.g. serve the JXL file if the browser supports it, otherwise fall back to JPEG). Of course this does not guarantee that the HDR functionality will ever be exposed…

I’d be happy with AVIF, too, if it sees broader adoption

Regarding HEIF, my understanding is that uptake was hindered by potential patent issues. Please correct me if I am wrong.

I’ll have a look at the issue you linked to. Not sure I can help much on the Darktable side. However, if I manage to find or produce a “reference” JXL file of an HDR image, I might play a bit with the JXLook viewer to see if I can get it to render correctly (the code did not seem to be overly complicated).

@Entropy512

Anything that is not a dedicated video playback device (Smart TV, Chromecast, Android TV, etc) is likely to be problematic.

Oh, I see. I have to admit that I don’t own a TV, and that almost all of the non-still pictures I watch come from YouTube So I did not realize the support was that bad on non-dedicated devices.

[Tamas_Papp (Tamas K. Papp)]

@kmilos

we’re not there yet with the HEIF/AVIF ecosystem after several years and investment by big players, what makes you think JXL uptake and (good) SW implementation of both content creation apps and viewers is going to be quicker/better?

I don’t know what you mean here — Windows 10 and OS X have supported HEIF for years now, also Ubuntu 20.04 (LTS), and Android since 8. This means that if you updated your device within the last 2 years (and you should do that more often, for security), it supports HEIF.

I expect that something similar will happen to JXL once it is matures a bit. Early adoption will mean that you will face a few hurdles, but that’s all, sooner or later everyone will be able to read it.

Will camera manufacturers get their act together and adopt one of these formats to replace JPEG? That’s hard to say, and may not be relevant: people who edit their images in any nontrivial way (beyond cropping and “artistic” filters for Instagram) may already be using RAW, and JPEG is certainly good enough for social media etc.

[kmilos]

@Tamas_Papp

I don’t know what you mean here

See the links to other forum posts - there are still plenty of apps that render HEIF/AVIF content in the wrong way. We wouldn’t be talking about this if it was as well implemented/hashed out as JPEG (which to be fair, is less complex in terms of color management features)…

[kmilos]

@JLTastet

DWA compression does not seem to be exposed in DT

Will add this w/ 16-bit float support as well.

[Tamas_Papp (Tamas K. Papp)]

@kmilos

there are still plenty of apps that render HEIF/AVIF content in the wrong way

Please be more specific. There are always bugs, but without details it is impossible to know if they are still relevant or have been fixed.

[kmilos]

Experimenting with HLG and HDR10 for still images

Yep, the same sequence (JPEG sRGB / AVIF HLG + CICP / AVIF HLG + ICC) in Windows Chrome: [image] Definitely some way to go for the AVIF ecosystem

To be fair, I haven’t checked if this example has indeed been fixed.

[Entropy512 (Andy Dodd)]

@Tamas_Papp

Please be more specific. There are always bugs, but without details it is impossible to know if they are still relevant or have been fixed.

Here’s a specific one:
The subject line of this thread is “for HDR displays”

As of now: No Linux distribution I know of supports 10-bit HDMI output, nor any of the HDR transfer functions (PQ, HLG, etc) in their display output pipeline, nor setting any of the metadata flags in HDMI to tell a display what the transfer function is (PQ or HLG). There’s no compositor support for an HDR pipeline. (There’s been talk of ways to do it, but no implementation I’m aware of.)

Intel was starting to put in some of the infrastructure in a few years ago, but to this day, 75%+ of the infrastructure is completely missing in Linux.

Ubuntu 20.04 having HEIF support is irrelevant, because it effectively only supports the SDR subset. Anything higher bitdepth than 8 will get tonemapped down for display. Anything HDR will either get tonemapped down to SDR using some method of compressing the peak luminance without clipping, or appear severely underexposed. Just like playing HLG or PQ video content in VLC will tonemap it down to SDR.

(Which leads me to a small annoyance - while the FOSS still image processing world moved everything to float32 or better pipelines years ago, nearly all video editing tools are based on MLT, and MLT is still stuck with an end to end int8 pipeline.)

@Tamas_Papp

Will camera manufacturers get their act together and adopt one of these formats to replace JPEG?

Quite a few have, and that itself has exposed many of the flaws in HEIF’s ecosystem. Sony’s HEIF implementation makes a lot of viewers choke for example, but no one can point to a noncompliance with any aspect of the standard that allows Sony to be directly blamed other than “hurr durr it doesn’t work with viewer X, bad Sony!”

[Tamas_Papp (Tamas K. Papp)]

@Entropy512

No Linux distribution I know of supports 10-bit HDMI output

My understanding is that all recent versions of Xorg support it if the graphics card does, you just change one config line.

[Entropy512 (Andy Dodd)]

So, maybe 5% of the progress to the goal. Still no HDR compositor, no transfer functions other than sRGB, no HDR metadata support to tell a display that PQ or HLG is in use.

The only actual reliable and consistent cross-platform HDR output system I know of is Blackmagic’s Decklink/Ultrastudio product lines, which sacrifice hardware acceleration in exchange for giving a developer complete and total control of the output signal, bypassing all color management.

[kmilos]

@JLTastet Feel free to give this a spin for more OpenEXR export options: https://github.com/darktable-org/darktable/pull/11001

[JLTastet]

Thanks a lot @kmilos!

I have never built Darktable from source yet, but I have a long weekend ahead of me and it’s peak omicron here… So I’ll try to do it and give your PR a try. (If I succeed, I’ll also try to test the JXL PR.)

Regarding your previous comment about JXL:

@kmilos

The encoding options seem overly complex and incomprehensible to me , so there is a real risk of not getting it right (any help deciphering and testing those is welcome; the overall complexity of the codec makes me think it’ll take longer for its uptake, not quicker than, say, AVIF)

I had a look at the issue, and indeed, the fields intensity_target, min_nits and relative_to_max_display are a bit confusing. On the one hand, some metadata is probably needed if we want to render HDR content as consistently as possible across devices. On the other, these properties seem a bit redundant (e.g. isn’t intensity_target defined by the transfer curve itself?) and at the same time insufficient.

It seems to me that what we ideally want is some “HDR rendering intent” that tells the viewing software how the image should preferably be rendered, and how it should be tone-mapped if it cannot fit in the display’s color volume (with possible options such as “absolute luminance” vs “relative luminance” and “clip highlights” vs “flatten highlights” vs “scale luminance down/up” vs “reduce contrast” vs “filmic(white_exp,black_exp,contrast,lat,…)”)

But this would probably deserve its own standard, and we are still very far from there In the meantime, setting some sane (if arbitrary) defaults as suggested by Jon might be the way to go.

[age]

@JLTastet

(e.g. isn’t intensity_target defined by the transfer curve itself?)

No, for example pq curve allows 10000nits but most hdr10 blurays are just 1000nits.

IMHO SDR intensity_target should be set to 100 (nits), the problem is that we have pratically 2 sdr standards

1)middle-gray at 18 nits and white at 100 nits

2)middle-gray at 26 nits and white at 200-300 nits

The latter is used in commercial non calibrated tvs, however sdr blurays are graded using the real standard gray at 0.18 and white at 1.0 so this should be the default.
It could be usefull to allow the conversion between this two standard using a power function curve.

Min_nits could be paratically set always to 0.

relative_to_max_display: here it’s important the linear_below option, the tone mapping operator will only roll-off the highlights, values under the linear_below are left unchanged (similat ro the bt.2390 tone mapping)

[JLTastet]

@age

No, for example pq curve allows 10000nits but most hdr10 blurays are just 1000nits

Thanks for clarifying! Does that mean that, in this case, Y = 1 represents a displayed luminance of only 1000 nits? Or is Y restricted to a value < 1 which corresponds to 1000 nits (and Y values corresponding to 1000-10000 nits are unused)?

IMHO SDR intensity_target should be set to 100 (nits) […]

Yes, 100 nits seems quite common, especially on monitors/laptops (maybe because it is close to the 80 nits of sRGB?). On macOS, all the “video” presets have an SDR luminance of 100 nits.

[age]

@JLTastet

Or is Y restricted to a value < 1 which corresponds to 1000 nits (and Y values corresponding to 1000-10000 nits are unused)?

This one, with the pq curve 100 nits are mapped to roughly y=0.5 and 1000nits to roughly y=0.75. If 1000 nits is the peak hdr brightness values above y=0.75 are unused.

With HLG in theory is possible to encode up to 5000nits, but for absurdly reasons this standard doesn’t has metadata and it’s supported in the bt.2100 standard with only 1000nits.
This means that hlg could be displayed correctly only if the peak brightness is 1000nits because there aren’t metadata.
With hlg 1000 nits should be represented with y=1

[JLTastet]

@age

This one, with the pq curve 100 nits are mapped to roughly y=0.5 and 1000nits to roughly y=0.75. If 1000 nits is the peak hdr brightness values above y=0.75 are unused.

Looking at this comment again, it seems that intensity_target denotes the maximum theoretical luminance of the curve (i.e. 10000 nits for PQ, even if the image is limited to 1000 nits), so that multiplying it by Y gives the displayed luminance.

If the output color profile specifies the screen luminance (e.g. 80 nits for sRGB), it would make sense to use this value. But better double check that with Jon or someone who knows the JXL standard well, to make sure that decoding SDR images does not rely on the special value of 255.

[JLTastet]

I just noticed that Google Chrome is already capable of displaying HDR images in the JPEG XL format.

See my comment in the JXL PR discussion for more details.

The trick is to enable the flag chrome://flags/#enable-jxl and, using the above-mentioned DT patch, to export the images in JPEG XL format with the PQ Rec2020 RGB profile. The images are a bit slow to load in Chrome, and lossless JXL files never load, but apart from that it works, with accurate colors and contrast. HLG looks okay too, but the shadows are a bit too contrasty, while the linear transfer curve leads to posterization and JPEG artifacts in the shadows.

A 24 MP picture now weights only 4.5 MB in JXL format (for quality=12) vs. nearly 300 MB in EXR format. And it looks even better since the EXR was not color managed properly.

When I find time, I will try to tinker with JXLook and see if I can get it to display the HDR luminances instead of clipping them.

[age]

Simple steps to PQ encoding with darktable and ffmpeg (x265 output)

P1000657.RW2 (18.8 MB)
P1000657.RW2.xmp (20.7 KB)

1. load the image
   

   

2. add exposure compensation to taste
   

   

3. map 10000nits to 0-1 range (exposure setting -6.42)
   

   

4. export as rgb pq rec2020 tiff

5)encode with ffmpeg, in this step we only convert to ycbcr and attach the metadata

ffmpeg -loop 1 -t 10 -i P1000657.tif -vf scale=out_color_matrix=bt2020nc,format=yuv420p10 -c:v libx265 -preset medium -x265-params “colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:range=limited:master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1):max-cll=1000,0.0001” out1.mp4

Simple steps to HLG encoding with darktable and ffmpeg
It can’t be done right now but the steps for hlg encoding are the follows
https://www.bbc.co.uk/rd/sites/50335ff370b5c262af000004/assets/592eea8006d63e5e5200f90d/BBC_HDRTV_PQ_HLG_Transcode_v2.pdf

1. load the raw and add exposure correction

2. map 1000 nits to range 0-1 ( pixels divided by 10 or exposure corretion tool -3.321 ev)

3. apply system gamma (γ=1.2+0.42Log10 (lw/1000)=1.2+0.42Log10 (1000/1000)=1.2)

4. eventually expand the 0-1 range to 0-12 range (based on what formula darktable use)

5. convert from “linear” to hlg

[JLTastet]

Amazing!!!

I followed your instructions, and this time it worked perfectly! I managed to obtain a MP4 file which correctly renders as HDR in both QuickTime and Infuse, and for the latter it seems to match the JXL file quite well (when assuming that the SDR brightness is about 64 nits, not sure why…). See the comparison below:

(Unfortunately, not many open-source viewers seem to support HDR )

It also renders consistently on my iPhone XS (after adjusting the brightness), but obviously this one isn’t in the picture.

This time I have used the base curve (since Aurélien said it is not bad any more ) to flatten the highlights a bit, and I have set the max luminance to about 1000 nits (by adjusting the histogram after base curve to the right, then adding an exposure correction of log2(1000/64)=+3.97 for JXL, log2(1000/10000)=-3.32 for H.265).

I have also tried to use filmic and then correct the exposure, but I could not get a good looking result.

Btw, it is interesting to see that depending on which method we use, “luminance=1.0” (i.e. histogram to the right) is mapped to either the SDR luminance (when exporting to JXL) or to the HDR one (when exporting to TIFF and using ffmpeg to encode it as H.265).

[JLTastet]

Below are all the sample images from my original post, in JPEG XL format (quality=14, i.e. visually lossless) and full resolution. I have used the PQ curve since it gives perfectly accurate results in Chrome.

samples_jxl.zip (33.9 MB)

I had to zip them because the forum software does not recognize the .jxl extension yet and won’t allow me to upload them one by one.

Sorry, the file you are trying to upload is not authorized (authorized extensions: jpg, jpeg, png, gif, ico, dtstyle, txt, scm, pp3, svg, xmp, bz2, xcfbz2, py, arw, apng, orf, cr2, nef, dng, tif, patch, zip, 7z, rtc, raw, exr, hdr, raf, rw2, pfi, xcf, xcf, pef, icc, icm, pto, blend, dcp, xcfxz, xcfxz, ods, mp4, mkv, ogv, webm, cr2, nef, dng, cr2, pdf, kra, ntp, arp, fits, seq, gz, ssf, cr3, heif, heic, avif, crw, dtpreset, nrw).

[gregbenz]

The future of HDR images on the web will be “gain maps”. This allows an image to look optimal both on SDR and HDR displays (as tone mapped HDR looks pretty mediocre on an SDR display, and sharing 2 versions of an image online is cumbersome, especially given mixed support for media queries and the tag).

Adobe has released a spec and sample images in JPG, AVIF, and JXL formats: Gain Map in Adobe Camera Raw

Chrome Canary already supports JPG gain maps under a dev flag (they refer to the format as Ultra HDR JPG).