This post comes from the one previously published in Reddit (darktable master gets a JPEG default pipeline order) since I was suggested to publish here. There are already a few responses there if you want to read them.
After doing a lot of research without success, I’ve decided to ask in this forum.
Firstly, I want to clarify that I’m not a professional and, maybe, I will write some nonsense when explaining my problem. For this and my bad English, I want to anticipate my excuses.
I usually process my RAW files with Darktable (DT) under the scene-referred paradigm. No problem here.
Now I’m trying to process a set of shots in JPEG format since they were taken with a friend’s camera without RAW functionality. And at this moment some doubts assault me. Let me explain.
According to my understanding, a JPG file from a camera fits with display-referred due to the data has been already compressed into a range that represents pure black as 0 and pure white as 1, fixing mid-gray at 0.5. Based on this assumption I believe that I must use a display-referred paradigm to process it in DT.
Yet in DT, I change the order of modules to “v3.0 JPEG” and choose display-referred workshop. Then “input color profile” is moved much more before, leaving modules as for instance “Exposure” after. With display-referred workshop activated I can see modules like “Exposure” (in base category) or “Color Balance RGB” (in color category), however, DT informs that their input must be “Linear, RGB, Scene-referred”.
Here are some of my doubts:
Can I use modules like Exposure and Color Balance RGB with JPG files when the input I believe is a non-linear?
Is it OK to change the modules’ order to “v3.0 JPEG” and process the image the same as in scene-referred method, as it seems DT invites to do?
I have many more doubts regarding the context I explained, but I think that solving the previous ones will be enough to solve the others.
If you look at the “input profile” module, it describes its output as “linear, RGB, scene-referred”. So that fits with what the following modules expect…
So, the answers to your questions are “yes” and “yes”. With respect to the module order, I’d say it is even highly advisable to set it to “v3.0 JPEG” when editing jpegs, precisely because that puts the “input profile” before the modules that expect linear data.
The “input profile” module knows in what colour space the jpeg data is (sRGB), so it can reverse the non-linearity. And one big advantage of image editing in programs like dt is that you see what the different operations do, including when something breaks. So as long as nothing breaks, and the results are what you may expect, you should be ok for your editing…
I think there’s quite a bit of time spent in making sure the different module orders work as expected, even if they aren’t perfect for all situations (they can’t be).
It does not reverse the nonlinear mapping from scene to jpg (or display) gamut as it cannot know which operations have been done, and even if, some of them are not reversible. What it does is to remove any nonlinear curves used e.g. for efficient coding of the picture data in the jpg format, and it makes sure that colour spaces are handled correctly. But this is anyway the best you can do, in terms of the image data you are technically linear after input profile, but it will not bring back the original raw data of the jpg image.
Edit: Just want to mention, this is not an issue, you can work with the jpg easily in darktable.
Scene-referred is an approach to processing. So, while it isn’t ideal, one can still edit JPGs with such an approach.
Camera raw files contain image data that don’t have the JPG baking[1] and much more metadata (data about data) that post-processors can use.
[1] The baking not only includes sRGB but looks or styles and edits the camera manufacturer decides their JPGs should have. By the way, a universal sRGB profile doesn’t exist, but they mostly look the same. At capture, take care to ensure the camera settings are as neutral as possible. Neutral is boring, but it makes the image easier to edit in post-processing because it decreases the issues I describe below and the details I left out.
One challenge of editing JPGs is that information is missing due to nonlinearity. White and black points clip, and points close in value or to clipping points round and merge into gelatinous blobs.
In addition, JPGs may exhibit banding due to the rounding described above, compression artifacts due to keeping the file size low and softening due to excessive denoising. Colour shifts and other issues can arise for these and other reasons.
I note these issues not because they are unsurmountable. However, they are much harder to fix because of the limited information we have from the JPG.
Strict scene-referred photography requires photographic memory, tools and skills to correct scene inaccuracies and discrepancies. dt has some tools to help with that, but photo and scene restoration is a craft that takes time and mastery beyond general post-processing.
Edit: sorry about the multiple edits. Hopefully, after the dozen edits, this post makes more sense.
The problem with this statement is that the input is NOT sRGB - with every single camera on the market, it has had an additional tone curve applied.
To properly linearize the input, one needs to construct an input profile that actually accurately describes the transfer function.
This can potentially be done with a bit of effort, although so far my experience has been that you can only get an approximation. I may be spending some time over the holidays working on tweaks to Robertson’s algorithm that may handle some of the corner cases I’ve observed failure on a bit better. But for a general input color profile, maybe what I’ve come up with so far is sufficient.
It is possible to reverse engineer a transfer function. See OpenCV: High Dynamic Range (HDR) - ignore the stuff about HDR recombining, pay attention to calibrateDebevec and calibrateRobertson - these allow you to estimate the actual transfer function from a set of JPEGs taken with an identical scene but varying exposure.
As others have mentioned, some cameras may have JPEGs where the processing went beyond even just a simple per-channel tone curve in which case reversing it will be even more difficult if not impossible, but most cameras on the market are still doing per-channel tone curves.
Sorry, I have to disagree here. A jpeg file is in sRGB, which describes the relation between the (r,g,b) pixel values and observable (CIE) colours. Part of that relation is a well-defined function to go from scene-referred to display-referred data, and that part can be reversed to get linear data. Whatever has been done to the data from the sensor prior to the storage in a jpeg file has no bearing on how the data is encoded.
We are not trying to get the raw sensor data back, we’re only concerned with getting the data in a form suitable for darktable’s scene-referred workflow to allow further editing (with the caveats described earlier by @afre).
Not for almost any real camera on the market. Take a look at | Help Guide for Creators | What is Picture Profile? for example, to see an example of all of the different ways in which a Sony JPEG can be not sRGB
Ideally a camera would tag its images with an appropriate ICC profile that described what actually is in the image, but in reality, that is never the case.
In fact the only gamma settings that DO correspond directly to an actual transfer function that can be linearized without reverse engineering (e.g. compliance to a published formula) are ITU709, the various S-Log modes, and the HLG options. Assuming that ANY of them are sRGB = failure to linearize properly. It is literally impossible to make a Sony camera output actual sRGB.
Which is only possible if you accurately characterize the transfer function of the input - which is almost never actually sRGB. sRGB is an extremely poor approximation for what actually comes out of almost any modern camera’s JPEG processing line. If you don’t linearize it, it’s not suitable for the scene-referred workflow.
Indeed. If an ICC profile is embedded in the JPEG, its TRC can be inversed. That’s exactly what’s done when an ICC profile transform is done - the image is converted to linear XYZ and then converted to the destination profile’s gamut/tone.
Even with just a ‘sRGB’ tag, one can just assume the sRGB TRC per IEC 61966-2-1:1999. This may not be a correct assumption, but probably close enough for gov’t work…
All of the above assumes the JPEG was encoded to the full specification of sRGB gamut and tone; if not, well have fun!
To the OP’s original question, IMHO no, it does not make sense…
Much fun… But if someone claims to follow a standard, and doesn’t, not much you can do… And indeed, other color spaces are possible within a jpeg, and the color space used can be described in the EXIF metadata.
@priort: The encoding used for the jpeg isn’t really relevant here: on loading, the image is decoded into an uncompressed rgb image before we deal with colour spaces.
The encoding is relevant for the image degradation caused by the jpeg compression, and perhaps for the remaining leeway in editing before you get (more) artifacts. But that’s a different issue.
As an example, for Sony cameras, this is the effective transform applied after sRGB encoding (The S-curves look a little nicer in gamma-compressed space, which is likely why RT’s tone curve tool still operates there):
RT’s AMTC was quite off here, not sure if it’s an error in my math or lens distortion threw it off.
From a darktable perspective, this means that the only valid workflow is one based on the “legacy” model where basecurve is near the beginning of the processing pipeline - because a tone curve has already been applied. (To be clear, the legacy pipeline where basecurve is at the beginning, but with basecurve turned off, again - because it has already been applied.)
If a scene-linear workflow is desired, the tone curve needs to be reverse engineered and an appropriate ICC profile generated that embeds it (hopefully the gamut isn’t mangled too badly and THAT can at least be assumed). I’m planning on cleaning up and enhancing the tools used to generate the data from in the plots above over the holidays and putting them up on github.
Edit: This gets into past discussions about the nature of the data at the output of basecurve/filmic/sigmoid - it’s linear encoded but NOT scene-linear. I’ve used “display-linear” to describe this before. Or another way of saying it - if you linearize with a naive sRGB curve, you get display-linear data and must assume in your pipeline that you’re working with display-linear data. If you linearize with the camera’s actual response curve (calibrated using Robertson or Debevec), you get scene-linear data that is appropriate for a scene-linear pipeline. LuminanceHDR is probably easier to use for determining the response curve, but converting that response curve to an ICC profile is a little harder.
This discussion is one of a few things that has caused me to dust off some old efforts from last year with respect to reverse engineering camera response behaviors.
I pushed them to GitHub - Entropy512/camResponseTools: Miscellaneous tools for reverse engineering camera response curves - sometime over the holidays I’m going to whip something up that generates an ICC profile from the derived transfer function, whether via these scripts or via LuminanceHDR’s calibrators - that will allow doing a scene-referred workflow without breaking the core assumptions of most of the pipeline. It won’t be a perfect reverse engineering, but it’ll be much better than simply ignoring the fact that the camera did a significant scene->display transform internally
(Does require access to the camera to get the calibration data though…)
The image has effectively had a tone curve applied twice, excessively boosting midtone contrast and also leading to some weird artifacts in the clipped areas.
Here’s the same image, using an ICC profile that properly describes the transfer function of the input image and no other changes to settings:
After installing imagecodecs-master I tried to exec your script by this command “python3 response_to_icc.py --input DSC03425.JPG”, but it fails writing:
"$ python3 response_to_icc.py --input DSC03425.JPG
Traceback (most recent call last):
File “/home/ferran0sabate/Desktop/Foto/response_to_icc.py”, line 19, in
response = np.load(args[‘input’])
File “/usr/lib/python3/dist-packages/numpy/lib/npyio.py”, line 445, in load
raise ValueError(“Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False”
That’s because the script is meant to work with the other scripts in that repo:
fs_gradient_capture displays a fullscreen gradient and captures bracketed images every 1/3EV over a fairly wide range - the idea is to point the camera you are calibrating at the screen to get calibration data. Manual bracketing in-camera may be sufficient, I’m not sure how good the resulting response curve will be. It definitely helps to have a smooth gradient that you are capturing. I may change this script to use just a white gradient instead of multiple colors, as I’ve seen reports that out-of-gamut inputs to the camera can throw off response recovery.
robertson_process takes the data captured above and processes it using OpenCV’s calibrateRobertson to determine the response curve. This saves out an .npy file with the response curve. Alternatively you can use LuminanceHDR in response recovery mode (make sure to actually choose an output file):
I just pushed support for the response curve output file that this saves to my repo. For the time being. LHDR seems to do a better job than my OpenCV script in terms of quality of the response curve. At some point over the next few weeks I’m going to reimplement Robertson’s algorithm in Python so I can tweak it to handle the corner cases better.
Once you have a response curve file (.npy from my script, or .m from LuminanceHDR), the latest script will turn that into an ICC profile. Note that for now, the script assumes a camera’s response is common to all channels and that the G channel has the best data. At some point I may change this, one of my TODOs is to try and force the calibration to use a common response curve across all channels (since most cameras do this)
Also, the script requires a version of imagecodecs released two days ago at the earliest, and may require today’s release. Not sure if those have propagated to PyPi yet
