Why aren't we utilizing embedded previews more in Darktable?

Software darktable
1

I had this idea a while ago but today I was kinda fighting the windmills with the Mavic raw files and it reminded me of this idea I had.

So it turns out that embedded previews in raw files get the same processing as the in camera jpegs just at the reduced file size. Actually as it turns out, a perfect file size. For Canon 5DmkIV that ends up being a 6729px x 4480px 2.2MB large embedded preview.

It’s a perfectly usable jpeg file, and quality wise, for 2.2MB at that resolution there’s nothing wrong with it. It’s actually perfect.

So why aren’t we using it?

I can think of a number of examples when I would have loved to have an easy access to those embedded jpegs. And I’ll list the main two here:

1.] Delivering previews to a client.
If I shoot raw and let’s say I shoot 1000 images. I have to spend the entire day doing basic adjustments to those images to make them a bit more presentable than a pure jpeg converted raw so that I can send them to a client who will then pick a few (or maybe a hundred) that he likes and those images go for further processing.

Now, with Lightroom you don’t have that problem. You imort your raw files and they look a lot like a camera JPEG but with Darktable I’m sorry to say it looks nothing like it (sometimes it does, sometimes it doesn’t. It’s pretty good actually but still). Now that’s not a bad thing it means I’m making the look, not the software. But it also means I have to spend at least a minute on every image before I can export it as previews for the client.

Now one might say. Just send the gray dull ones to the client to pick. And I’ve tried that, then they pick the ones where the raw accidentally has more color. They are not photographers.

Currently there are 2 options. Shoot raw + JPEG and send those JPEGs. Or shoot raw and spend time processing previews. The first one takes up storage and heats up your camera, the second one wastes your time.

It would be a great thing if we had a third option! If Darktable had an option to export the embedded previews possibly at even lower size and possibly with watermarks.

That would be epic. No going trough the raw editing pipeline. Blazing fast, already “edited” previews at low file size to sent to the client.

2.] I think since Darktable doesn’t try to replicate in Camera jpeg, at least with the scene linear workflow, it would be a good idea to have “a button” to show the embedded preview so we can compare it to the edit we are doing.

I’m sure others will have more ideas but it’s astounding to me that we have excellent 2.2MB in camera processed JPEGs taking up storage space on our machines and we do nothing with it. Just to put it in a perspective for you; I store more than 3TB of embedded previews inside my raw files. It’s crazy that we don’t do anything with that and we gotta change that.

edit: before anyone says use a script, that’s not an integrated solution.

7 Likes
2

Because I want the raw data. If I wanted the jpeg, I’d shoot in jpeg or raw+jpeg.

But I have no intention of matching the jpeg processing.

You can extract the jpeg preview with exiftool:

exiftool -b -JpgFromRaw anyfile.dng>full_preview.jpg

You can use some lua to get that functionality into darktable itself.

3 Likes
3

You didn’t understand me.

When you shoot a lot of images (for a client), even after your personal selection you might end up with a few hundred images. A client also wants you to send all previews to him to select before you do the editing. I’ve had situations where I sold an image that I didn’t even wanna include in the previews. So this means that I’m sending it all today. But you won’t edit 1000 images just so that a client can pick 80.

You do some light processing, contrast noise etc. And send that for selection to the client. Then he picks a few or more and you do real editing on those.

The process is this:
Shoot > Send previews > Get final selection from the client > Edit > Get feedback from the client > Final edit

This means that it would be a great idea to have those jpegs to send as previews for the client to pick before final editing. It also wouldn’t require a photographer to shoot Rar+JPEG. Otherwise you need to do some light editing on every image before you can send it and that just wastes time if most of those images will go to trash.

Most likely you’ll need to remove color noise on all images that you are sending as previews. Even if that can be as simple as pasting the history to all the images. Exporting that can take a few hours. If you are on a low end laptop it will overheat and the app will crash. So why don’t we use the already decent jpegs?

A script is great and that’s how I do it but it would be great if DT made that as a feature. I think other people might find that very useful.

2 Likes
4

I did, that’s why I said:

2 Likes
5

Oh, then you might find this command of better use:

exiftool -b -PreviewImage [anyraw]>preview.jpg

6

I guess the post wasn’t clear enough. I should have mentioned that the ordinary users don’t know how to extract metadata. And imho we shouldn’t require them to.

It would be a nice thing that photographers could make use of already existing jpegs in the embedded previews if they needed to without becoming power users.

That was the entire point of the post. Not how do I extract it.

1 Like
7

I’m on windows. I use the free Fastone Image viewer to copy over my raw files from sd card to my PC. It creates the date wise folders and renames the files as well. It can read the jpg preview from the raw just fine, and I could save out all these jpgs to a separate folder with a couple of clicks.

I’m sure Linux will have something similar, if not better.

8

I think the issue you’ll have is that most of the developers (myself included) would much prefer to just use the command-line option and therefore wouldn’t see the need to work on a (probably) more complicated solution to integrate it into darktable.

If you want to do a comparison with the embedded preview you can do so in the lighttable though. As long as you don’t have the “don’t use embedded preview JPEG but half-size raw” option selected, any raw image that has not yet been edited in the darkroom displays in lighttable using the embedded preview. Just create a duplicate of your current edit, discard history and you can compare the two images.

5 Likes
9

There is a lua script, examples/multi_os, that extracts the embedded jpg, then imports it and groups it with the raw

5 Likes
10

That’s exactly why most cameras are equipped with a raw + jpg option. Darktable is a raw processor for those wanting to get more than the camera processing is capable to deliver.
If you need a jpeg editing tool then expecting this in darktable is like expecting a steak in a vegetarian restaurant …

2 Likes
12

Read the entire post.

I wasn’t talking about jpeg editing. I was talking about embedded preview extraction with >>>possibly<<< adding a watermark if you can call that editing. Just so since we have a raw file, let’s make use of the entire raw file.

I covered the issue with raw+jpeg too. But basically, why would one need to shoot raw+jpeg and waste more storage if there are already perfectly usable jpegs in the metadata. One just needs an easy way to access it.

3 Likes
13

Perhaps this answers it:
embedded jpeg: 1616 × 1080 pixels,
full raw: 4000 × 6000 pixels.

This is for a Sony camera, so it might be different for other brands/models

Such a jpeg size might be enough for some, but no more full screen after even the slighest crop. Also, while the processing at low ISO could be acceptable, I find high-ISO treatments often a bit too aggressive, leading to loss of detail and halo formation

P.S. I noticed you cited as an example use for this functionality use under professional conditions. Please keep in mind that that is a use aimed at generating revenu. Of course it’s nice to use a free tool (and I have no problem with that), but don’t you think it’s a bit strong to then demand that the tool is adapted to your particular use case (for free…)? Like @MStraeten said, darktable is a raw processor, not a jpeg editor/metadata extractor.
And you got two recipies on how to extract those previews, one for windows (Fastone image viewer),; one for Linux (exiftool), both free tools.

14

Where am I failing to communicate my thoughts?
I didn’t demand anything. I was amazed at how much space those embedded previews take but aren’t being used at all. I mean, aren’t you at least a bit surprised? I wanted to explore this idea with this forum. And I got my answer for that particular idea. But I feel like nobody is quite getting what I’m trying to say (and I’d say most of it is my fault for not communicating it well enough and writing needlessly long posts).

I also wanted to find out how others are making use of those previews. My thinking was that the discussion is going to lead in that direction. A constructive direction. So I can maybe find out some cool uses for those. Instead it’s headed towards my defending the idea for no reason.
Fine, nobody likes the idea, that’s what I wanted to find out. It was never my intention to convince anyone that we need that particular feature, just that we might try to think of cool ways how to use those embedded jpegs.

The question still remains. Is there anything that those previews can be used for more practically? B/c at the moment they seem to just take up storage.

I mean, I fully intend to use them in a way that I mentioned above. I’m never shooting raw+jpeg ever again xD

3 Likes
15

And that settles my use case :smiley: Thanks!

16

I think about using the embedded JPEGs for my ‘proof’ images once in a while. Right now, what I do is to take my ‘dailies’, the collection of raws I capture in a session, put them into my picture directory structure and batch-process them to a set of 800x600 ‘proof’ images. Two things vex me from just extracting the embedded JPEGs:

  1. No control over the final size. I’ve come to rather like my 800x600s; they’re easy to schlep over networks both near and far. In my web viewer that my wife uses, that size keeps me from having to generate separate thumbnails for the contact sheet view.
  2. With rawproc/img, baseline processing is embedded in the proof JPEG. I use img for that proof batch script, and it collects the processing as it’s applied and sticks it in the ImageDescription EXIF tag when the JPEG is saved. Later, if I drag such an image to the rawproc icon on my desktop, rawproc will first look in that tag for a toolchain; if found, it asks me if I want to either open JPEG itself, or open the source image instead and re-apply the ImageDescription toolchain. Sadly, Nikon doesn’t embed a rawproc toolchain in their embedded JPEGs… :laughing:

FWIW…

1 Like
17

Thanks. Also I have been searching for an easier solution than just using the Terminal and Exiftool/Exiv2.

18

RawTherapee can use the embedded JPG to apply the same distortion corrections as implemented by the camera. Lens/Geometry - RawPedia

darktable can also use the embedded JPG ‘thumbnail’, e.g. for Ctlr+W focus peaking. https://elstoc.github.io/dtdocs/lighttable/lighttable-modes/full-preview/

For your original use case, though (providing quick reviews), I’d use a script to quickly extract the embedded JPG, maybe resize it for the screen, and add a watermark.

For such previews, even if you start with the raw file, you can probably skip noise reduction or apply some profiled noise reduction with default settings in wavelet mode, which is pretty quick on a GPU; exporting at full HD (without the ‘high quality resampling’ option) is rather quick. You could rely on the automatic exposure correction (as you said the embedded JPGs would be useful to you, I suppose they are well exposed) and default filmic settings, or use a straight (in fact, logarithmic) filmic setting and apply a 3D LUT of your choice after filmic has done the tone mapping.

2 Likes
19

It’s only a matter of adding a GUI button that starts a special export where the embedded thumbnail is copied to the output.

Not super difficult, as usual it will be more Gtk coding bullshit that actual pixel code, and a valid use-case is my opinion.

5 Likes
20

in fact i was always unhappy about our use of embedded thumbnails in darktable:

  1. okay they are fast, but they are inconsistent to the raw rendition. this lead to a lot of confusion among users, now you have to explain which one is the thumbnail and what’s the really developed version. this is so hard to explain that i completely disabled the use of those thumbnails in my config because i would forget myself which is which.

  2. the thumbnail quality is inconsistent between cameras/raw formats. some store full res jpg, some ridiculously small (like VGA res or something). some even have multiple versions. so you can’t guarantee a solid workflow for all, or at least you’re facing a lot of special casing work to implement.

also, if i understand correctly you’re not so much worried about the look of the jpg vs raw but about the processing speed? i think most of the discussion goes away if thumbnails can be computed at higher speed.

6 Likes
21

I find it useful having a camera jpeg to compare when I’m editing in Darktable. This is quite important imho as often times an editor can get carried away with the colors and tones so it helps to have a look at what camera manufacturer thought would be “the best” look. That said, I hate shooting RAW+JPEG and storing those JPEGS.

Not quite, I don’t want to process the embedded previews in terms of color and tonality, they are already processed. I’d just like a fast way to export them possibly with a watermark to send to the client as previews before I do any editing on the raw files.

Now if I’m trying to imagine how a full feature would look like in terms of UX with anything that one might possibly need. This assumes that we are trying to take full advantage of the embedded previews and cover everything. It would look something like this:

  • It shouldn’t create a separate file to view the embedded jpeg. Is it possible to show the embedded preview without first extracting in into separate jpeg file? Reason being, we aren’t saving any space at all if we need to create the new jpeg file to view it.

  • There should be a button, toggle or a keyboard shortcut to quickly switch between the raw file and the embedded preview or a compare view of some sorts in Lighttable and Darkroom.

  • (OPTIONAL) Since it’s meant to compare the two files, some processing should be applied for the embedded preview like crop, rotate and perspective correction otherwise it’s a bit harder to compare.

  • There should be an option to export the embedded previews instead of raws (like a checkbox in the export module).
    (OPTIONAL) That export should not touch colors only crop, rotate and perspective if applied on the raw.

This is how I would make use of those personally but I understand most people are not thinking of adding this to their workflow. They just shoot RAW+JPEG

I’m just proposing that we might get rid of shooting RAW+JPEG and instead take advantage of those embedded jpegs in the metadata. Even at much lower resolutions those are completely usable for above described purposes.

I also understand this seems like a case of a solution looking for a problem. I think it’s a cool and radical idea and that many might like so I just wanted to share my thoughts here so everyone is aware of them and maybe as time goes on and people keep using JPEGs for various stuff and shooting RAW+JPEG maybe we can build this idea up into a real feature proposal that would be really helpful and maybe cover some use cases we’re not even aware of yet. Or maybe no one needs it or no one will care and we just let it stay here safe and sound, fishing for other interested ppl :smiley: :stuck_out_tongue:

3 Likes