When a raw file is developed and then exported to an image file (e.g., jpg, tif, png, etc) a sidecar file gets created using the filename of the image file with an ext of xmp added. When subsequently editing that same raw file it may be desired to start in the same state as the exported image file. This is pretty common and intuitive in other software but I’m having a hard time figuring out how to do it in Darktable.
See “develop history” in the section darktable 3.8 user manual - export
You can embed the processing parameters in your export file, then later load the metadata of the exported file and it’ll load the history.
In the light table view , on the right side there are buttons for ‘write xmp’ and ‘read xmp’.
Write will always use the filename of the input file, with read you get a file selector.
And the history is embedded in jpegs by default i believe , so you can ‘read a jpg as xmp’ to get the history from it.
Somewhere in the options is a setting to always write a sidecar file , or not (keep it only in the database).
This is good to know but typically the objective is to create what I might call conventional image files (i.e., those that can be used by a wide variety, ideally all, other software/apps that support said file types. Of course metadata is a part of that and a whole other story with lots of complications of its’ own. I’d be concerned that adding such information, as XMP type metadata which certainly is NOT confined to the XMP processing parameters contained in sidecar files, could have undesired consequences and of course a major problem is that I have NO way to know what software/apps the image files I create might encounter.
Sidecar files a good, from my way of thinking, because they have the effect of keeping all of the post processing considerations out of the final image files.
Looks like there is more to this than I would have thought which increases the benefit for posing this question above what I originally expected.
However, what appears to be called “load sidecar file” and “write sidecar file” in the History section of the right panel looks to be just what I contemplated should be possible.
My prior reply to Mica explains my present thinking regarding metadata. While being a newbie on Darktable I am undertaking a lot of rethinking my present idea is that I’d prefer NOT to have this information stored in the metadata of my image files. This brings up another question which is “how to turn it off?”.
Wouldn’t you just create a duplicate…then start your new edit???
Well darktable’s processing instructions have their own namespace in the XMP header of the output file, so another application would have to specifically try and read that metadata in order to try and make use of it.
If you want a sidecar file, you’ll need to use the duplicates feature and stop developing the image after you export.
Personally I would use duplicates to track this. You can name them and then use that as a variable in your exported file name to help track what is what…
Duplicates are peculiar to Darktable, at least in my case, so I’m still contemplating how to take advantage of that feature. However, sidecar files are used in all of the other software I’ve used for developing raw files (i.e., Canon DPP4, Rawtherapee, ART). Canon allows for updating the raw file to include such data. However, I’m a fan of non-destructive editing and make a point of NOT doing that and rather use the options of creating a separate sidecar file instead.
At present it looks to me like the Duplicate feature of Darktable is nice for when you can plan ahead. However, the case I had in mind, that triggered this post, is the one where you produced what you thought was a final result, possibly long ago, and the processing for an image is done. Then for some unexpected/unplanned reason you decide to try something else on that previously (as in yesterday or a year or more ago) developed raw file. Having the sidecar file is an approach that works for all that other software. I’m inclined to think that generally speaking you cannot count on the sidecar file produced by an old version of the software to work properly with the version presently being used. My solution to that problem is to keep all of the old versions of the software which allows for also using such a sidecar file with the proper version of the program. This gets into a different post I made on this forum advocating for running such software in portable fashion. The only reason for bringing that up now is that I’ve also figured out that Darktable is NOT amenable to that type of installation. The problem appears, to me, to be the use of SQL Databases that must update in real time. USB portable drives are typically much slower at writing than reading data and when I did this the performance degradation of Darktable was so bad as to render it unusable. This is another problem that I’m still working on but thought it better to learn how to use it before worrying about how to install it. For now I am using a conventional installation. Anyway the idea of putting release dependent data in my work product is an anathema to me.
Yes, darktable too. When you use the duplicates feature, you get multiple XMP sidecar files, one for each duplicate.
darktable is non-destructive all the way through and it can make multiple sidecar file per raw file, so it isn’t clear what you’re talking about.
No, I use it for exactly what you’ve described above; this is the duplicates feature. I used the duplicates feature when switching from the display referred to scene referred workflow. I didn’t want to loose some of the ideas I had done when editing in display referred, but also wanted to use scene referred. So now for a lot of my older photos, I have a sidecar for each: the old displlay ref’d edit, and the new scene ref’d.
This is technically true, it is a best effort to keep backward compatibility. But the great thing about FOSS is that you can grab whatever version you want, compile that version, and use it.
Yes… but is this a USB2 drive? USB3 should be plenty fast.
I think you’ll be very sad with a lot of software then, darktable included.
TL;DR: I think you need to spend some more time poking the duplicates feature.
1 Like
Exactly…creating a duplicate is all about the side car and if you name the duplicates you can have a preset that uses version_name variable added to the export so you know exactly what the export is wet other exports… This should cover the original question or I am missing something. Your duplicate will give you a fresh side car with standard processing ie original option or it can be a duplicate to keep all the steps then you got to the spot in the history you want to start from and make your new edit…this is the new sidecar now for this new edit?? To me this sounds like the question and use case presented by the OP of this thread??
You can bypass the use of the darktable database, and use only sidecar files. See:
https://docs.darktable.org/usermanual/3.8/en/special-topics/program-invocation/darktable/
and read the description of --library <library file>
1 Like
Its really not too hard and you can point everything to a copy on a usb or wherever. You need to install it once somewhere but it can then be made portable…
Keep in mind that a program reading xmp metadata (or any other xml) should ignore tags it doesn’t understand. Processing information is almost by definition specific to the program that generated that information, so should be ignored by other software. On the flip side, so far, darktable devs managed not to break old edits, even while deprecating some (common) modules.
Writing to raw files has its own set of problems, especially for software like darktable: part of the metadata is in a proprietory section, the “maker notes”. Contents of that section are not published by camera makers, so only their software knows the ins and outs ot the maker notes. That makes writing metadata to raw files a bit dangerous for anyone else. And I have seen some stories where even the maker’s program got it wrong, making raw files unreadable for other software…
Another issue is whether you want a third party to have your processing information (or anything else you put in the metadata four your use, like geolocation data). But that’s a different discussion.
As @paperdigits already said, not at all. I used it in the exact same way he described when filmic came out (still do). I also use it when I want to try another approach to editing an image. Duplicates then give me a kind of before/after view, even a few days later (often a better time to compare the two versions anyway).
I do run Linux on some computers. While I find it to be a good server system it is NOT so good for what might be called desktop applications. Windows ability to run programs in what’s called portable mode allows subsequent versions of programs to be added without making any alteration to the prior version. In my case, it involves updating a script used to start the program. Therefore, I’m always able to start every version of that application that I’ve ever added in the past. (Note: I’ve used the word add in order to avoid confusion with Windows’ use of the work install).
While the process works best for applications packaged for adding without running what Windows calls an installer the Windows installers typically allow selection of a path other than on the C: drive for locating the software. There can be issues for programs that make extensive use of the Windows registry or hard code the location used for user specific data. The later can be addressed by inserting junctions or symbolic links in the file system. The data stored in the registry is seldom problematic.
Bottom line is that it is much easier than you suggest for me to invoke prior versions of my portable applications. In fact, it is NO more difficult than invoking the current version.
No! My case is with a USB 3 solid state drive. I’ve decided to put off further experimentation in favor of trying to learn how to use Darktable. If that succeeds I’ll probably continue investigating this problem.
The Darktable (DT) concept of Duplicates is new to me. I do see merit to this idea but my present idea/problem is how to fit DT into a workflow that already exists and does include the use of lots of other software. My real interest in DT pertains to developing raw files. The whole Lighttable, library (database) idea is something I haven’t yet seen a need for. With that said, if I can learn to develop raw files well enough with DT I will probably try to learn more about Lighttable. However, at present, I do NOT foresee permanent maintenance of DT libraries (i.e., SQL databases) becoming part of my workflow.
I might point out that I do some of what Lighttable appears to offer by using hard links in the file system to create what I call Galleries (i.e., the same image file can be accessed via multiple directory entries) but appear to be pretty close to what DT calls Collections.
Just run it with :memory: as the library option Then you basically can use it as an editor for as much of your workflow as you want and move on…you can likely forego the database given how it sounds like you will use DT.
I have tried this and I do like the fact that such an option exists. However, as mentioned elsewhere I need to learn how to develop raw files with Darktable before getting to do deep into such features.
As mentioned, metadata is a huge problem of its’ own making. One of the things that has driven me toward Darktable (DT) is that I acquired some new Canon cameras that surprise-surprise use a different format for raw files than my prior Canon cameras. DT seems to support the new format whereas Rawtherapee does NOT.
I do make extensive use ExifTool, including GUI for ExifTool to customize the metadata. This also includes scripts I’ve written to do various things such as extracting preview images. I’m pretty sure I can remove unwanted XMP groups should that be desired.
Therefore the embedded metadata is NOT a big problem but it does seem like something that would obviously warrant being optional.