Personally I would use duplicates to track this. You can name them and then use that as a variable in your exported file name to help track what is what…
Duplicates are peculiar to Darktable, at least in my case, so I’m still contemplating how to take advantage of that feature. However, sidecar files are used in all of the other software I’ve used for developing raw files (i.e., Canon DPP4, Rawtherapee, ART). Canon allows for updating the raw file to include such data. However, I’m a fan of non-destructive editing and make a point of NOT doing that and rather use the options of creating a separate sidecar file instead.
At present it looks to me like the Duplicate feature of Darktable is nice for when you can plan ahead. However, the case I had in mind, that triggered this post, is the one where you produced what you thought was a final result, possibly long ago, and the processing for an image is done. Then for some unexpected/unplanned reason you decide to try something else on that previously (as in yesterday or a year or more ago) developed raw file. Having the sidecar file is an approach that works for all that other software. I’m inclined to think that generally speaking you cannot count on the sidecar file produced by an old version of the software to work properly with the version presently being used. My solution to that problem is to keep all of the old versions of the software which allows for also using such a sidecar file with the proper version of the program. This gets into a different post I made on this forum advocating for running such software in portable fashion. The only reason for bringing that up now is that I’ve also figured out that Darktable is NOT amenable to that type of installation. The problem appears, to me, to be the use of SQL Databases that must update in real time. USB portable drives are typically much slower at writing than reading data and when I did this the performance degradation of Darktable was so bad as to render it unusable. This is another problem that I’m still working on but thought it better to learn how to use it before worrying about how to install it. For now I am using a conventional installation. Anyway the idea of putting release dependent data in my work product is an anathema to me.
Yes, darktable too. When you use the duplicates feature, you get multiple XMP sidecar files, one for each duplicate.
darktable is non-destructive all the way through and it can make multiple sidecar file per raw file, so it isn’t clear what you’re talking about.
No, I use it for exactly what you’ve described above; this is the duplicates feature. I used the duplicates feature when switching from the display referred to scene referred workflow. I didn’t want to loose some of the ideas I had done when editing in display referred, but also wanted to use scene referred. So now for a lot of my older photos, I have a sidecar for each: the old displlay ref’d edit, and the new scene ref’d.
This is technically true, it is a best effort to keep backward compatibility. But the great thing about FOSS is that you can grab whatever version you want, compile that version, and use it.
Yes… but is this a USB2 drive? USB3 should be plenty fast.
I think you’ll be very sad with a lot of software then, darktable included.
TL;DR: I think you need to spend some more time poking the duplicates feature.
1 Like
Exactly…creating a duplicate is all about the side car and if you name the duplicates you can have a preset that uses version_name variable added to the export so you know exactly what the export is wet other exports… This should cover the original question or I am missing something. Your duplicate will give you a fresh side car with standard processing ie original option or it can be a duplicate to keep all the steps then you got to the spot in the history you want to start from and make your new edit…this is the new sidecar now for this new edit?? To me this sounds like the question and use case presented by the OP of this thread??
You can bypass the use of the darktable database, and use only sidecar files. See:
https://docs.darktable.org/usermanual/3.8/en/special-topics/program-invocation/darktable/
and read the description of --library <library file>
1 Like
Its really not too hard and you can point everything to a copy on a usb or wherever. You need to install it once somewhere but it can then be made portable…
Keep in mind that a program reading xmp metadata (or any other xml) should ignore tags it doesn’t understand. Processing information is almost by definition specific to the program that generated that information, so should be ignored by other software. On the flip side, so far, darktable devs managed not to break old edits, even while deprecating some (common) modules.
Writing to raw files has its own set of problems, especially for software like darktable: part of the metadata is in a proprietory section, the “maker notes”. Contents of that section are not published by camera makers, so only their software knows the ins and outs ot the maker notes. That makes writing metadata to raw files a bit dangerous for anyone else. And I have seen some stories where even the maker’s program got it wrong, making raw files unreadable for other software…
Another issue is whether you want a third party to have your processing information (or anything else you put in the metadata four your use, like geolocation data). But that’s a different discussion.
As @paperdigits already said, not at all. I used it in the exact same way he described when filmic came out (still do). I also use it when I want to try another approach to editing an image. Duplicates then give me a kind of before/after view, even a few days later (often a better time to compare the two versions anyway).
I do run Linux on some computers. While I find it to be a good server system it is NOT so good for what might be called desktop applications. Windows ability to run programs in what’s called portable mode allows subsequent versions of programs to be added without making any alteration to the prior version. In my case, it involves updating a script used to start the program. Therefore, I’m always able to start every version of that application that I’ve ever added in the past. (Note: I’ve used the word add in order to avoid confusion with Windows’ use of the work install).
While the process works best for applications packaged for adding without running what Windows calls an installer the Windows installers typically allow selection of a path other than on the C: drive for locating the software. There can be issues for programs that make extensive use of the Windows registry or hard code the location used for user specific data. The later can be addressed by inserting junctions or symbolic links in the file system. The data stored in the registry is seldom problematic.
Bottom line is that it is much easier than you suggest for me to invoke prior versions of my portable applications. In fact, it is NO more difficult than invoking the current version.
No! My case is with a USB 3 solid state drive. I’ve decided to put off further experimentation in favor of trying to learn how to use Darktable. If that succeeds I’ll probably continue investigating this problem.
The Darktable (DT) concept of Duplicates is new to me. I do see merit to this idea but my present idea/problem is how to fit DT into a workflow that already exists and does include the use of lots of other software. My real interest in DT pertains to developing raw files. The whole Lighttable, library (database) idea is something I haven’t yet seen a need for. With that said, if I can learn to develop raw files well enough with DT I will probably try to learn more about Lighttable. However, at present, I do NOT foresee permanent maintenance of DT libraries (i.e., SQL databases) becoming part of my workflow.
I might point out that I do some of what Lighttable appears to offer by using hard links in the file system to create what I call Galleries (i.e., the same image file can be accessed via multiple directory entries) but appear to be pretty close to what DT calls Collections.
Just run it with :memory: as the library option Then you basically can use it as an editor for as much of your workflow as you want and move on…you can likely forego the database given how it sounds like you will use DT.
I have tried this and I do like the fact that such an option exists. However, as mentioned elsewhere I need to learn how to develop raw files with Darktable before getting to do deep into such features.
As mentioned, metadata is a huge problem of its’ own making. One of the things that has driven me toward Darktable (DT) is that I acquired some new Canon cameras that surprise-surprise use a different format for raw files than my prior Canon cameras. DT seems to support the new format whereas Rawtherapee does NOT.
I do make extensive use ExifTool, including GUI for ExifTool to customize the metadata. This also includes scripts I’ve written to do various things such as extracting preview images. I’m pretty sure I can remove unwanted XMP groups should that be desired.
Therefore the embedded metadata is NOT a big problem but it does seem like something that would obviously warrant being optional.
Probably because I use lots of different programs to process raw files I definitely favor non-destructive editing. To me that means the raw files are NOT altered in any way. The mention of “Makernotes” does ring true to me. When I tried using Adobe DNG Converter to solve my raw format problem I found that the Makernotes completely disappeared from the resulting .dng format files. Likewise, it omitted the camera produced preview image in favor of one of its’ own making. Bottom line is that while .dng format might preserve the raw sensor data the file itself differs from the camera produced raw file in a lot more ways than just format.
This is an option that interests me. Might that be a reason for wanting to preserve the processing parameters in a sidecar file rather than a Duplicate?
Again, duplicates and sidecar files are not mutually exclusive, as your phrasing makes it seem. If you have XMP writing turned on in darktable, when you use the duplicates feature, a new sidecar file is created. The Duplicates feature let’s you select which edit you want to use by using buttons in the GUI.
1 Like
Just once again to clarify in DT the duplicate is just an extra xmp file with your additional edits…say two versions of your edit…one + filmic and one without as an example…Your option to make a duplicate creates a sidecar copy that is either an exact copy of your first edit or one that would be the standard created when you open and image (called original in the duplicate manager)…. I could be wrong but is this clear as to how it works… the duplicate option is not distinct from the use of sidecars??
I thought Darktable was writing a sidecar file that contained the processing parameters for the file being exported with the name of the exported file (i.e., the image file being created) suffixed with .xmp. However, that is NOT happening now.
In the settings there is a selection identified as “Storage>write sidecar file for each image” which has 3 options that are “on import, never, after edit”. At present it is set to “on import” which I think is the default that has not been changed. However, this sounds to me like it pertains to the file being edited (e.g. raw file) as opposed to the file being exported (e.g., image file created from raw file).
I can NOT find a setting for writing a sidecar file when an image file is created (i.e., exported). I think this should be a sidecar file with the same content as an up-to-date version of the one associated with the raw file being edited.
Could use some help finding the referenced setting?
That sidecar would be the sidecar that is saved along side your image…
On import creates a basic one on import
On edit the same …so this was added as users don’t like sidecars for files not yet edited…
None will use the database I presume I have never set it that way…
This is how the files are managed…
If you use duplicates and name them…
You can append something like _$(VERSION_NAME) from my example exporting the first one will create a file with _Edit1 appended and the second Edit2 appended.
The xmp files will have a version number in this case 0 and 1. So if you look in you image directory this is where you will find those two sidecars along with your raw image.
Maybe you have a look at the manual if a software doesn’t work as you ‚thought‘.
1 Like