How to synchronize metadata and file names between darktable and other photo/metadata editors?

Software darktable
1

I am seriously considering using darktable as my image processing application. I will probably also use RawTherapee, Gimp and a few other applications. After lots of research and reading it appears that darktable has the following behaviors that will make interchanging data between these programs “problematic” to say the least:

  1. darktable only reads the metadata from the file once, unless you set the option to reread said metadata when darktable opens up. Now, this is only a partial solution because most people keep multiple programs open and switch between them. So, darktable will not reread the metadata. There does not seem to be a way to force darktable to re-import the metadata without closing the program and reopening it.

  2. darktable stores its metadata in non-standardly-named .xmp files. Essentially, it tacks on an additional .xmp extension AFTER any existing file extension. So, if the original file was photo.jpg, darktable creates photo.jpg.xmp. All other programs would replace the .jpg extension with .xmp, creating a file named photo.xmp (without the jpg part). Now, I understand the logic behind this. Simply changing the file extension might overwrite any .XMP file created by some other program. Using a non-standard file naming scheme sort-of ensures that darktable gets its own, fresh .XMP file to play with.

But this creates a big dilemma when attempting to use any kind of utility to rename files in bulk. I have done a lot of research and some of the programs I have found have been able to recognize that photo.xmp is a sidecar file for photo.jpg. But NONE of them have any clue that photo.jpg.xmp is a sidecar file for photo.jpg. So, it seems the only way to rename any files, once they have been touched by darktable, is to do it manually, one… at… a… time. Taking care to also rename the sidecar file and any other related files. darktable’s FAQ is quite adamant that they will NEVER include a way to rename files from within the program. This seems a little austere, considering how difficult they have made it to rename said files outside of the program.

darktable’s unique sidecar naming scheme also presents a problem when any other program wants to modify related metadata. No other programs recognize that photo.jpg.xmp is a sidecar file for photo.jpg. Therefore, if I want to use some other program to copy tags from darktable’s .XMP files back to the original .JPG file or even to a standardly-named ,XMP file, I seem to be out of luck there as well.

For an open-source program, it sure seems as if the creators of darktable have done everything they can to lock people in to ONLY using darktable for everything.

Now, I have discovered ExifTool, and it looks as if solving these problems MAY be possible using ExifTool. However, ExifTool has a very steep learning curve.

I also know about Lua and the API. However, that, also, has a very steep learning curve.

So, to my questions:
A) Has anyone figured out how to overcome these hurdles? Preferably using a program (or collection of programs) that is as simple as possible. I find it hard to believe that I am the first one to recognize these issues.

B) Is there a way to force darktable to re-import metadata from other sources without closing and reopening the program?

C) If I do contrive a way to update the metadata stored in darktable’s .XMP files, will darktable get confused? Everything I have read says that darktable prioritizes the data in its own database over its .XMP files. So, I cannot be sure if closing and reopening darktable (with the option to re-import metadata turned on) will read the changed data from darktable’s .XMP file. It seems that it may only re-import metadata from the OTHER file types and then, subsequently, overwrite its own .XMP file with only what IT wants to be in there.

2

File name isn’t really an important part of my workflow. I rename them on import and don’t change them after that. That being said, I don’t think any of the database-based software likes it when you change the filenames out from underneath them.

Regarding your issues with sidecar files, have you looked at digikam? It has been a while since I looked at it but I seem to remember it used the same xmp conventions as darktable.

It shouldn’t as long as both pieces of software use the same tags and you aren’t making changes concurrently.

Ultimately, if you want to use darktable as a DAM you will need to live with it’s limitations and adjust your workflow to match. Alternatively, you could use something else as a DAM and use darktable as a standalone editor in which case you could more or less ignore it’s library/lighttable.

It might help if you explained what your workflow looked like so that other people could make recommendations that were more in-line with what you would like to accomplish.

3

I agree with @dalto, you need to spell out your workflow for us, step-by-step. From what you have written, it seems a little weird, like you’re trying to rename raw files. For me, I rename & import my raw files in one step.

I can see why you’d want to rename the resulting jpeg or tiff file, but that doesn’t have much to do with darktable, since once you’ve exported that file, you’re essentially done with darktable.

For the record, both digiKam and Geeqie will read darktable’s style of sidecar.

4

First of all, I don’t really have a “workflow” yet. I am trying to figure one out. Said workflow will necessarily depend on what various pieces of software can/will do. The basic plan is to:

  1. copy files from cameras into a temporary folder.

  2. Cull bad images.

  3. Do lossless rotation if necessary.

  4. Use some kind of batch renaming program to rename the files according to a pre-decided file naming scheme. Part of the name will be the taken date from the EXIF data. Part will be a code to indicate some stuff about the image (doesn’t matter what that is). And part will be a sequential number that is unique within any of the previous codes. Given that some of my cameras save .JPG and RAW, I will need a program that recognizes file pairs and sidecar files. So far, the only one I can find is ExifTool. (Note: My file naming scheme does allow for additional descriptive text to be added later. I may not decide what that should be until after the image has been processed through darktable. Remember, I did not ask whether YOU want to rename files. I asked how I can rename files.)

  5. Do initial processing in darktable.

  6. Do whatever else is necessary in other programs. Because these other programs are separate from darktable, and darktable is non-destructive, I will have to “export” “finalized” files from darktable.

  7. At some point in the future, I may want to adjust the file name for all related files.

Quite frankly, my workflow is irrelevant here. I am not asking if I should want to do what I want to do. I am not asking for you to tell me what I should want to do. I am asking if anyone knows how to do what I said I want to do.

Due to the limitations and klunky way that darktable assigns tags, I will likely use some other program to assign all my tags. But then, I would like be able to make use of those tags while looking for images in darktable. I will definitely need to use some other program to edit most of the IPTC or XMP metadata for my files. I will likely also indicate initial ratings for these files using some other program. However, I know it is highly likely to want to change those ratings later. In fact, anyone who says they never change their mind on any of this data after they have first imported it into their main processing program is either fooling themselves or bullshitting everyone else. If there is no way to synchronize that dynamically changing information to and between other metadata locations, then it seems darktable truly is trying to paint everyone into its own private corner.

Again, for an open-source program, it really is starting to look as if the philosophy of the creators is “My way or the highway.”

5

I do step #1&#4 at the same time(first). I use Rapid Photo Downloader to do that. It has flexible rules and will probably support whatever you need. You can use it to add metadata(tags/keywords/etc) or you can do that in a later step.

I then import into darktable and do my culling there. Darktable is not great for culling so many people cull before they import into darktable using something like geeqie, digikam or similiar.

I use darktable to assign tags and other metadata and have not found it to be restrictive, it just has a learning curve. That being said, you could certainly do it in another program if you wish.

#7 is where you are going to have issues. This is true of pretty much any of the database oriented solutions though. The idea is that the source filename shouldn’t matter. It is sort of a fundamental concept and if you don’t agree with it you might be looking at the wrong kind of solution for your needs.

Once again, this is how these types of programs are intended to work. It sounds like you are looking for more of a traditional file-based workflow, that is not really what darktable is about.

That being said, if you don’t have a specific workflow you are accustomed to, it might be worth giving it a shot. Many people find there are a lot of benefits to an integrated, non-destructive workflow. It isn’t for everyone though.

6

I think both @dalto and I keep poking at the file renaming part of this because it isn’t clear what is gained by renaming the file so far into the workflow. I’d personally think you gain nothing by renaming any time after initial import and before “final” export. But I’d be happy to learn why it is important.

Perhaps you want to look at a more fully featured DAM, such as digiKam.

The jabs at darktable are not appreciated and will not make anyone more likely to answer your questions nor add features; in fact, it is likely the opposite.

Further, open source does not mean support every possible option. In the same vein, it doesn’t do exactly what I want it to does not equate to my way or the highway.

2 Likes
7

  1. Import the folder again using the tools at your top left of lighttable view.

  2. Yes its very annoying and not helpful for a multi tool workflow. I believe its because dt store tool settings etc in the xmp files not just regular metadata. These tool settings might then conflict with various file structures.

Exiftool is your best bet at managing these issues.

8

It’s really less about “my way or the highway” and more about not spending time focusing on this particular feature as the dev team might not see it as a priority.

On that note, let’s look at solving your problem. I think we’re roughly looking at how to rename files based on exif data and some other provided information.

Exiftool is a very capable program for doing this, but that capability does come with the caveat that you’ll likely need to read the manual. On the plus side, we might be able to work out an answer for you together, so you’re not really alone! :slight_smile:

@Elle has also written a bit more detailed information about using it on her website here:

Maybe some stuff there can be helpful for you?

If you had a more concrete example of what you might want to rename files as, we could start figuring out a good exiftool command to accomplish what you need. Do you have something that might work as a good example for us to work with?

Oh, also, what OS are you using? I ask because there are some awesome tools on linux for renaming files quickly and easily with advanced replace options (regular expressions using the perl rename command). On windows it might still be usable, but might require the use of something like cygwin or WSL (Windows subsystem for Linux). I’d have to look around for a good windows option as I normally just use cygwin when I’m on windows to maintain my sanity. :wink:

9

I use a freeware app called Bulk Rename Utility. It has a CLI counterpart.

image
image926×704 74.5 KB

1 Like
10

For exiftool on windows, you can make your command name the file name of the .exe file, then drag and drop files on to the exe.

11

I tag my images within darktable with the support of some lua scripts from here:

1 Like
12

I have done quite a bit of testing on how darktable and digikam play together. Like mentioned before, they use the same naming convention for xmp files and digikam can be configured to write metadata to xmp sidecars only. There are some subtle differences to how certain kinds of metadata is handled, e.g. color flags.

Yes, see https://redmine.darktable.org/issues/12304 for example. The only workaround I could find was to use darktable with a :memory: library (i.e. darktable --library :memory: . or change the config file). That way, starting darktable takes a bit longer, but when working with not-too-large folders it is acceptable for me.

For the other way around, changing metadata in darktable and having digikam reload it automatically, there are also some bugs (c.f. 397340 – Update image file timestamps if metadata is written to sidecar file [patch] 39833 – Kmail crashed after notification of new mail), which should be fixed in the upcoming digikam 6.0.0. :slight_smile:

In my opinion, the situation is still far from ideal, but when being careful it should be quite possible to use digikam for culling, organizing and managing descriptions/tags and darktable for editing.

1 Like
13

I do the first four steps in digikam since its version 6.0.0(which is still in beta). I used to use RPD for import, rename and initial sorting and Geeqie for initial culling but now since digiKam improved a lot and got way faster I do everything in there. I do prefer darktable over digikam for RAW processing though.

1 Like
14

Thank you all for your replies. I will answer many of them here, in this one post.

I have tried using Bulk Rename Utility. It is my go-to file renaming utility. However, I ran into a wall when it came to sidecar files. I did some research and the BRU author simply discounts this as an issue to be dealt with. So that’s out.

I am on Windows 10. Sorry, I should have mentioned that.

I am most definitely NOT merely interested in renaming files. I can do that with BRU. I am interested in synchronizing metadata and any file name changes, as stated in the subject of this original post.

I don’t rightly care whether you think I should be renaming my files after they have been imported into darktable. I want to be able to do that. If I have to delete all the files from darktable’s database and reimport every single one of them every day, I don’t really care, as long as all that metadata is stored in the files or sidecar files which darktable will read upon re-import.

I will take a look at the Lua scripts. However, I am not really interested in learning yet another scripting language just for one program. In the end, I would almost rather just develop my own Java program to do what I want, using one of the many ExifTool wrapper libraries for Java. There is really only a certain number of things that I want to do, so it would not be like completely rewriting XnView in Java. It would be more like writing a slightly more complicated bash script to run ExifTool with some conditionals thrown in for different file name formats.

I just wanted to know if there was something already available. However, it appears that there is not.

15

Digikam and exiftool are available on Windows. Both tools can rename files based on EXIF information. Digikam is graphical versus exiftool which is a command line interface only.
I don’t import to darktable at all. All I do is just open a raw I want to develop with darktable (from digikam). All tagging, sorting, etc. done in digikam. Both digikam and darktable set to write to XMP sidecars. Works a charm.

2 Likes
16

Bingo! You are a winner!

This is the exact kind of answer I was looking for. Plus, Digikam is free and open source.

It’s funny how the only way to get an answer on the internet is to state that the answer doesn’t exist.

So, if Digikam writes to an .XMP file, then I open the associated image file in darktable, does darktable use the exact same .XMP file?

I’m gonna try it out myself. But, if that’s what you are saying, then this will be perfect.

17

Or you could state your needs clearly and concisely, then answer questions that others may have to clarify the question at hand…

4 Likes
18

Um… I did. Other people just didn’t like the question I was asking. Stop being a jerk. The internet’s got enough people like you already.

19

Please stay friendly!

20

Tell that to the person who was a jerk.