Duplicated version number created for existing image

Software darktable
1

Originally posted to darktable-user list on Tue, Jul 25, 2023 while pixl.us was down. Guillermo Rozas (GR) kindly provided some responses and I’ve included their comments.

dt 4.2.1 (OBS), Linux Mint 21,Ubuntu 22.04 jammy

I have an image from March 2020 developed in darktable. I went back to it today to try another edit on it (its a monochrome rendition that I just can’t get ‘right’). The RAW image plus duplicates are in a group.

However, when I created a duplicate of the image in dt 4.2.1 Lightable, it was given version number ‘3’ - which already exists for that image (there are seven pre-existing duplicates). I saw this after I’d applied some edits. dt also gave the new duplicate a different ‘image id’ to the original RAW image.

My workflow is that I always create a new version (duplicate) of the base RAW for a different edit so I can trace back any final output that may result. My filenaming system is ‘_’ where filename is composed of ‘<YYYYMMDD_projectname_original camera filename>’. Original camera images are renamed during download onto my workstation via a bespoke script (ie outside dt). I use variables in the dt export module to ensure any output follows this format. This provides unique identification of every image and its derivatives across my libraries, even when intermediate tiffs are involved in say, focus stacks.

This is critical for me - I can’t have two different edits of a RAW with the same filename! Why has it happened and what can I do about it?

GR asked about xmp files and I replied:
a) xmp’s are named ‘<$VERSION>.RAWextension.xmp’
eg:
original raw: ‘20200325_BonallyWoods_NIK_1413.NEF’
version 3 xmp: ‘20200325_BonallyWoods_NIK_1413_03.NEF.xmp’
b) I’ve checked the files and the new duplicate ‘3’ has overwritten the existing xmp for the previous version 3. Also all xmp files in that group new have new modified dates - 24 July 2023, when I created the new duplicate

I subsequently posted the following 26 July.
I have extracted data from the dt db image table for the image and versions concerned from three backup instances of the dt database going back 3 years:
a) 2020-12-14 - pre-dt3.4.0, closest I can get to the original shot date
b) 2023-07-22 - dt4.2.1, before new duplicate created
c) 2023-07-24 - dt4.2.1, after new duplicate created showing duplicate version ‘****’

Hope this sheds more light on the issue.

pre-dt3.4.0-library.db from backup archive 2020-12-14
select key, value from db_info;
key value
version 30

select id, group_id, film_id,filename,version,max_version from images where filename = “20200325_BonallyWoods_NIK_1413.NEF” order by id;

id group_id film_id filename version max_version
3574 3574 135 20200325_BonallyWoods_NIK_1413.NEF 0 7
3578 3574 135 20200325_BonallyWoods_NIK_1413.NEF 1 7
3582 3574 135 20200325_BonallyWoods_NIK_1413.NEF 2 7
3583 3574 135 20200325_BonallyWoods_NIK_1413.NEF 3 7
3789 3574 135 20200325_BonallyWoods_NIK_1413.NEF 4 7
3790 3574 135 20200325_BonallyWoods_NIK_1413.NEF 5 7
3791 3574 135 20200325_BonallyWoods_NIK_1413.NEF 6 7
3792 3574 135 20200325_BonallyWoods_NIK_1413.NEF 7 7

library.db-snp-20230722155434 from backup archive BEFORE new duplicate created
select key, value from db_info;
key value
version 37

select id, group_id, film_id,filename,version,max_version from images where filename = “20200325_BonallyWoods_NIK_1413.NEF” order by id;

id group_id film_id filename version max_version
3574 3574 135 20200325_BonallyWoods_NIK_1413.NEF 0 7
3578 3574 135 20200325_BonallyWoods_NIK_1413.NEF 1 7
3582 3574 135 20200325_BonallyWoods_NIK_1413.NEF 2 7
3583 3574 135 20200325_BonallyWoods_NIK_1413.NEF 3 7
3789 3574 135 20200325_BonallyWoods_NIK_1413.NEF 4 7
3790 3574 135 20200325_BonallyWoods_NIK_1413.NEF 5 7
3791 3574 135 20200325_BonallyWoods_NIK_1413.NEF 6 7
3792 3574 135 20200325_BonallyWoods_NIK_1413.NEF 7 7

library.db-snp-20230724163328 from current backup AFTER new duplicate created
select key, value from db_info;
key value
version 37

select id, group_id, film_id,filename,version,max_version from images where filename = “20200325_BonallyWoods_NIK_1413.NEF” order by id;

id group_id film_id filename version max_version
3574 3574 135 20200325_BonallyWoods_NIK_1413.NEF 0 0
3578 3574 135 20200325_BonallyWoods_NIK_1413.NEF 1 3
3582 3574 135 20200325_BonallyWoods_NIK_1413.NEF 2 3
3583 3574 135 20200325_BonallyWoods_NIK_1413.NEF 3 3 ****duplicate
3789 3574 135 20200325_BonallyWoods_NIK_1413.NEF 4 3
3790 3574 135 20200325_BonallyWoods_NIK_1413.NEF 5 3
3791 3574 135 20200325_BonallyWoods_NIK_1413.NEF 6 3
3792 3574 135 20200325_BonallyWoods_NIK_1413.NEF 7 3
9744 3574 135 20200325_BonallyWoods_NIK_1413.NEF 3 3 ****duplicate

The dt database has an index, “images_filename_index” ON “images” ( “filename”, “version” ); which means that a ‘duplicate’ is related to a particular image filename. There are two implications from this which may be relevant to the issue I raised:

a) The index is is not unique, it allows duplicates. Therefore the database allows (and cannot trap) insertion of a new image table row with a version number that already exists for the given filename value. The fact I have a duplicate version for a filename suggests the dt code also does not trap this.

b) This index assumes filenames are unique across the whole dt database, which probably is not realistic given how cameras from the same manufacturer can generate common filenames.

While a unique id is given to each imported image by the dt db to ensure images with the same filename are permitted and can be handled, it seems the ‘duplicate image’ functionality does not recognise this potential.

Thanks in advance
Dusenberg

2

I run the windows versions of dt, and I frequently install the weekly update. Over the last couple of years I have run into the same problem of duplicate versions. I never found a solution other than reinstall the previous working version of dt. Eventually a new version would come out that did not do it.

3

Thanks for replying. I don’t understand how waiting for a new dt update will resolve the situation caused by a duplicated version number as I described?

Am I understanding correct: you have seen dt create an image version that has the same version number as an existing version for the same RAW image when you ‘create duplicate’ in lightable or darkroom?

If so how do you deal with these duplicated versions? What do you do about the overwritten xmp files of the previous existing version?

4

You should open a big report on GitHub.

5

Bug report opened: #14962

6

To begin, I have dt set to only use memory, so nothing is stored in the database. I put each raw into a separate folder, download the xmps from a Play Raw, and identify them by name of contributor. I create duplicates, say version 01 to 10, and load each contributor’s xmp into a version.

What I’ve seen happen if I come back to that folder to work on, say, an edit of my own, is when I create a new duplicate, it might create another version 01. If I immediately delete the new version, I find that the original 01 has not been affected.

If I try to create another duplicate, it will create a new 02. Again, if I delete it immediately, the original 02 is not affected. Whether that will work when using the database, I don’t know.

I also run into the problem of a dt version deleting the version 0 xmp from my folder. In either case, the only solution is to revert to an older version of dt and try again when another updated version comes out.

7

Thanks for explaining. That is very much what I have seen, but as I don’t use the versioning functionality as much as you, I didn’t know to check before I edited, or how to get out of the problem. Clearly there is an underlying issue/bug.

I need to check that with a persistent dt database (ie not memory) the history stacks of the original versions remain intact or are overwritten by the newly created duplicate with the same version number.