Corrupted my db file (maybe?)

Hi all.
Just been upgrading my dev version to the latest 3.9 build, and somehow in the process, I seem to have created some kind of corruption in my 3.8 database (even though I have them in separate directories).
I can no longer save a new collection in 3.8.
Anyone got any suggestions as to what I might try to enable the saving of further collections?
Thanks in advance.

OS: Linux Mint 20.3, Cinnamon 5.2.7
Kernel: 5.4.0-109-generic

Hi Bruce,
just to make sure (and because I fell for it in a different matter): the directory you try to write in is still writeable and you have still space left?

Well, I THOUGHT you might have been on to something there… I went and checked the folder properties and Group Folder Access was set to none.
But changing that to Create and Delete Files did not improve things.
Damn.
And yes, there is space on the drive.

1 Like

Not sure if I completely understood what you did. You ran 3.9 master build against your 3.8 database ?
This commit image flags : set image type from the beginning by AlicVB · Pull Request #11572 · darktable-org/darktable · GitHub changed the database version, so every database created by a dt version after this commit will no longer work with 3.8.
But maybe I misunderstood your question…

No, what I did was installed 3.9 in its own directory alongside my current 3.8 build.
Then, I use a separate bash script to launch each version. And each of those scripts instructs dt to use a custom directory for the database, the config and the cache.
So I don’t understand how my 3.8 database could have somehow become corrupted from this, as its folders are different to the folders used for my 3.9 build.
It’s really odd. I’ve not had this happen in the past.

Just to add more variables (and maybe help narrow down the problem)… I have just realised that I can no longer add new tags to images in my 3.8 build, either.

The first I would try is an integrity check of the database files (backup first for safety). Pragma statements supported by SQLite

Edit : if there is no evidence the next question would be : does darktable report any errors / warnings if launched in debug mode?

Does this mean : adding tags does not work if you try to attach a tag that already exists (has been already attached to another image).
Or does this mean : cannot create a new tag (tagging module → new)

And another question : did you try to work with (a copy of) your version 3.8 database on 3.9? What happens when the database conversion process from 3.8 to 3.9 is running? Any error messages? Can you work with your database afterwards on dev version 3.9?

Are you working on new images or previous ones…any chance you exposed your images to 3.9 and now they won’t work in 3.8

  1. I had a quick lok at the SQLite link you supplied and my eyes glazed over almost instantly. :slight_smile:
  2. Adding tags… I cannot create a new tag. I can add tags which already exist in the database to any images.
  3. Did I try to work on my 3.8 db in 3.9? No. Like I said, I launch each version from a separate batch script, and that script includes custom folders for db, config and cache. I COPIED my 3.8 db file to the new (3.9) folder, but I never ran 3.9 using the 3.8 db in the 3.8 folder. That’s why I’m confused by what happened. They SHOULD have been separate.
  4. There weren’t any errors when I converted my (copy of my) 3.8 db to 3.9. My 3.9 build runs just fine!
  5. Yes, I can work in 3.9 ok. But for my videos, I would prefer to still be working in 3.8 so that the average user is working on the same version I am.

Todd,
The images have been seen by 3.9, yes. But in a different copy of the db file.
Or are you suggesting that the .xmp format has changed and so 3.8 now has an issue with xmp files created/modified in 3.9?

I was…I might get time tonight to try that scenario…so 3.8.1 vs current master??

Actually, I was on 3.8.0+8~gfb39ffd86 (because I don’t understand enough about git to know how to get an exact version number!).

Well I have had too many pints of Guinness tonight to tackle this…I think though if you let your 3.9 write xmp files instead of turning if off that might be the issue but just conjecture on my part… I hope you can sort it out… I have little program that versions my xmp files so it this ever happens I should be good…I will try to find some time tomorrow to see if I could recreate the issue by doing this…

In this case a new entry has to be created in data.db, table tags

data-tags

where the new tag gets an id, and this new tag (tagid) is attached to your image assigning it to the imgid of your image (table tagged_images in library.db).

library-tagged-images

Both databases (data.db and library.db) are beeing extendet and have to be tested. You can run an integrity check via command line (example for data.db on Ubuntu) :

cd ~/.config/darktable
cp data.db data.db.bak
sqlite3 data.db
sqlite> PRAGMA integrity_check;
ok
sqlite> .quit

The sqlite executable on Ubuntu is called sqlite3. Here you find the documentation : Command Line Shell For SQLite

Then it seems unlikely that your database is damaged.

Quite understandable. But for this purpose you could possibly start with a fresh database on 3.8 and re-import the few images you need for your video. The new release 4.0 will come up soon.

Edit:

If you are not familiar with git you can download the source code for 3.8.1 release here : Release darktable 3.8.1 released · darktable-org/darktable · GitHub (scoll down to bottom of page, “Assets”).

Thanks for the suggestions.
I have decided to let this go. 3.9 works, and 4.0 will be out in a month.
I’ve done as Pehar suggested and started 3.8 with a clean database and I’ll use that for now.

One other question though… where does library.db live?

g’day Bruce,

Normally in /home/YOUR_UID/.config/darktable/

Have fun!
Claes in Lund, Sweden

Ah… d’oh!
I have a custom name for it in my bash script, so it’s not called library.db! What a clown.
Don’t mind me. I’m new here. I’m just learning. :slight_smile:

When running 2 or more versions…it can also be good not to write xmp for the advanced ones to avoid any potential issues by unintended modification…

Todd, good tip. One i should make an effort to remember! :smiley:

Did you figure it out??