Folder Session Mode — proposta per migliorare Darktable senza stravolgere la sua modalità operativa.

Folder Session Mode — proposta per migliorare Darktable senza stravolgere la sua modalità operativa.

Ciao a tutti,
mi chiamo Lorenzo Fontanella, fotografo e utente di Darktable ormai da diversi anni.
Negli ultimi mesi ho lavorato su un’idea per rendere Darktable più flessibile e vicino ai flussi di lavoro di altri software, ovviamente senza cambiarne la filosofia originale.

Ritengo personalmente che date le ottime caratteristiche che lo contraddistinugoano anche rispetto ad altri software “Professionali” sarebbe una valida scelta per “sostituire” molti altri software.
Tuttavia ci sono alcuni aspetti che, a mio avviso, ne limitano l’adozione.

Questa proposta nasce da un problema pratico che molti di noi conoscono bene:
oggi, per poter lavorare su delle immagini, bisogna importarle prima nel catalogo.
Solo dopo l’importazione, si possono visualizzare, modificare e gestire.
Questo approccio funziona bene per grandi archivi, ma può diventare pesante o limitante in altre situazioni.


Situazione attuale

  • Ogni file RAW deve essere importato nel catalogo.
  • Le modifiche possono essere salvate nel database o nei file sidecar .xmp accanto ai RAW.
  • Se il catalogo si danneggia, è un problema.
  • Se si salvano i sidecar, le cartelle diventano caotiche (piene di file .xmp).
  • Se si aggiungono nuovi file a una cartella già importata, bisogna ricordarsi di rifare l’import.
  • Se si spostano cartelle, il rischio è che il database centrale non le riconosca più, in quel caso, si incorrerebbe anche nella problematica di avere delle ramificazioni e delle anteprima già generate, ma con percorsi nulli.
  • In generale, con grandi volumi di immagini, il database cresce rapidamente e rallenta il software.

In breve, il sistema funziona, ma per molti utenti è troppo legato alla logica del “catalogo unico”, e tutto ciò che esso comporta, quindi alcuno, molti, evitano o di cambiare applicativo, poiché si trovano già con questo problema (vedi lightroom) oppure semplicemente non vogliono dover rifare tutto da zero.


Situazione futura (proposta)

L’idea è di introdurre una modalità alternativa e opzionale, chiamata Folder Session Mode.

In questa modalità:

  • Darktable può aprire direttamente una cartella del sistema, senza importazione preventiva.
  • Nella cartella, il programma creerà automaticamente una sottocartella dedicata (PREVIEWS/) dove salverà:
    • le anteprime in JPEG, selezionabili per dimensione, in modo da stabilire quanto devono essere “pesanti” da gestire.
    • i file sidecar relativi alle lavorazioni eseguite,
    • un piccolo file session_info.yaml che contiene tutte le informazioni relative a quella cartella.

Esempio:

/Servizio_Matrimonio/
 ├── RAW/
 ├── PREVIEWS/
 │   ├── IMG_001_preview.jpg
 │   ├── settings/
 │   │   ├── IMG_001.xmp
 │   └── session_info.yaml

Questo piccolo file YAML sostituisce, in modo leggero, una parte del database per quella cartella:
contiene le impostazioni, i riferimenti e le miniature, rendendo quella sessione portabile e indipendente.


Come si potrebbe integrare

  • Il catalogo principale rimane invariato.
  • Ogni cartella lavorata, e quindi già mappata, diviene una mini-sessione autonoma.
  • Il database principale può eventualmente tenere solo un indice di puntamento alle sessioni presenti, e con una funzione di aggiornamento automatico o manuale, verificare che nulla sia cambiato oppure sincronizzare i cambiamenti.
  • Se si apre una cartella, da DarkTable, il software in automatico aggiornerà automaticamente le anteprime e le informazioni YAML di quella sola cartella, in modo che terrà subito traccia di eventuali modifiche.

Inoltre, alcuni software già funzionano con questa logica, (vedi Capture One) , questo offrirebbe tra i vantaggi, il fatto che gli utilizzatori utilizzano già quel metodo di lavoro, quindi la migrazione diverrebbe più veloce, inoltre si potrebbe valutare una procedura di import da altri software di editing (Lightroom, Capture One, Exposure, ACDSee), in che sarebbe solo un bene per tutta la community e per DarkTAble in generale.


Vantaggi pratici post integrazione

  • Nessun obbligo di importazione. Si può aprire e lavorare direttamente da una cartella.
  • Niente sidecar sparsi. Tutti i .xmp sono ordinati in PREVIEWS/settings/.
  • Backup facilissimo. Basta copiare la cartella.
  • Scansione automatica e riassegnazione percorsi logici, in caso di spostamento cartelle fisiche, senza dover rigenerare le anteprime interne, poiché già contenute nelle sottocartelle dedicate.
  • Indicizzazione automatica e generazione nuove anteprime per tutti i file creati ex-novo (ad esempio in fase di esportazione da altri software di editing, che generalmente esportano a fianco del file di riferimento)
  • Database più leggero e veloce.
  • Compatibilità con altri software.
  • Adozione immediata da parte di molti più utilizzatori
  • Crescita della Community
    Struttura molto simile a quella usata da Capture One (Sessions) o RawTherapee.
  • Facile da condividere. Si può spostare o archiviare una sessione senza rischiare di rompere il catalogo.

Compatibilità e filosofia

La cosa importante è che questa sarebbe una modalità opzionale.
Chi preferisce il flusso classico con catalogo può continuare a usarlo esattamente come ora.
Non cambia nulla per chi non attiva la funzione.

L’obiettivo è di proporre l’aggiunta di una seconda possibilità di utilizzo per chi lavora per progetti, servizi o sessioni singole, mantenendo Darktable sempre coerente con la sua filosofia:
libero, flessibile e modulare.


Stato del progetto

Ho già preparato una bozza tecnica (RFC) con codice sorgente dimostrativo, diagramma e documentazione, disponibile qui:

Io non sono un programmatore, ma Fotografo e fruitore, non lo so fare, quindi NON è un codice pronto per il merge, ma una proposta concettuale concreta, che mostra come la funzione potrebbe essere integrata nel core di Darktable senza rompere nulla del sistema attuale.


Invito alla community

Questa proposta non vuole cambiare Darktable, ma ampliare le possibilità di un software che personalmente ritengo eccezionale, ma, e per questo, complesso.
Credo che rendere il programma più aperto anche ai flussi “session-based” possa aiutare molti utenti, anche nuovi, ad avvicinarsi a Darktable, quindi nel complesso aiutare tutti.

Mi piacerebbe sapere cosa ne pensate, soprattutto dal punto di vista tecnico e dell’integrazione con il database e la gestione delle anteprime.

Grazie a tutti per l’attenzione e per l’enorme lavoro che fate ogni giorno su Darktable :heart:
Lorenzoeffe

Well, a post for discussion in Italian on a forum where the common language is Italian.

Dat zal heel veel antwoorden krijgen… (That’ll get a lot of replies)

2 Likes

I agree with rvietor. While I think italian sounds and reads awesome I don’t understand anything of what you wrote. Maybe push your post through some translation tool and try again :slight_smile: .

@Lorenzoeffe do you know that you can configure DT to not generate XMPs at all, and that you can specify the DB path when you start it? So, if you want, you can store a different db in each directory. You can write a shell script to which you pass a directory path and it will start darktable with a new db in that directory. No code changes required.

Using the --memory option you can create a virutual db for each session and this will be erased when you exit. I am not sure if this is essentially the crux of what the OP is trying to do…

One can even supply an image or a directory to open.

<input file>|<image folder>
Optionally supply the name of an image file or folder. If a filename is given darktable starts in darkroom view with that file opened. If a folder is given darktable starts in lighttable view with the content of that folder as the current collection.
(darktable user manual - darktable)

The GitHub repository is useless AI-junk. A very similar topic was already deleted this morning.

1 Like

I think the OP is genuinely trying to suggest something that he would find useful. He doesn’t know that DT can already by and large support his use case (i.e., per directory db). He is not a developer and he used an LLM to generate something plausible to back up his proposal. He has been a bit cumbersome and naive in his approach to this forum but I believe that he is well intentioned.

1 Like

but it isn’t that plausible if you actually ready it. He obviously did not look at the github repo at all, and instead just passed it along to us. If you can take the time to look at your own AI slop, why should we? It is very rude.

I don’t believe his intentions are bad, the attempt is just misguided.

1 Like

Well, plausible for someone who has no idea what they are doing :slight_smile:

Folder Session Mode — a proposal to improve Darktable without disrupting its operating mode.
Hello everyone,
My name is Lorenzo Fontanella, photographer and Darktable user for several years now.
In recent months, I have been working on an idea to make Darktable more flexible and closer to the workflows of other software, obviously without changing its original philosophy.
I personally believe that, given its excellent features that set it apart from other “professional” software, it would be a valid alternative to replace and even “definitively replace” many other editing software programs.
However, there are some aspects that, in my opinion, limit its adoption.
This proposal stems from a practical problem that many of us are familiar with:
today, in order to work on images, you first have to import them into the catalog.
Only after importing can you view, edit, and manage them.
This approach works well for large archives, but can become cumbersome or limiting in other situations.


Current situation
• Each RAW file must be imported into the catalog.
• Possible changes are saved in the database or in the .xmp sidecar file alongside the RAW files.
• If the catalog becomes corrupted, this is a problem.
• If you save the sidecars, the folders become chaotic (full of .xmp files).
• If you add new files to a folder that has already been imported, you must remember to re-import it.
• If you move folders, there is a risk that the central database will no longer recognize them. In that case, you would also encounter the problem of having branches and previews that have already been generated but with null paths.
• In general, with large volumes of images, the database grows rapidly and slows down the software.
In short, the system works, but for many users it is too tied to the logic of the “single catalog” and everything it buys, so some, many, avoid changing applications because they already have this problem (see Lightroom) or simply do not want to have to start all over again from scratch.


Future position (proposal)
The idea is to introduce an alternative and optional mode called Folder Session Mode.
In this mode:
• Darktable can open a system cartridge directly, without prior import.
• In the folder, the program will automatically create a dedicated subfolder (PREVIEWS/) where it will save:
◦ JPEG previews, selectable by size, so that you can determine in advance how “heavy” they should be to manage.
◦ Sidecar files related to the processing performed.
◦ A small session_info.yaml file containing all the information related to that folder.
Example:
attached image

This small YAML file replaces, in a read-only manner, part of the database for that cartridge:
it contains settings, references, and thumbnails, making that session portable and independent.


How could it be integrated?
• The main catalog remains unchanged.
• Each folder that has been processed, and therefore already mapped, becomes an autonomous mini-session.
• The main database can only hold an index pointing to the existing sessions, and with an automatic or manual update function, verify that nothing has changed or synchronize the changes.
• If a cartridge is opened from DarkTable, the software automatically updates the previews and YAML information for that folder only, so that it immediately keeps track of any changes.
Furthermore, some software already works with this logic (see Capture One), which would offer the advantage that users coming from other software, already accustomed to this method, would feel more comfortable and therefore be more inclined to migrate to DarkTable, thus speeding up the migration process.
In addition, an import procedure from other editing software (Lightroom, Capture One, Exposure, ACDSee) could be evaluated.
All of this would only be good for the entire community and for DarkTable in general.


Practical advantages after integration
• No import required. You can open the software and work directly from a folder.
• No scattered sidecars. All .xmp files are sorted and placed in the PREVIEWS/settings/ subfolder.
• Easy backup. Just copy the folder and the mini catalog will always be contained within it.
• Automatic scanning and reassignment of logical paths, in case of physical folder relocation, without having to regenerate internal previews, as they are already contained in dedicated subfolders.
• Automatic indexing and generation of new previews for all newly created files (e.g., copies generated outside DarkTable or resulting from export from other editing software, which generally export alongside the reference file).
• Lighter and faster database.
• Compatibility with other software.
• Immediate adoption by many more users.
• Community growth.
Structure very similar to that used by Cattura uno (sessions) or RawTherapee.
• Easy to use. You can support or archive a session without the risk of scrolling through the catalog.


Compatibility and philosophy
The important thing is that this would be an addition to the normal operating mode.
Those who prefer the classic catalog flow can continue to use it exactly as they do now, by disabling Folder Session mode, but I believe that once implemented, the advantages would be such that it would be preferable.
Nothing would change for those who do not activate the function.
The goal is to offer a second option for those who work on projects, services, or individual sessions, while keeping Darktable consistent with its philosophy:
free, flexible, and modular.


Project status
I have prepared a technical draft (RFC) with demonstration source code, diagrams, and documentation, available here:

I am not a programmer, but a photographer and user, so I don’t know how to write code. That’s why what I’m proposing is NOT code ready for merging, but a concrete conceptual proposal that shows how the feature can be integrated into the core of Darktable without breaking anything in the current system.


Invite the whole community
This proposal is not intended to change Darktable, but to expand the possibilities of a software that I personally consider very complete, rich, and vast, and therefore… exceptional. But, for this reason, it is also quite complex.
I believe that making the program more open to external “session-based” workflows could help many current and new users to approach Darktable, and therefore, overall, help everyone to grow.
I would like to know what you think, especially from a technical point of view and in terms of integration with the database and preview management.
Thank you all for your work and for the enormous effort you put into Darktable every day.
Lorenzoeffe

@Lorenzoeffe are you also reading what we write? You can already have per-folder db and no XMPs in the directory with the images. It’s just a matter of using the right command line arguments.

Yes, I read.
As far as I know. I don’t think that’s possible.
Is it?
How?
Could you tell me what to read about it?
Thank you.

Yes, this morning it was still me.
I used a clumsy and wrong approach, and I apologise.

Unfortunately, the user manual is not available in Italian.

When you start darktable, you can pass command-line options.
One such option is --library :memory: – in this case, no database is created. Or you can pass --library c:\path\to\library.db (with a script, you could replace the fixed path with the image directory you are opening, and store a library file per directory).
You can also pass the path to the directory (or image file) you want to open.
This is all described at darktable user manual - darktable.

Maybe it would be a lot easier to create a script to launch darktable that way, and disable XMP generation in darktablerc, if you don’t want sidecars.

If you have questions, ask.

Very good!
I understand, I’ll definitely give it a try.
Unic
The fact that it generates sidecar files is great, especially because if there weren’t a central database, it would be a problem.
One question:
Can I have them generated in a subdirectory, rather than next to each image file?

If you don’t save the sidecars you can’t restore your edits if the database becomes corrupted. The sidecars are the database backup of last resort.

In fact, I’m fine with sidecar files being created, but I’d prefer them not to be next to the other image files, but in a dedicated subfolder.
And I believed that the “mini-catalogue” system distributed across folders was an improvement for one simple reason…
I use Digikam to manage my database.
I have noticed that importing everything into DarkTable slows it down or causes it to crash.
I am talking about almost 400k files including raw files from various cameras and JPEGs.
That is why a system based on sessions and mini-catalogues within folders, which connect to a central database, would be ideal in my case.

I use a system whereby when I export my edits to jpg or other formats for viewing I use the variables in the export dialogue to create a mirrored set of folders.

So if my raw files or unedited files are in say just for example C:\photos\dir1\dir2 or whatever structure then on export they will be saved to C:\photoedits\dir1\dir2.

I can then view this set of folders with any viewing software or the files system without xmp files in the mix and I can either use a separate library other command line option like the :memory one and view them in DT without adding xmp to that set of folders… This is my way to separate them ie the edited files from the starting images and have it xmp free…

This seems to work pretty well for me but might not work for at all for you or others…I’m just throwing it out as another way to manage the intermingling of xmp files…

Since you mentioned other software. DT can be set to not write sidecar files and only store the edit steps in the DB. This is similar to what Lightroom does, but I see that as a weakness because if the database gets corrupted your editing steps are lost. I have set DT to only write xmp files upon editing. I don’t need an xmp file if I haven’t edited. But when I have edited the xmp files being stored as a sidecar file in the same folder has proven a godsend when moving folders to external drives or having my DT database corrupted because of my own stupidity.

One suggestion that may achieve what you want is to sort the folder post editing and select all the xmp files and cut and paste them to a new location. Personally I don’t see the problem of the xmp files being a sidecar file in the same folder. Looks messy at first but them when something goes wrong I really appreciate them being where they are.

Good luck with DT. I find it a great program.

Perhaps I didn’t explain myself clearly, or you didn’t read carefully.
I would like sidecar files, I would like a centralised database, I would like YAML databases in individual folders, I would like sidecar files generated in their respective subfolders, and I would like all of these things to coexist simultaneously. It is possible, given that other software already does this.
Having only one of these things is reductive, and since it seems like a good idea to me, I submitted it to you…
Answer… If you don’t want the sidecar files or the database, no problem, just disable it… What the f***!
I don’t know how to explain it, but today, just like five years ago, I encountered the same resistance, and I wonder why!