I would be interested to know how others are maintaining their files for archival purposes.
Currently (on darktable) All of my images are maintained as RAW+XMP sets. As I need images for printing or distribution temporary files are created and then discarded. For anybody not familiar with my system (and darktable) my end product is basically unavailable.
I am contemplating producing a reduced set of my most important images that will also have attached 16 bit TIFFs in the mix.
I would be interested in how others are viewing this question of archival storage.
1 Like
I also keep RAW + xmp sidecar file. For some developments I also keep an 8-bit PNG file.
The idea is to keep āimportantā developments in case the raw editor changes in some way. I donāt keep more than 8 bits, because when I want to change an image and the the xmp should no longer work as expected, I would start from scratch.
Thomas ā¦ a nasty thought; what would happen if you were knocked down by a bus tomorrow? Would access to your body of work (in full 16 bit format) be realistically accessible to others? Or do you care?
I think the bus scenario is well worth considering. I output to jpeg, sometimes 4 files - reduced resolution in sRGB and AdobeRGB, and full resolution also in those two spaces. (I edit with an Adobe RGB monitor)
Monitors steadily improve. Whatās the most future-proof output type? I donāt know. How easy would it be to make a slideshow in 10 years time if you just had raws and XMPs (possibly obsolete).
Regarding raws, xmps amd jpegs, I think it would be good if you could turn on something in preferences so that dt would output an XMP whenever you output a processed image, and it had the same filename as the jpeg (or whatever file type).
A bus would be would be unfortunate 
. At the moment I develop for 8-bit output (print or screen). So, a full size 8-bit PNG is okay for me.
Thrown under the bus. 
I took your statement on PNGs to mean in addition to, which is reasonable.
Why PNG rather than TIF ā¦ both about the same size.
Does not make much difference, the output is the same. I prefer PNG compression instead of the (optional) compression of TIFF.
Darktable used to include the xmp in the output jpeg, such that you even could load such a jpeg instead of a xmp to apply the same processing to another picture. In recent versions, the output metadata can be controlled in a more flexible way, but still you could probably include the xmp in the output jpeg.
Yes, metadata is where it is at. TIFs and JPGs have that advantage over PNGs. I use PNGs when I want to share and view images on the internet or browser. Their appearance is more reliable than JPGs.
I have a two tier approach:
-
images that I define as āvery good or betterā are kept as RAW + XMP + (full resolution, very high quality) JPG. The RAW+XMP allow me to re-export and re-edit, the JPG is just in case RAW reading becomes impossible. In that remote situation, I expect the high quality JPG to be a reasonable trade-off for a much bigger 16bit TIFF (it also covers better the ābus scenarioā)
-
images that I define as āgood but must be keptā (mainly for emotional or documentary reasons) are kept only as half resolution, very high quality JPG. Re-edition of these images is very improbable, and many have technical problems anyway.
Of course, I can do this because I donāt take too many pictures. If I had an order of magnitude more images I would probably switch to a JPG-only archive (both because of the size and because I would have very little time to re-edit images).
But the most important part is: backup, backup, backup 
I make a 800x600 āproofā JPEG from every raw with a batch tool, and save both in a file hierarchy that keeps them close together. The JPEGs are in a path thatās served by a webserver, so the family can easily browse them. The JPEGs contain the processing chain used to produce them, and when I want to do a separate renditon, full-sized, small, TIFF for GIMP, etc., I use the proof JPEG processing as the starting point.
I have a second computer in the basement to which I rsync my ~/Pictures directory after I process a session. I also periodically copy the directory to a portable drive and give it to one of the offspring to store āoff-siteā.
Iāve worked on this workflow for about 5 years now, to the extent of writing my own software to support it. My key takeaway, with any software, is to use the original raw file as the essential record of the image; any processed file from that record is a rendition for a specific purpose, and can be deleted on a whim. Iāll even delete proofs and reprocess them based on another proofās tool chain, which is how I handle ETTR and general highlight preservation exposure.
@RawConvert you can use your JPG as a sidecar the info is in thereā¦try itā¦load your jpg as a sidecar file back on to your raw and it will confer the settingsā¦the info is stored in the meta data. of the jpg although I think this can be turned on and off in the preferences so checkā¦
1 Like
Not only it can be turned off in the preferences, but also the JPG header has a maximum allowed size. If the edit history doesnāt fit in there, it will be truncated (there is a comment about this in the manual).
Hi David,
Iām an archivist by training and profession and a photographer by hobby.
If you (or, heaven forfend, your estate) donated your photographs to my archives, I would request the raw files + sidecar files and the resulting derivative tiffs or jpegs (16bit tiff preferred).
Raw files would be normalized to TIFFs using normalization software built into our digital preservation system. This mitigates any future risk of raw format obsolescence.
Of course, the normalized copies would not reflect your edits, so your workflow and end results would be demonstrated via the sidecar file and any derivative/final image you created.
If you donāt provide copies of the finished images and youāre a photographer of importance I might install your raw editor and create access copies that reflect your artistic vision. This is also dependent on collection size and resource availability.
To sum up: Archives want the original raw files (negatives), the side car (details of editing), and as high a resolution and bit depth copy of your finished work as possible. File formats should be open, in common use, and well-documented.
Thanks,
M.
3 Likes
Yes but I have had edits with 30 to 40 history entries and they are all fine but you are correct in what has been statedā¦.it would be nice to be able to rely on it and perhaps to have some way to indicate that the history was saved. I guess you could always test it with one of your most complicated edits and see if you can break itā¦.
One other thing I noticed is that the darktable sidecar file is very clear about what software was used to create it. Rawtherapee pp3 files, on the other hand, give no indication what software was used to create them except for a cryptic āModel=RawTā.
This makes it more likely that the darktable sidecar files will remain more useful longer, so long as the software can still be found and hasnāt deprecated many of the settings recorded in the XMP file.
Archivists have to operate on the timescale of years, decades, even centuries, so this stuff matters.
M.