Advice needed: Pro photographer workflow on linux

[PitsLamp (Filip Bunkens)]

TL:DR:

• What is a good workflow and progs for photography on linux provided I have to handle thousands of images a year. Currently using LR (almost never passing in PS) on OS X
• What distro is?
  • distro package for design/photography or standard
  • What distro and desktop enviroment

---

So first let me quickly present myself before I start posing difficult questions.

I’m a freelance photographer based in Belgium and I have been using a mac for the last 10 years. My workflow is rather simple.

• Import in Lightroom in a working catalog
• Manage photos in LR
• Process in Lightroom (my photography requires seldom passing in PS)
• export to disk/dropbox in several resolutions
• archive photos every year in 3 catalogs:
  • Work
  • Personal
  • Film // Scanned

Now as my mbp is getting older, I need to switch to a new mbp or a pc. I definitely don’t want to use windows. I have quite a geek background and am definitely not adverse of diving in CLI.

---

My questions are the following:

1. Software for post-processing:
As I’m going the linux route, I figure there is not 1 program to replace LR. So I would need multiple that work nicely together.
Requirements:

• quick import, with direct backup to network drive or usb drive (depending location, office/road)

• after import photos are synced to program for quick selection

• raw edits are either in selection program or external editor, that syncs metadata and “flags” with organizing prog

• export to disk possible to jpg in multiple resolutions, from same source file

  • synced with previous to progs.
  • presets are a plus I have to do these exports on location throughout the day cfr publish collections lightroom
  • while export runs, other task should be possible on other photos.
  • after export if files are changed, would be great to have them highlighted cfr publish collection lightroom
  • social media exports are a plus

It would be great to have all the above programs be able to read and apply settings from xmp sidecar files from LR, so that if needed I can re-edit or export them again without redoing all the edits.

2. Linux Distro

I’ve been looking around already, trying out some distros:

• Antergos: Like that I have full control over everything that is installed and love uar cli installs, I like the rolling releases
• Ubuntu studio:
  - low latency kernel: Don’t know if I need the low latency and I experience randomn freezes
  - normal: seems quite ok. except for wireless problems with the mbp.
• Fedora design suite: going to test it out later today

so 3 questions related to that:

• Do I use a custom build or just a standard distro and adapt it. Wondering if the kernel of the pre configured distros have optimized kernels
• Rolling releases vs Release cycles, I need to be able to use my system almost constantly.
• Which Desktop environment for max compatability with above packages.

3. Hardware

I’ll make a seperate topic for that and link it here.
Edit: Hardware topic

[houz]

@PitsLamp

So first let me quickly present myself before I start posing difficult questions.

I’m a freelance photographer based in Belgium and I have been using a mac for the last 10 years. My workflow is rather simple.

Welcome! I hope you will find all the answers you need here.

My proposed solution will be based on darktable as that’s what I know and use. That doesn’t mean that other programs wouldn’t be capable of doing what you want. You can find links to all projects here.

Import in Lightroom in a working catalog
Manage photos in LR
Process in Lightroom (my photography requires seldom passing in PS)
export to disk/dropbox in several resolutions
archive photos every year in 3 catalogs:
Work
Personal
Film // Scanned

darktable doesn’t have an easy way to use several libraries/databases in parallel, but in general that isn’t required as all editing information is stored in XMP sidecars, so you can just archive the raw files + XMP and import them again when needed.

My questions are the following:

1. Software for post-processing:As I’m going the linux route, I figure there is not 1 program to replace LR. So I would need multiple that work nicely together.Requirements: - quick import, with direct backup to network drive or usb drive (depending location, office/road)

While darktable can import from your camera directly to disk I would still suggest to use Rapid Photo Downloader for that task. I suppose it can do backups on the fly, too.

after import photos are synced to program for quick selection

raw edits are either in selection program or external editor, that syncs metadata and “flags” with organizing prog

darktable is editing and organizing tool, so syncing isn’t needed.

export to disk possible to jpg in multiple resolutions, from same source file
synced with previous to progs.

By default it’s only possible to export a single resolution at a time. So you either have to start multiple export jobs or invest the time once to write a small Lua script (I bet the community would help with that) that exports your required set of image sizes at once.

presets are a plus I have to do these exports on location throughout the day cfr publish collections lightroom

darktable supports presets. Both for image editing operations and for export settings.

while export runs, other task should be possible on other photos.

Yes, that’s totally possible.

after export if files are changed, would be great to have them highlighted cfr publish collection lightroom

darktable assigns a tag to images when opened for editing and clears the tag when exporting. That might be enough for your use case.

social media exports are a plus

Currently darktable has exporters for Flickr, G+ and Facebook.

It would be great to have all the above programs be able to read and apply settings from xmp sidecar files from LR, so that if needed I can re-edit or export them again without redoing all the edits.

While darktable uses XMP sidecars too there are limitations wrt. taking over your settings from Lr. Things like ratings and tags/keywords are read, but editing steps are only supported rudimentary. So expect only crop, exposure, whitebalance and similar basic things to work.

2. Linux Distro

I’ve been looking around already, trying out some distros:

Antergos: Like that I have full control over everything that is installed and love uar cli installs, I like the rolling releases
Ubuntu studio:
low latency kernel: Don’t know if I need the low latency and I experience randomn freezes
normal: seems quite ok. except for wireless problems with the mbp.

Fedora design suite: going to test it out later today

This is a HIGHLY personal preference and everyone will try to convince you that only the distro they are using is worthwhile. That being said, for image editing there is no need to use RT kernels, self assembled systems or even self compiled kernels. Just use whatever feels right for you. A rolling release might be a bit more convenient as you get updates for your used programs all the time and there isn’t the big risk of “will the upgrade break my system”. It does however require a little more care and knowledge as every update has some potential to break things. Looking at the proposed changes from your packet manager should tell you that however, so as long as it doesn’t suggest “uninstall half the system” you should be fine.

I personally use Debian/sid since many years, but I am also aware that it’s not for everyone.

so 3 questions related to that:

Do I use a custom build or just a standard distro and adapt it. Wondering if the kernel of the pre configured distros have optimized kernels
Rolling releases vs Release cycles, I need to be able to use my system almost constantly.
Which Desktop environment for max compatability with above packages.

As said above: Standard distro, standard kernel, desktop environment shouldn’t matter. Using a GTK based program on KDE works great, same as QT applications cause no issues on GNOME. And the best thing: as the programs are not part of the desktop environment you are free to change your mind in a month or a year or whenever you want and switch DE while keeping your programs.

3. Hardware

I’ll make a seperate topic for that and link it here.

Good, as I know almost nothing about hardware.

[PitsLamp (Filip Bunkens)]

@houz

Welcome! I hope you will find all the answers you need here.

My proposed solution will be based on darktable as that’s what I know and use. That doesn’t mean that other programs wouldn’t be capable of doing what you want. You can find links to all projects here.

Thx, and wow what a reply.

@houz

While darktable can import from your camera directly to disk I would still suggest to use Rapid Photo Downloader for that task. I suppose it can do backups on the fly, too.

I used rapid photo downloader a couple of days ago, and had it completely crash on the backup to my network drive. But I saw that @paperdigits uses git-annex, which seems like a nice solution in combination with a script to have to copy over.

@houz

My proposed solution will be based on darktable as that’s what I know and use. That doesn’t mean that other programs wouldn’t be capable of doing what you want.

I’ve been playing around with darktable, digikam and rawtherapee already and I seem to prefer darktable for the moment. But definitely not yet fixed on any of the 3. A lot will be decided by the rest of the workflow.

@houz

By default it’s only possible to export a single resolution at a time. So you either have to start multiple export jobs or invest the time once to write a small Lua script (I bet the community would help with that) that exports your required set of image sizes at once.

That would indeed be a solution, Need to look at Lua scripting for automating other things too.

@houz

darktable assigns a tag to images when opened for editing and clears the tag when exporting. That might be enough for your use case.

Did not know that yet. That is indeed perfect, as I can filter on these tags, I suppose.

@houz

While darktable uses XMP sidecars too there are limitations wrt. taking over your settings from Lr. Things like ratings and tags/keywords are read, but editing steps are only supported rudimentary. So expect only crop, exposure, whitebalance and similar basic things to work.

That is not really good news. That means I have to find a good solution for non-destructive edits in my archive. I do off course have the jpg exports, but sometimes a client needs another resolution then those exports with the same edits. Going to figure something out.

@houz

That being said, for image editing there is no need to use RT kernels, self assembled systems or even self compiled kernels. Just use whatever feels right for you

Ok, that’s good news, it gives me a lot more options. ;-)[quote=“houz, post:2, topic:2014”]
A rolling release might be a bit more convenient as you get updates for your used programs all the time and there isn’t the big risk of “will the upgrade break my system”. It does however require a little more care and knowledge as every update has some potential to break things. Looking at the proposed changes from your packet manager should tell you that however, so as long as it doesn’t suggest “uninstall half the system” you should be fine.
[/quote]

Then I think I will go for a rolling release distro, because I’m very tech savvy and can figure out what I can install and what not.
Also I’m dabbling with linux for the last 16 years on and off but just not for desktop use. (server, IoT, old computers, home automation), so I do know a bit already about upgrades. ;-)[quote=“houz, post:2, topic:2014”]
Using a GTK based program on KDE works great, same as QT applications cause no issues on GNOME. And the best thing: as the programs are not part of the desktop environment you are free to change your mind in a month or a year or whenever you want and switch DE while keeping your programs.
[/quote]

Ok, that was something I wasn’t sure about, as some programs are clearly labelled k… or g… and I supposed they work better in those environments.

[chris]

@PitsLamp

I used rapid photo downloader a couple of days ago, and had it completely crash on the backup to my network drive. But I saw that @paperdigits uses git-annex, which seems like a nice solution in combination with a script to have to copy over.

There’s rapid development of rapid photo downloader currently happening, which, as judged by early alpha versions, will be a major overhaul and bring a lot of new features. Check this site for details, the author of RPD, @damonlynch, is active here. So I would keep it in mind and maybe wait for the final new version, using scripts or doing things manually in the file manager.

Git-annex is great, so this makes perfectly sense. That means, raws and output files handled by git-annex and sidecars directly by git. As it may help to repeat feature requests occasionally, a darktable integration with git and git-annex would be great ;-).

@PitsLamp

I do off course have the jpg exports, but sometimes a client needs another resolution then those exports with the same edits.

If it is just resizing, I would export all files as tiff in reasonable quality in LR and import them in DT’s photo management.

@PitsLamp

publish collections lightroom

I don’t know LR personally, but from what I have seen in videos etc. the collections concept of LR is something DT does not have directly, but you can use tags to recreate most of the functionality in DT.

[damonlynch]

@PitsLamp

I used rapid photo downloader a couple of days ago, and had it completely crash on the backup to my network drive.

Welcome to the free software world. Developers and your fellow community users appreciate bug reports or else the problem might not get fixed. FWIW coincidentally at this very moment I’m working on some unfinished elements of the backup code in the latest alpha.

[houz]

@chris

As it may help to repeat feature requests occasionally, a darktable integration with git and git-annex would be great

What exactly would be needed there? A button for “add selected images to git annex” which runs gti annex add foo.raw followed by a git push, a button “commit sidecar to git” which runs git commit -m "snapshot from <datetime>" foo.raw.xmp and then git push? Maybe even automatically add images in import to git annex if they are not there yet? Anything else?

PS: If this is becoming too off-topic we can move that elsewhere.

[PitsLamp (Filip Bunkens)]

@damonlynch

Welcome to the free software world. Developers and your fellow community users appreciate bug reports or else the problem might not get fixed. FWIW coincidentally at this very moment I’m working on some unfinished elements of the backup code in the latest alpha.

@damonlynch:

As I’m testing out different distros and different programs I hadn’t considered to file a bug report yet.

It was on a antergos system, with the uar install. I had a backup set over nfs and the import and backup went asynchronous, after about 100 files it stalled completely and had to be cancelled manually.

[damonlynch]

@PitsLamp

It was on a antergos system, with the uar install. I had a backup set over nfs and the import and backup went asynchronous, after about 100 files it stalled completely and had to be cancelled manually.

I don’t know what a uar install is, nor an antergos system. However if the version of Rapid Photo Downloader you were running is 0.4.x, don’ t bother reporting a bug because that code has been rewritten for the new 0.9.0 version, currently in alpha.

[PitsLamp (Filip Bunkens)]

@chris

There’s rapid development of rapid photo downloader currently happening, which, as judged by early alpha versions, will be a major overhaul and bring a lot of new features. Check this site for details, the author of RPD, @damonlynch, is active here. So I would keep it in mind and maybe wait for the final new version, using scripts or doing things manually in the file manager.

going to try again to use RPD this time under fedora, last time was antergos.

@chris

Git-annex is great, so this makes perfectly sense. That means, raws and output files handled by git-annex and sidecars directly by git. As it may help to repeat feature requests occasionally, a darktable integration with git and git-annex would be great .

I definitely have to checkout git for my photography workflow also. I’m using it for my personal scripting and my website development, but not yet as a sort of a backup solution for my photography.

@chris

If it is just resizing, I would export all files as tiff in reasonable quality in LR and import them in DT’s photo management.

That would be a solution, except that we are talking of litterally hunderds of thousands of files.

@chris

I don’t know LR personally, but from what I have seen in videos etc. the collections concept of LR is something DT does not have directly, but you can use tags to recreate most of the functionality in DT.

going to check out if that is feasible.

[PitsLamp (Filip Bunkens)]

@damonlynch

I don’t know what a uar install is, nor an antergos system. However if the version of Rapid Photo Downloader you were running is 0.4.x, don’ t bother reporting a bug because that code has been rewritten for the new 0.9.0 version, currently in alpha.

Antergos is arch linux install and aur is the package manager of arch linux.

It was indeed version 0.4.x so I’m going to try out 0.9 I don’t mind beta/alpha version as long as it is not on my workstation, as I’m testing a new workflow this should be ok.

[chris]

@houz

What exactly would be needed there? A button for “add selected images to git annex” which runs gti annex add foo.raw followed by a git push, a button “commit sidecar to git” which runs git commit -m “snapshot from <datetime>” foo.raw.xmp and then git push? Maybe even automatically add images in import to git annex if they are not there yet? Anything else?

Now looking at it written down it does not sound too complicated. Plus maybe automatic sidecar snapshots whenever darkroom is left and something changed, or whenever in lighttable a threshold of tags/ratings/styles have changed, or whenever dt is left. Plus a button “save film roll state to git” with text field beside which generates a commit with a meaningful comment of all changes and added files of the current film roll.

A more elaborate system could deal with the fact that in git-annex it may be necessary to transfer files to the local file system before their content becomes available (requires to run a git-annex command), and this may require even adding an external file system, e.g. an archive disc (see https://git-annex.branchable.com/location_tracking/). It would be good if dt would not get confused by this fact and provide tools to deal with it to the user. E.g., if a raw file is opened in lighttable which is not available on the local file system, git annex get file could be automatically run, and if the file system has to be made available, a dialog could appear with this information, but allowing to cancel the operation at all without messing up DTs database or the xmp.

@houz

PS: If this is becoming too off-topic we can move that elsewhere.

Agreed.

[damonlynch]

Cool. I suggest turning off back ups in alpha 4. When alpha 5 is released the underlying backup functionality will be complete.

[damonlynch]

The code I added tonight to Rapid Photo Downloader’s backup feature was the generation of thumbnails for file managers like Nautilus of RAWs and TIFFs. For example in this screenshot the drive “stuff2” is an external drive I use for testing backups (and “raw samples” happens to be a large collection of many types of RAW that I used for testing):

[paperdigits (Mica)]

I do use git-annex and I think it is fantastic!

I use it from the CLI, I guess GUI integration would be nice, but I don’t see it as necessary.

Currently I check all raw files into git-annex, then sync out to all my backup disks. I’m evaluating checking xmp files into git (not git-annex) and it seems reasonable.

[PitsLamp (Filip Bunkens)]

Would you mind describing how you use git-annex with your photos. I would really appreciate it.

[paperdigits (Mica)]

My workflow is approximately this:

I have six external hard drives that hold my git-annex repos. Three of them are unencrypted and stay in my apartment, the other three are gpg encrypted and get rotated off site.

1. Mount camera medium on computer
2. Change directory into the camera medium directory
3. Run exiftool script that renames files by their creation date and copies them into a date stamped folder in my git-annex raw files repo
4. Change directory into the git-annex raw files repo
5. Run git-annex add . to add the files into the annex
6. Run git commit -m "some message" to commit the changes to my git annex repo.
7. Plug in other git annex external drives and mount them.
8. Run git-annex sync --content to sync up content to all the connected drives.

This is used when I am importing. I usually import then sync to at least one other drive for redundancy. I’ll then start darktable or rawtherapee, both of which are pointed at the git annex raw file repo.

[PitsLamp (Filip Bunkens)]

That’s a great system.

I just have to script this that it does most of it automatically, otherwise this will be too time consuming in my case. But I do like the fact that the files are in a git system. The great thing is that I don’t have to specify where I’m backing up when I’m on the road. definitely going to check out how to set all this up.

Thx

[PitsLamp (Filip Bunkens)]

@damonlynch: sadly enough the alpha is really not working for me. too much is still missing. like the tab for renaming the files.

I’ll file a bug report for the things I found that are working but behave not like expected.

Also I submitted a feature request to add an extra option next to job name. Client name/user inputted text 2, would be good as well. This is because of my file structure. more info in the feature request.

[luxagraf]

@paperdigits Your setup sounds similar to mine except that I just rsync to the external drives (and remote servers). I’m familiar with git and git-annex (my photos are about the only thing I don’t keep in a git repo), but I’m curious what git-annex gets you in this case… can you give a little detail on the advantages of git-annex over rsync or why you ended up at this workflow? (Do you make commits with every photo edit? Call a hook from Darktable or something? Now that would be cool. Hmm).

[paperdigits (Mica)]

Of course!

@luxagraf

why you ended up at this workflow?

For many of the reasons listed below, but mostly I wanted to be able to take a drive full of my encrypted data off-site, not worry about adding data, then rotate the off-site data back to my apartment, updated it, then rotate it out again. I was trying to follow the 3-2-1 backup strategy at first, but once I got going with git-annex, I realized that I could just add drives indiscriminately to my backup/parity routine… so I did.

I still don’t have them on different medium and haven’t found a good way to do such a thing; my current thinking is just making multi-part tar.7z files, then burning them to DVD. Does SSD count as a different medium from HDD? I somehow doubt it

@luxagraf

Do you make commits with every photo edit?

With git-annex, I’m only using it for my RAW files, so they only get committed once upon import and git-annex makes them read-only by default. Read-only RAW files has been working well for me, I can’t accidentally delete them or otherwise mangle the file and RAW files really only need to be read anyway (not write or execute). Since git-annex can be used in tandem with regular git, I’m currently evaluating committing my metadata sidecar files (.xmp and .pp3) into git (not git-annex). That looks like it’ll work well too, but I’m super conservative when changing my workflow. I’m not sure if’d I’d call a hook from my editor or just do it on the CLI. Darktable does have some nice lua scripting capability.

Git annex also supports metadata and can show and hide files based on that metadata.

---

There are several advantages of git-annex over an rsync solution.

• Git annex repos are aware of one another, git annex whereis file.ext returns a list of repos that contain the file. I can use annex.numcopies to specify a minimum redundancy number for my files. Currently that is set at 3, so there are at least 3 copies of my files. I don’t have to manage that manually, it is done for me. On the flip side of that, when one of my drives start to get full, I can git annex drop file.ext (which considers annex.numcopies) then it removes the file from the repo, freeing space on the disk.

• I can add data to the repo on any disk. They’re all configured to talk to one another. While I do have a “master” disk, that is just a naming convention, git annex is just as distributed as git itself.

• File hashing comes free. Rsync means your file got there intact, but rsync doesn’t safeguard against bit rot. You can git annex fsck -q and it’ll hash all the files and tell you which files don’t match their hash. If a file comes back bad, you can git annex get file.ext and pull a known good copy of that file (or replace the disk or whatever you need to do).

• PGP encryption is cheap and pretty easy to set up. You can then PGP encrypt files or the whole repo using git gcrypt. I use this for my offsite backups.

• Multiple cloud storage systems are supported (Amazon S3 & Glacier, Rackspace, etc, etc). If cloud providers are not your thing, you can set up git annex over ssh on your own server and use it that way. If your personal host doesn’t support git annex, you can use git annex to push just the files.

• Integrates with git hosting solutions; I’m using it with gitolite and like it, and the enterprise version of gitlab supports git annex.

• Even if git-annex disappears from the face of the planet, all your files are still there on the filesystem in .git/annnex/objects named by their file hash.

• Git annex is flexible and I’m confident I can get what I need out of it now and in the future