Feedback wanted about Rapid Photo Downloader

[damonlynch]

Hello, I’m the developer of Rapid Photo Downloader. If you use the program, or have used it in the past, I’d like to know what points of pain you have when using it, meaning things that annoy you or you wish were handled better.

Let me give an example: I use Rapid Photo Downloader to download photos from my DSLR, but it would be handy to download photos and videos from my smartphone too. The thing is, I’d like to download them to a different place than from my regular camera, with a different naming convention. It’s a real pain in the butt to have to reconfigure the naming and destination program preferences every time I do this.

Now is a good time for feedback. I’m currently working on the next version. The next version will use a completely different way of storing program preferences, so it’s a good opportunity to make changes that are different from what exists today.

The next version already does some things better than the current release, although much of the user interface work is still to be done. Some screenshots from the most recent code:

This shows a mix files that have been downloaded a previous time and ones that have never been downloaded. Previously downloaded photos are dimmed. The third photo has an audio file associated with it (on some cameras you can record a little audio note - very handy).

This shows files from a camera with dual memory cards. The emblems indicate which card the files were found on. In this example, I set my camera to record photos to both cards simultaneously. I then deleted the third photo, which deletes it from the primary memory card, but not he secondary (also very handy at times!). The movie file was saved only to memory card 2.

For more detail on what has changed since the last release, I’ve copied and pasted from the ChangeLog:

New features:

• Refreshed user interface, with bigger thumbnails and easier visual
  identification of different file types.
• Download from all cameras supported by gphoto2, including smartphones:
  • You can download from multiple cameras/smartphones simultaneously.
  • Please note Rapid Photo Downloader will automatically unmount the camera/
    smartphone so it can gain exclusive access to it using gphoto2. If you
    attempt to mount it in another application (e.g. Gnome Files) while Rapid
    Photo Downloader is accessing it, the download could be interrupted.
  • Please also note that the phone should be unlocked before running Rapid
    Photo Downloader, or else it will be inaccessible.
• Remember files that have already been downloaded. You can still select
  previously downloaded files to download again, but they are unchecked by
  default, and their thumbnails are dimmed so you can differentiate them from
  files that are yet to be downloaded.
• Cache thumbnails generated when a device is scanned, making thumbnail
  generation quicker on subsequent scans.
• Cache photos and videos on cameras and phones used to generate thumbnails,
  speeding up their download if they are downloaded in the same session.
• Change the order in which thumbnails are generated, so that
  representative samples (based on time) are prioritized. This is helpful when
  downloading thousands of files at a time or over slow USB 2.0 connections.
• Generate freedesktop.org thumbnails for downloaded files, which means
  RAW files will have thumbnails in programs like Gnome Files and KDE Dolphin.
• Added progress bar when running under a Unity desktop

Removed feature:

• Rotate Jpeg images - to apply Lossless rotation, this feature requires the
  program jpegtran. Some users reported jpegtran corrupted their jpegs’
  metadata. To preserve file integrity, unfortunately the rotate jpeg option
  must be removed.

Switch to:

• PyQt 5.4, from PyGtk 2
• Python-gphoto2 to download from cameras, from Gnome GIO
• Python 3.4, from Python 2.7
• ZeroMQ for interprocess messaging, from python multi-processing
• GExiv2 for photo metadata, from pyexiv2
• Exiftool for video metadata, from kaa-metadata and hachoir-metadata

Missing features, still to be developed:

• Program configuration (preferences) cannot yet adjusted though the Graphical
  User Interface (GUI). However they can be adjust manually using a text
  editor.
• Clicking the eject icon by a device currently does nothing.
• There is no error log window.
• The GUI will not prompt for a Job Code before downloading.
• Most menu items do nothing.

Thanks in advance for any feedback.

Best,
Damon

[chris]

Hi Damon,

I must admit that I did not use Rapid Photo Downloader (RPD) very extensively, but I tested it a couple of months (years already?) ago and watched the project from time to time. Since you are so kind to ask users for their feedback, I think I can add my 0,02 €.

The reason that I did not use the tool beyond basic testing is that it did not adapt very well to my workflow, what may have changed due to your list of features. I have a workflow that uses directories in the style YYYY[-MM[-DD]]-Title_of_the_event (MM and DD optional) and do not alter file names. Videos of the event go to the same directory. When I tested RPD the last time there was no benefit compared to using a file manager, so I did not stick to it. But especially for the “download only new files” feature that may change soon .

So, first the list of suggestions especially for my workflow, and a question later:

• It would be nice to have “number of selected pictures/number of all pictures” in the status bar. Seems you already have this in the dev version.
• How about grouping images and adding job codes to these groups in the overview already, so it would be easier to chop the whole lot of images into the jobs they belong to and download with one click.
• It would be great to have a larger preview of single images on click/keypress (maybe full window or even 1:1 of the embedded thumbnail). Sometimes it’s hard to figure out which event a picture belongs to from the small thumbnails.
• Another little feature would be to always display/preview what will happen when I start the file transfer, e.g. by showing an example transfer (from directory + filename → to directory + filename). An option could be added to disable this feature once the user is more familiar with the tool.
• (related to the last item) The preconfiguration should never allow to overwrite a file and a big warning sign should be presented if that would happen.
• Selecting and deselecting images in groups (e.g. click, shift+click for group selection) should be as easy and as consistent as possible. An example: Current status is: If I want to select multiple images, I click the first image, shift+click the last image, and then hit one of the little check marks. You need to be very precise to hit the check mark, especially on my 12" full-HD notebook without external mouse this is very tricky. Improvement: Every normal click toggles the selection state, but shift-click still does the right thing (toggling all images except the one selected first, since this was already toggled. Furthermore, ctrl+click should not act the same as shift+click. It should allow to add single images to the selection in the current version and in my proposed change it should just toggle the image.
• A checksum generation per job would be handy (add missing checksums of the new files to md5sum.txt, hash algorithm and file name configurable of course, maybe checksum per file possible as well).

The last point brings me to my question, because I thought a lot about the problem as well (now the question): How do you recognize files already downloaded? Do you store infos on the card? What information is it based on, file name + creation date? DNG files can have checksums in their metadata which might be handy, but I guess other files don’t have this feature?

So, thanks for providing this great tool (even if it did not adapt to my workflow yet, I guess it makes a lot of people extremely happy, and the upcoming version looks very promising) and thanks for asking for suggestions, I hope I did not waste too much of your time.

Best regards

Chris

[damonlynch]

Hi Chris,

Thank you very much for your detailed feedback. This kind of feedback is precisely what I’m looking for. A few points:

1. Since very first version, 0.0.1, it will never overwrite files in the download destination - it will either fail to download it, or if the user specifies, add a suffix to the filename of the file being downloaded. This latter option is a double-edged sword. It probably needs to be there, but it would be preferable if users used sequence numbers, so filenames would never conflict.
2. There is the option to overwrite backups – if the user specifies that option – but I think I’ll remove that choice entirely now that the remembered files feature is there.
3. The existing release has a preview option. Double click on the thumbnail or click the small button immediately above the Help button. Probably this feature is too hard to discover.
4. In the existing release, you don’t have to use the mouse to check / uncheck files. The space bar will work too, including with multiple selections, as long as another button in the UI does not have focus.
5. Previewing the folder and file names before the download is trickier than it first seems. It requires reading the exif data, which is a relatively slow operation, because it’s computationally expensive. It’s even worse in the case of gphoto2, which does not allow the extraction of only the file exif/iptc/xmp metadata from files on the camera, so to get the metadata, the entire file must be downloaded from the camera over USB. Moreover, with the option to download from several devices simultaneously, sequence numbers/letters cannot necessarily be accurately guessed in advance. Having said all this, it’s possible to use the file system modification time as a proxy for the exif date/time, and to extract real exif data from representative samples, with the user being forced to wait until the folder and file name previews are generated in the background. The code to implement all this is not trivial.
6. In the development version, a file is judged unique by comparing file modification time (not exif), file size, and filename (not including the path). Sqlite is used to store successful downloads.
7. I’ve long wanted the option to group the display of files by date / time, but never set aside the time to implement it. Ideally I’d prefer it if there were simply the equivalent of a line break in the icon grid, but I have no idea if that’s easy to do in Qt without a lot of custom code. In any case, if there was that option, the option to assign job codes to files before hitting the download button would be a nice feature.
8. I’m not sure what you mean by “checksum generation per job”. Can you go into more detail?

Thanks again!

Best,
Damon

[Ivan_Tarozzi]

Hi Damon,
thanks for your great work.

As discussed by mail some time ago, I tried to compile dev version but I have some problem about dependencies. I have not so much time to investigate the problem, but meanwhile I want to reply to your call here.

My favourite new feature is “Remember files that have already been downloaded.”
I have a doubt about this:
Usually I download all pictures, then I open my DAM&RawDevelopment software (in my case darktable) and select or reject. Rejected pictures are deleted from filesystem.
I wish the next time I download from the same SD card RPD could retrieve last downloaded file and not download again deleted files. I’m sure you already have tought about it.

I think I’ll continue to download from SD card reader instead of direct camera because my Pentax K7 is not well supported by gPhoto2.

Thanks again,
best regards

Ivan

[chris]

@damonlynch

Hi Chris,

Thank you very much for your detailed feedback. This kind of feedback is precisely what I’m looking for. A few points:

Good to hear . Writing “feature requests” (and this is what my post essentially is, more or less) is always a bit tricky since some developers tend to be upset by such posts from users that “always want their features implemented but do not contribute back”. So again, thanks for asking!

1. Since very first version, 0.0.1, it will never overwrite files in the download destination - it will either fail to download it, or if the user specifies, add a suffix to the filename of the file being downloaded. This latter option is a double-edged sword. It probably needs to be there, but it would be preferable if users used sequence numbers, so filenames would never conflict.
2. There is the option to overwrite backups – if the user specifies that option – but I think I’ll remove that choice entirely now that the remembered files feature is there.
3. The existing release has a preview option. Double click on the thumbnail or click the small button immediately above the Help button. Probably this feature is too hard to discover.

Wow, next time I read the manual first, I promise . I guess you just gained a new user , it is something my file manager can’t do.

4. In the existing release, you don’t have to use the mouse to check / uncheck files. The space bar will work too, including with multiple selections, as long as another button in the UI does not have focus.
5. Previewing the folder and file names before the download is trickier than it first seems. It requires reading the exif data, which is a relatively slow operation, because it’s computationally expensive. It’s even worse in the case of gphoto2, which does not allow the extraction of only the file exif/iptc/xmp metadata from files on the camera, so to get the metadata, the entire file must be downloaded from the camera over USB. Moreover, with the option to download from several devices simultaneously, sequence numbers/letters cannot necessarily be accurately guessed in advance. Having said all this, it’s possible to use the file system modification time as a proxy for the exif date/time, and to extract real exif data from representative samples, with the user being forced to wait until the folder and file name previews are generated in the background. The code to implement all this is not trivial.

I thought on something more simple, e.g. to allow the user double-checking paths before transferring files. E.g. just an example based on the first file. In the current version, I see the origin and the base path the files will be transferred to, displaying an example subdirectory and an example file name, both based on the actual configuration, would already be enough. As in the mockup below:

6. In the development version, a file is judged unique by comparing file modification time (not exif), file size, and filename (not including the path). Sqlite is used to store successful downloads.

I guess this is reliable enough. Thanks for the info!

7. I’ve long wanted the option to group the display of files by date / time, but never set aside the time to implement it. Ideally I’d prefer it if there were simply the equivalent of a line break in the icon grid, but I have no idea if that’s easy to do in Qt without a lot of custom code. In any case, if there was that option, the option to assign job codes to files before hitting the download button would be a nice feature.
8. I’m not sure what you mean by “checksum generation per job”. Can you go into more detail?

The backup feature is a nice thing, but if the backup goes to a remote location (nfs share or manually copied or whatever), bit rot may leave you with different versions of the same file. To check the file integrity remotely in the backups and also locally I use checksum files in every directory. These are generated after downloading the images and always propagate with them to the backups etc. It would be great if this could be automated. At the moment I am running the following script “md5new” after downloading the pictures:

IFS="$(echo -e "\n\r")"
for EACHFILE in `ls -1 | grep -v md5sum.txt | grep -v ".xmp$"`
do
    if ! egrep -q  " $EACHFILE$" md5sum.txt; then
    md5sum $EACHFILE
    fi
done

with

md5new | tee -a md5sum.txt

This will give me a checksum file for all images and additional data but without the xmp files generated by darktable. Running this script automatically after every import in RPD (that means having a hook in RPD where I can add a script to be run after import) or, even better, additionally allow RPD to generate the md5 sums and save them with the images (in the format I explained in the last post), would be a great feature.

Since you are storing time stamp, size etc. of downloaded images in an sqlite database, additionally storing checksums in the DB (configurable, only per user request) after download would be extremely handy. Maybe in a db filed “checksum” that contains a string that can store more than one checksum, e.g. “md5:68b329da9893e34099c7d8ad5cb9c940 sha1:adc83b19e793491b1c6ea0fd8b46cd9f32e592fc sha256:01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b”. And as userinterface something like that:

With the options “per folder”, “per file” and “in database”. I guess that would allow for each and every workflow concerning checksums.

Thanks again for listening (reading) and for your software.

Best regards, Chris

[t3mujin (João Almeida)]

I’m a heavy user of Rapid Photo Downloader for the last couple of years. I have a couple of features I would really love:

• have separate counters per job name
• have a way to set default metadata for the imported photos (color label, default keywords, city, location), even if it means creating a sidecar file

[damonlynch]

Hi Chris,

I understand the concept of bitrot. Given I have about 2 TB of raw files, it’s something that concerns me too. However it’s outside the scope of Rapid Photo Downloader. Bitrot is a problem to be solved at the file system level, or with a specialized system like Jason DeRose’s Dmedia: Dmedia in Launchpad

Best,
Damon

[damonlynch]

Hi Ivan,

Don’t worry, in the next version you can always select files that are remembered as previously downloaded. It’s just they are not selected by default. Also, I don’t know if it’s clear, but Rapid Photo Downloader will not analyze if the downloaded file still exists, or it’s name changed, etc. All it will consider is if the file was downloaded previously or not.

Best,
Damon

[damonlynch]

Hi Joao,

I’m not sure about separate sequence values per job name. The GUI could become quite difficult. I’ll have to think about it.

Regarding your second point, you’re asking for a free software version of Photo Mechanic,which is a massive job. I don’t see myself doing that. Maybe someone else might.

Best,
Damon

[chris]

Hi Damon,

bit rod is not the only concern, files could be accidentally overwritten or there could be hardware defects. They even could be accidentally deleted. Checksum files are useful in many ways. The per file checksums (IMG_0815.CR2.md5) could also be used to verify file transfers over a network or provided along the original file for downloads etc. And, of course, it’s always good to know that the integrity of the files is intact.

File system based integrity checks as, e.g., ZFS implements do not help because there could still be errors while copying files to the backup locations. The checksum files provide evidence that everything is transferred well.

Of course, one could argue that copying with rsync would be safe, but still, having the checksum files is transparent through all copy processes and media and operating systems etc.

Anyway, if this is beyond RPDs scope, a hook to run a script after an import could still be handy for many use cases. If you have no idea what I mean, then have a look at the git hook scripts: Git - Git Hooks. Besides automatic checksum generation, use cases could be to automatically trigger a rescan of the base directory by the image management software, to trigger external backup solutions, to add files to git-annex or similar tools, to automatically generate contact sheets etc. I guess there would even be people nowadays that would love to automatically post to facebook that they have downloaded 748 images of the last holiday from the camera .

Even João Almeidas wish of automatic xmp generation could be solved with a little scripting and without altering RPD by using this hook script idea. It would allow a lot more flexibility for those users that do not fear a little bit bash/zsh/python/whatever scripting.

Best regards

Chris

[darix]

Just reading through thread and didnt see it mention yet: any thoughts on airnef?

Airnef for Desktop Wifi Transfer (Nikon)

DPReview forum moderataor Horshack (Adam on Testcams.com) has created a desktop application for wirelessly transferring photos from a wi-fi enabled Nikon camera to the computer (apparently something that Nikon doesn’t offer?). The project is called [airnef]. It’s a Python application that he has licensed GPL3 and is cross platform across Linux, OSX, and Windows. [airnef]: airnef - testcams.com [airnef - Wireless image and movie downloads for Nikon cameras!] It appears that it wi…

[t3mujin (João Almeida)]

I agree that multiple ounters can become cumbersome (I’m a software developer so I can imagine it in my mind), so take this suggestion really as a “nice to have”.

The second one does not need to be as complex as Photo Mechanic, imagine something as simple as this: an option in configuration to define a command that is executed when the import ends (with a few and well known substitution variables), so when an import finishes is easy to automate the execution of exiftool to set default keywords or labels, or even launch someone’s metadata editor or app to set geotagging, for example.

[damonlynch]

Given airnef has a command line interface, it probably wouldn’t be too difficult to incorporate it within Rapid Photo Downloader. After all, the two most basic steps are: (1) scan the camera to see what’s available and get the file name, size, modification time and thumbnail, and (2) copy the files to a local directory so that Rapid Photo Downloader can process them. Unless I’m mistaken, airnef command line version can do those steps already. I don’t own a camera with wifi, however, so it’s difficult to imagine me being the one to write and test the code. I’d be glad to assist someone who could do it, however.

[martijn]

Hi,

I’ve been thinking about photo import, and here’s the features I’ve come up with for my ideal application

1. it has the option to not show thumbnails. generally I know what I’ve shot, and usually thumbnails just slow you down.
2. it lets me group photos based on how much time elapsed between consecutive photos. This is useful for example when you photograph at a music festival, so you can separate photos from each performance.
3. it lets me name and tag each of the time-based groups
4. it can name the files while copying, and you can use the name from the previous bullet point in the naming template
5. it creates (darktable-compatible) xmp files with the tags
6. after copying, adds the raw files to git-annex and the xmp files to the corresponding git repository

[rht (Robert Thorsby)]

I would add #7. Provides for execution of another application after downloading.

[patdavid]

Just an FYI, I’ve moved this topic into the Software category, as it seemed a better fit there. (Nothing has changed on your end - just an FYI).

[damonlynch]

@martijn

lets me group photos based on how much time elapsed between consecutive photos

This will be in the next version. The next version’s interface is undergoing major changes, most of which are yet to be written. Having said that, what I call proximity groups will look something like this:

In this example, the proximity is 1 hour. It could be practically anything at all, which the user will be able to define themselves within reasonable limits.

My intention is that grouping the photos/videos like this will make it easier to assign job codes to them. Job codes will be able to be assigned to sets of files before the download button is clicked.

[Philippe]

Hello and thanks for your work. For my future use, when I’ll be familiar enough with darktable, I’m waiting for à feature of your FAQ: xmp import. I use rawvision on Android and i’d like to import stars and color labels (and sorry for m’y english…)

[damonlynch]

The current development code already imports and renames XMP files too, along with THM files. I didn’t realize some smart phone apps create XMP files. That’s interesting. I’m glad the code already works. Although one thing it doesn’t do is record the original file name in XMP, before renaming the file.

[anon48273813]

For me, the most wanted addition would be a tick box allowing me to delete contents on SD card once download has completed.

Keep up the awesome work Damon!