Rapid Photo Downloader AppImage for testing

appimage

(Carmelo Dr Raw) #1

I have started working on an AppImage package for rapid photo downloader.

Things are not yet perfect, for example the fonts look ugly on my system, however it would be nice if people could already start testing the package on as many Linux systems as possible, and report issues.

The usage of the appimage is simple:

  • get the package from here
  • make the file executable with
    chmod u+x rapid-photo-downloader-0.9.0b6.glibc2.14-x86_64.AppImage
  • execute the appimage:
    ./rapid-photo-downloader-0.9.0b6.glibc2.14-x86_64.AppImage

Thanks in advance!


(Damon Lynch) #2

First of all, thank you very much indeed for your assistance! A few notes:

  1. As you probably noticed, there are no application icons. Perhaps that has something to do with the following XKB problem?
    xkbcommon: ERROR: failed to add default include path ././/share/X11/xkb Qt: Failed to create XKB context! Use QT_XKB_CONFIG_ROOT environmental variable to provide an additional search path, add ':' as separator to provide several search paths and/or make sure that XKB configuration data directory contains recent enough contents, to update please see http://cgit.freedesktop.org/xkeyboard-config/ .
  2. There is a problem with the GI gstreamer bindings (they appear to be missing), which means thumbnails will not be generated. Perhaps the gstreamer library is not installed, or the GI bindings are not finding it?
  3. I strongly suggest ignoring the PyQt / Qt that comes with Debian and using PyQt5 from Pypi, which comes with its own copy of Qt. It should make life easier, given the PyQt developer maintains it and has designed it to be portable across different distributions and platforms.
  4. Python ZeroMQ is too new. Version 15.1.0 is good.
  5. libmediainfo is missing / unable to be located.
  6. Was python built with openssl support? The python requests module was unable to check for a newer version, perhaps because it’s making an HTTPS request. I have some notes on building Python with openssl if that would help.

Thanks again.


#3

Thanks for making an AppImage. However, ExifTool is currently missing from the AppImage.

ExifTool missing

Please bundle it. An AppImage is supposed to bundle everything that cannot be expected to be part of every target system (distribution default installation) in a recent enough version. This is definitely true for ExifTool.


(Carmelo Dr Raw) #4

@probonopd @damonlynch

Updated package available here

Done.

One of the points I need to dig into…[quote=“damonlynch, post:2, topic:4474”]
There is a problem with the GI gstreamer bindings (they appear to be missing), which means thumbnails will not be generated. Perhaps the gstreamer library is not installed, or the GI bindings are not finding it?
[/quote]

I am now working to also include gstreamer in the bundle[quote=“damonlynch, post:2, topic:4474”]
I strongly suggest ignoring the PyQt / Qt that comes with Debian and using PyQt5 from Pypi, which comes with its own copy of Qt. It should make life easier, given the PyQt developer maintains it and has designed it to be portable across different distributions and platforms.
[/quote]

Both QT and PyQT are compiled from sources

Downgraded to 15.1.0

I have tried to find the source package for libmediainfo, but got confused by the contents of the package… it does not look like a standard source package with a configure/cmake build system. Any idea?[quote=“damonlynch, post:2, topic:4474”]
Was python built with openssl support? The python requests module was unable to check for a newer version, perhaps because it’s making an HTTPS request. I have some notes on building Python with openssl if that would help.
[/quote]

Yes, it was… anyway, the appimage bundle is read-only, so upgrading any of the bundled packages is not an option.


#5

Check https://github.com/MediaArea/MediaInfoLib/blob/master/.travis.yml to see how they build it.


(Carmelo Dr Raw) #6

Thanks! I could figure out the proper way to build the code from the travis configuration.

Here is a new version with Exiftool, libmediainfo and GStreamer included, plus some fixes to the environment variables.

Could you check if it works as expected? One issue I’m aware of is the incorrect UI font…

The full set of scripts and configuration files needed to build the AppImage is now available on GitHub: https://github.com/aferrero2707/rapid-photo-downloader-appimage


(Sebastien Guyader) #7

I tried the latest appimage on Manjaro XFCE build. I had to install libselinux, as it was needed for the application to start (I guess it should be included in the next versions of the appimage).
I have one problem though, the app doesn’t detect neither my camera (Fujifilm X-T2) nor my cell phone (Alcatel Idol 4). In devices, in only see “vc”:

I have gphoto2 installed.


(Carmelo Dr Raw) #8

gphoto2 is bundled in the appimage… maybe @damonlynch has some ideas about what could be missing and thus preventing the camera detection? I have no clue…


(Damon Lynch) #9

Is the AppImage sandboxed in any way? if so, it most definitely should not be.

From the log files / debug output, the application is trying the scan the “xHCI Host Controller”, which strikes me as rather odd. I’ve never seen that before. “VC” is the name assigned by udev. Apparently gphto2 thinks this device is a camera. In fact, the log file says “Detected Nikon DSC D750 on port usb:001,001”, which is also strange, because no such camera is attached.

The icons are still not being loaded. That’s a problem because the program’s various menus are accessible only via their icons.

Am still getting the following output:

This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".

Available platform plugins are: minimal, offscreen, xcb, eglfs, linuxfb, minimalegl, vnc.

Reinstalling the application may fix this problem.

Finally, have you considered the PyQt5 from Pypi? The Qt bundled with that is ‘portable’ in the sense that it can be installer without sudo permission in a user’s .local directory (that package is a python wheel). That might help with the font / icon problem. PyQt 5.9 & Qt 5.9.1 are due very soon. Qt 5.9.1 looks to be a good stable release.


(Carmelo Dr Raw) #10

One more version, hopefully fixing the camera detection problem: https://filebin.net/du3zsx1tcekv4pka

EDIT: still the same problem with devices/cameras detection…

@damonlynch I have few questions:

  • which Linux system are you using for testing?
  • do you have suggestions for debugging the devices/cameras detection problem?
  • where can I find the missing icons? Are they being installed in .png/.svg format?

Thanks!


(Damon Lynch) #11

I’ve run it on a variety of systems; the earliest is Ubuntu 14.04 in Virtualbox. FWIW my development machine is Ubuntu 17.10. They all exhibit the same problems.

If you run the program with the following command line options, there is plenty of debugging output:

./rapid-photo-downloader-0.9.0b6.glibc2.14-x86_64.AppImage --log-gphoto2 --debug

However, even with that voluminous output I don’t know what the problem is, simply because I lack the experience to understand what’s going wrong. I’ll ask on the gphoto2 mailing list if anyone has successfully made an AppImage that uses libgphoto2.

There are no external icons to load. They’re embedded in the code – that’s how it works with Qt. There is something wrong with the Qt setup you’ve got in the AppImage. I’ve seen the problem of missing Qt icons certain smaller Linux distributions.


(Carmelo Dr Raw) #12

I found the reason for the broken device detection, which was simply due to the fact that I had no libusb development packages installed, and therefore libgphoto2 was built without USB support (this is the discussion that led me to the solution).

Here you can find the updated AppImage package that should properly detect the devices.

On the other hand, icons are still missing, and I am wondering if this is a problem with the source package of rapid photo downloader.
Looking at the QT resources, I see things like this:

./raphodo/resources.qrc:15:<file alias="report-error.png">../images/report-error.png</file>
./raphodo/errorlog.py:244:            ErrorType.serious_error: ':/report-error.png',

However, when un-packing https://launchpad.net/rapid/pyqt/0.9.0b7/+download/rapid-photo-downloader-0.9.0b7.tar.gz I do not find any images folder… should I get the images from some other package?


(Damon Lynch) #13

libgphoto2 is working now, thank you.

Some other points:

  • libraw is either missing or is the wrong version
  • gstreamer is not working:
    pipeline = Gst.parse_launch('playbin') GLib.GError: gst_parse_error: no element "playbin" (1)
  • pyzmq should be version 15.1.0. Versions higher than this cause the program to not exit sometimes.
  • All images are located in the pre-generated file qrc_resources.py. There’s a reason that file is 16,400 lines long :wink: This is how images are loaded in Qt / PyQt. I seem to recall Gtk+ has or is moving to a similar system to Qt these days too, for performance reasons. In any case, as I’ve been saying, something is wrong with the Qt environment.
  • All is not well with GIO. The code to unmount devices is not working. Is this a permissions issue? Missing the ability to unmount / eject devices wrecks usability with cameras and smartphones, because they’re totally inaccessible to Rapid Photo Downloader while any other application is using them.
  • The code to check a new version is not working - is network access blocked by default, or is openssl missing? I don’t see python being built with openssl support in build.sh. I’ve coded the application such that it will only offer to download a new version if it was installed using python’s pip. If not, it merely alerts the user that a new version is available.

(Carmelo Dr Raw) #14

Here comes the next version. I realized that my Glib was compiled without libmount support, which should explain why devices could not be un-mounted. I also compiled the missing gstreamer plugins, so this should be fixed as well.

I’m already using version 15.1.0 for pyzmq, or at least that is what should be bundled… tomorrow I will re-build everything from scratch, just to make sure I have nothing left from previous builds.

LibRAW is at version 0.18.2, which is quite new…

The main problem remains with the icons. I do have this file installed in the prefix:

-rw-r--r-- 1 photoflow photoflow 11M May 10 06:00 /yyy/lib/python3.5/site-packages/raphodo/qrc_resources.py

and there is also a compiled file here:

$ grep -r -n -e qrc_resources.py /yyy/
Binary file /yyy/lib/python3.5/site-packages/raphodo/__pycache__/qrc_resources.cpython-35.pyc matches
/yyy/lib/python3.5/site-packages/rapid_photo_downloader-0.9.0b6-py3.5.egg-info/SOURCES.txt:142:raphodo/qrc_resources.py
/yyy/_jhbuild/manifests/rpd:28:lib/python3.5/site-packages/raphodo/qrc_resources.py

Any idea of what could still be missing?

Thanks!


(Damon Lynch) #15

libraw should be 0.17.2, because its python binding rawkit is (incredibly) not yet updated to work with 0.18. I say incredibly because one of our forum members generously contributed the code for the project developers to make it work with libraw 0.18, but they haven’t bothered to make a release. Quite disappointing really.

Nothing is missing from the Rapid Photo Downloader code. If it were, there would be a python exception. The problem is with the Qt environment you’ve got in there. Sorry to say the same thing over and over, but seriously I think you really need to dump the custom compile of Qt 5.6 and stick with the Qt 5.8 supplied by the creator of PyQt itself. It’s so simple to install using pip, just like is already done with all the other python dependencies. That’s the approach taken in the example Python Appimage application. If there is still a problem with the Qt icons after installing Phil Thompson’s PyQt & Qt bundle then I can ask on the PyQt mailing list what’s going on.

Unfortunately, the version of gstreamer included has a bug that means it won’t generate the thumbnails for videos. That bug is present in versions of gstreamer circa Ubuntu 16.04, but is fixed in newer releases.

The unmount still is not working in this latest release.

Many thanks for your efforts.


(Carmelo Dr Raw) #16

Hi! One more version for testing here.

I downgraded LibRAW to 0.17.2, and installed several python packages (including PyQT5) using pip. This seems to at least allow to fix the “missing icons” problem. Thumbnails seem to be also generated correctly. On the other hand, I am not sure about the camera un-mounting… I have the impression it still does not work, but I’d like to get a confirmation. I have really no clue how to debug this specific problem…


(Damon Lynch) #17

Well it sure looks much more aesthetically pleasing now! Good job with the new Qt version! :thumbsup:

There is a tricky situation with libraw: currently, it’s not actually being used from within the AppImage. Here is why. rawkit uses ctypes, and as discussed at that previous link, when python’s ctypes goes looking for the library file, it’s looking everywhere but inside the AppImage! The solution is to hard code the filename, instead of using ctypes.util.findlibrary(). This is the point at which the rawkit code needs to be changed: https://github.com/photoshell/rawkit/blob/ede197ae5235c022eb100ce7de2b921cbbbc993c/libraw/bindings.py#L31. I haven’t seen what the name of the libraw library is in your AppImage, but that line of code should be changed to something like libraw = 'libraw.so.16'. The question is, how do we do that? Do we have to fork rawkit? Or is it easy for the AppImage build script to download the rawkit source, patch it, and then include it?

There is another problem too: it seems the AppImage is actually not self-contained in the sense that it can run code from outside itself. For example, when I run the AppImage on my main development machine it’s actually running newer Python code from my home directory. It can do that because my development code is on the python module search path. I don’t think the AppImage should be able to do that, personally. Is that something that can be changed? It might just be a matter of setting the environment variable PYTHONPATH to a default value? (As long as that doesn’t affect the host system!).

I stated earlier that the version of gstreamer bundled with the AppImage has a bug that means that it is unable to generate thumbnails for any video files. However I was perhaps wrong about that. The problem could be with PyGObject, not gstreamer. I realized it only now when adding additional version reporting to Rapid Photo Downloader when it starts up. Here are some combinations that do and do not work with respect to extracting video thumbnails:

  • Ubuntu 16.04: Gstreamer 1.8.3 PyGObject 3.20.0 success
  • Ubuntu 16.10: Gstreamer 1.8.3 PyGObject 3.22.0. failure
  • Ubuntu 17.04: Gstreamer: 1.10.4 PyGObject: 3.22.0 failure
  • Ubuntu 17.10: Gstreamer: 1.12.1 PyGObject: 3.24.1 success
  • Fedora 24: Gstreamer: 1.8.3 PyGObject: 3.20.1 failure
  • Fedora 25: Gstreamer: 1.10.4 PyGObject: 3.22.0 failure
  • Korora 25 (Fedora derivative) Gstreamer: 1.10.4 PyGObject: 3.22.0 success
  • LinuxMint 18.0 Gstreamer: 1.8.3 PyGObject: 3.20.0 failure
  • LinuxMint 18.1: Gstreamer: 1.8.3 PyGObject: 3.20.0 failure
  • openSUSE 42.2: Gstreamer: 1.8.3 PyGObject: 3.20.1 failure
  • openSUSE Tumbleweed: Gstreamer: 1.12.0 PyGObject: 3.24.1 failure
  • Debian 9: Gstreamer: 1.10.4 PyGObject: 3.22.0 success

As you can see, some combinations that work in one distro do not work in another. It’s difficult to understand how this makes sense, but perhaps some distros are applying a patch and others are not?

Regarding the unmount problem, a related problem is that GIO volume monitor is not always being informed that a new device has been mounted or unmounted either. To me that indicates something is not right with the GIO install in the AppImage. I’ve turned on some verbose debug messages relating to GIO that I had turned off some time ago, because they’re often too verbose. But they might help in this case.


(Carmelo Dr Raw) #18

If rawkit is installed from sources, and not via pip, then it is realtively easy to automatically patch the code. This is another advantage of using jhbuild, as it supports patching as part of the build process.

Here is where the libraw.so library is located inside the appimage:

This is probably the knot of the rope, and the reason why some modules are failing. For example, the GStreamer/PyGObject combination that is bundled is 1.12.1/3.24.1, which from your table seems to work well in Ubuntu 17.10. Could you tell me which versions are reported on your system, and how to enable the verbose version reporting?[quote=“damonlynch, post:17, topic:4474”]
I’ve turned on some verbose debug messages relating to GIO that I had turned off some time ago, because they’re often too verbose. But they might help in this case.
[/quote]

How do I enable the verbose GIO messages?

I am now re-building the AppImage with a patched rawkit, I will post the new package as soon as available.


(Damon Lynch) #19

Those using snap are facing exactly the same problem. Here is a proposed solution for the Ubuntu tool snapcraft, which is not shell based but python based: http://paste.ubuntu.com/24312683/
While the python part where it patches the code is probably not relevant, but perhaps the environment variable part is. In any case, this part is foggy to me because I don’t understand how by hardcoding the library name the program can actually locate it at runtime. My knowledge of Unix is a bit weak here.

To make those additional version checks and logging options, I’ve had to make some changes to the code since beta 7. If you don’t mind install bzr then the easiest way to make a tarball with them is this:
bzr branch lp:rapid cd rapid python3 setup.py sdist
The tarball will now be located in the dist directory. This is basically the 0.9.0 release, however I’m reluctant to release it as the official release because if there is something I can add to the code to make the AppImage easier, then I think it’s worth waiting for that.

To quickly confirm what versions of various libraries are being used, run:
rapid-photo-downloader --detailed-version

To see detailed logging output as you run:
rapid-photo-downloader --debug

Perhaps. However, we can’t rule out the problem is limited to when a user manually specifies their own directories using the env variable PYTHONPATH. It may not be picking up modules in ~/.local/lib/python3.5/site-packages

Just run with the --debug flag, or else you can always consult the log files in ~/.cache/rapid-photo-downloader/log (which have line numbers too).


(Carmelo Dr Raw) #20

Hi!

I’ve not yet debugged the GIO problem, but it seems I have at least fixed the rawkit library loading issue, with this patch.

The new AppImage package can be downloaded from here, and the updated set of build scripts can be found here. The new build script is called bootstrap2.sh.

This is what the --detailed-version option gives on my build system:

Rapid Photo Downloader: 0.9.0
Platform: Linux-3.16.0-38-generic-x86_64-with-debian-jessie-sid
Python: 3.5.3
Qt: 5.8.0
PyQt: 5.8.2
ZeroMQ: 4.1.2
Python ZeroMQ: 15.1.0 (cython backend)
gPhoto2: 2.5.14
Python gPhoto2: 1.7.0
ExifTool: 10.56
pymediainfo: 2.1.9
GExiv2: 0.10.04
Gstreamer: 1.12.1
PyGObject: 3.24.1
libraw: 0.17.2-Release
rawkit: 0.5.0
psutil: 5.2.2

Let’s hope that we are slowly converging…