I have started working on an AppImage package for rapid photo downloader.
Things are not yet perfect, for example the fonts look ugly on my system, however it would be nice if people could already start testing the package on as many Linux systems as possible, and report issues.
First of all, thank you very much indeed for your assistance! A few notes:
As you probably noticed, there are no application icons. Perhaps that has something to do with the following XKB problem? xkbcommon: ERROR: failed to add default include path ././/share/X11/xkb Qt: Failed to create XKB context! Use QT_XKB_CONFIG_ROOT environmental variable to provide an additional search path, add ':' as separator to provide several search paths and/or make sure that XKB configuration data directory contains recent enough contents, to update please see http://cgit.freedesktop.org/xkeyboard-config/ .
There is a problem with the GI gstreamer bindings (they appear to be missing), which means thumbnails will not be generated. Perhaps the gstreamer library is not installed, or the GI bindings are not finding it?
I strongly suggest ignoring the PyQt / Qt that comes with Debian and using PyQt5 from Pypi, which comes with its own copy of Qt. It should make life easier, given the PyQt developer maintains it and has designed it to be portable across different distributions and platforms.
Python ZeroMQ is too new. Version 15.1.0 is good.
libmediainfo is missing / unable to be located.
Was python built with openssl support? The python requests module was unable to check for a newer version, perhaps because itâs making an HTTPS request. I have some notes on building Python with openssl if that would help.
Thanks for making an AppImage. However, ExifTool is currently missing from the AppImage.
Please bundle it. An AppImage is supposed to bundle everything that cannot be expected to be part of every target system (distribution default installation) in a recent enough version. This is definitely true for ExifTool.
One of the points I need to dig intoâŠ[quote=âdamonlynch, post:2, topic:4474â]
There is a problem with the GI gstreamer bindings (they appear to be missing), which means thumbnails will not be generated. Perhaps the gstreamer library is not installed, or the GI bindings are not finding it?
[/quote]
I am now working to also include gstreamer in the bundle[quote=âdamonlynch, post:2, topic:4474â]
I strongly suggest ignoring the PyQt / Qt that comes with Debian and using PyQt5 from Pypi, which comes with its own copy of Qt. It should make life easier, given the PyQt developer maintains it and has designed it to be portable across different distributions and platforms.
[/quote]
Both QT and PyQT are compiled from sources
Downgraded to 15.1.0
I have tried to find the source package for libmediainfo, but got confused by the contents of the package⊠it does not look like a standard source package with a configure/cmake build system. Any idea?[quote=âdamonlynch, post:2, topic:4474â]
Was python built with openssl support? The python requests module was unable to check for a newer version, perhaps because itâs making an HTTPS request. I have some notes on building Python with openssl if that would help.
[/quote]
Yes, it was⊠anyway, the appimage bundle is read-only, so upgrading any of the bundled packages is not an option.
I tried the latest appimage on Manjaro XFCE build. I had to install libselinux, as it was needed for the application to start (I guess it should be included in the next versions of the appimage).
I have one problem though, the app doesnât detect neither my camera (Fujifilm X-T2) nor my cell phone (Alcatel Idol 4). In devices, in only see âvcâ:
gphoto2 is bundled in the appimage⊠maybe @damonlynch has some ideas about what could be missing and thus preventing the camera detection? I have no clueâŠ
Is the AppImage sandboxed in any way? if so, it most definitely should not be.
From the log files / debug output, the application is trying the scan the âxHCI Host Controllerâ, which strikes me as rather odd. Iâve never seen that before. âVCâ is the name assigned by udev. Apparently gphto2 thinks this device is a camera. In fact, the log file says âDetected Nikon DSC D750 on port usb:001,001â, which is also strange, because no such camera is attached.
The icons are still not being loaded. Thatâs a problem because the programâs various menus are accessible only via their icons.
Am still getting the following output:
This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".
Available platform plugins are: minimal, offscreen, xcb, eglfs, linuxfb, minimalegl, vnc.
Reinstalling the application may fix this problem.
Finally, have you considered the PyQt5 from Pypi? The Qt bundled with that is âportableâ in the sense that it can be installer without sudo permission in a userâs .local directory (that package is a python wheel). That might help with the font / icon problem. PyQt 5.9 & Qt 5.9.1 are due very soon. Qt 5.9.1 looks to be a good stable release.
Iâve run it on a variety of systems; the earliest is Ubuntu 14.04 in Virtualbox. FWIW my development machine is Ubuntu 17.10. They all exhibit the same problems.
If you run the program with the following command line options, there is plenty of debugging output:
However, even with that voluminous output I donât know what the problem is, simply because I lack the experience to understand whatâs going wrong. Iâll ask on the gphoto2 mailing list if anyone has successfully made an AppImage that uses libgphoto2.
There are no external icons to load. Theyâre embedded in the code â thatâs how it works with Qt. There is something wrong with the Qt setup youâve got in the AppImage. Iâve seen the problem of missing Qt icons certain smaller Linux distributions.
I found the reason for the broken device detection, which was simply due to the fact that I had no libusb development packages installed, and therefore libgphoto2 was built without USB support (this is the discussion that led me to the solution).
Here you can find the updated AppImage package that should properly detect the devices.
On the other hand, icons are still missing, and I am wondering if this is a problem with the source package of rapid photo downloader.
Looking at the QT resources, I see things like this:
However, when un-packing https://launchpad.net/rapid/pyqt/0.9.0b7/+download/rapid-photo-downloader-0.9.0b7.tar.gz I do not find any images folder⊠should I get the images from some other package?
gstreamer is not working: pipeline = Gst.parse_launch('playbin') GLib.GError: gst_parse_error: no element "playbin" (1)
pyzmq should be version 15.1.0. Versions higher than this cause the program to not exit sometimes.
All images are located in the pre-generated file qrc_resources.py. Thereâs a reason that file is 16,400 lines long This is how images are loaded in Qt / PyQt. I seem to recall Gtk+ has or is moving to a similar system to Qt these days too, for performance reasons. In any case, as Iâve been saying, something is wrong with the Qt environment.
All is not well with GIO. The code to unmount devices is not working. Is this a permissions issue? Missing the ability to unmount / eject devices wrecks usability with cameras and smartphones, because theyâre totally inaccessible to Rapid Photo Downloader while any other application is using them.
The code to check a new version is not working - is network access blocked by default, or is openssl missing? I donât see python being built with openssl support in build.sh. Iâve coded the application such that it will only offer to download a new version if it was installed using pythonâs pip. If not, it merely alerts the user that a new version is available.
Here comes the next version. I realized that my Glib was compiled without libmount support, which should explain why devices could not be un-mounted. I also compiled the missing gstreamer plugins, so this should be fixed as well.
Iâm already using version 15.1.0 for pyzmq, or at least that is what should be bundled⊠tomorrow I will re-build everything from scratch, just to make sure I have nothing left from previous builds.
LibRAW is at version 0.18.2, which is quite newâŠ
The main problem remains with the icons. I do have this file installed in the prefix:
-rw-r--r-- 1 photoflow photoflow 11M May 10 06:00 /yyy/lib/python3.5/site-packages/raphodo/qrc_resources.py
libraw should be 0.17.2, because its python binding rawkit is (incredibly) not yet updated to work with 0.18. I say incredibly because one of our forum members generously contributed the code for the project developers to make it work with libraw 0.18, but they havenât bothered to make a release. Quite disappointing really.
Nothing is missing from the Rapid Photo Downloader code. If it were, there would be a python exception. The problem is with the Qt environment youâve got in there. Sorry to say the same thing over and over, but seriously I think you really need to dump the custom compile of Qt 5.6 and stick with the Qt 5.8 supplied by the creator of PyQt itself. Itâs so simple to install using pip, just like is already done with all the other python dependencies. Thatâs the approach taken in the example Python Appimage application. If there is still a problem with the Qt icons after installing Phil Thompsonâs PyQt & Qt bundle then I can ask on the PyQt mailing list whatâs going on.
Unfortunately, the version of gstreamer included has a bug that means it wonât generate the thumbnails for videos. That bug is present in versions of gstreamer circa Ubuntu 16.04, but is fixed in newer releases.
The unmount still is not working in this latest release.
I downgraded LibRAW to 0.17.2, and installed several python packages (including PyQT5) using pip. This seems to at least allow to fix the âmissing iconsâ problem. Thumbnails seem to be also generated correctly. On the other hand, I am not sure about the camera un-mounting⊠I have the impression it still does not work, but Iâd like to get a confirmation. I have really no clue how to debug this specific problemâŠ
Well it sure looks much more aesthetically pleasing now! Good job with the new Qt version!
There is a tricky situation with libraw: currently, itâs not actually being used from within the AppImage. Here is why. rawkit uses ctypes, and as discussed at that previous link, when pythonâs ctypes goes looking for the library file, itâs looking everywhere but inside the AppImage! The solution is to hard code the filename, instead of using ctypes.util.findlibrary(). This is the point at which the rawkit code needs to be changed: rawkit/bindings.py at ede197ae5235c022eb100ce7de2b921cbbbc993c · photoshell/rawkit · GitHub. I havenât seen what the name of the libraw library is in your AppImage, but that line of code should be changed to something like libraw = 'libraw.so.16'. The question is, how do we do that? Do we have to fork rawkit? Or is it easy for the AppImage build script to download the rawkit source, patch it, and then include it?
There is another problem too: it seems the AppImage is actually not self-contained in the sense that it can run code from outside itself. For example, when I run the AppImage on my main development machine itâs actually running newer Python code from my home directory. It can do that because my development code is on the python module search path. I donât think the AppImage should be able to do that, personally. Is that something that can be changed? It might just be a matter of setting the environment variable PYTHONPATH to a default value? (As long as that doesnât affect the host system!).
I stated earlier that the version of gstreamer bundled with the AppImage has a bug that means that it is unable to generate thumbnails for any video files. However I was perhaps wrong about that. The problem could be with PyGObject, not gstreamer. I realized it only now when adding additional version reporting to Rapid Photo Downloader when it starts up. Here are some combinations that do and do not work with respect to extracting video thumbnails:
As you can see, some combinations that work in one distro do not work in another. Itâs difficult to understand how this makes sense, but perhaps some distros are applying a patch and others are not?
Regarding the unmount problem, a related problem is that GIO volume monitor is not always being informed that a new device has been mounted or unmounted either. To me that indicates something is not right with the GIO install in the AppImage. Iâve turned on some verbose debug messages relating to GIO that I had turned off some time ago, because theyâre often too verbose. But they might help in this case.
If rawkit is installed from sources, and not via pip, then it is realtively easy to automatically patch the code. This is another advantage of using jhbuild, as it supports patching as part of the build process.
Here is where the libraw.so library is located inside the appimage:
This is probably the knot of the rope, and the reason why some modules are failing. For example, the GStreamer/PyGObject combination that is bundled is 1.12.1/3.24.1, which from your table seems to work well in Ubuntu 17.10. Could you tell me which versions are reported on your system, and how to enable the verbose version reporting?[quote=âdamonlynch, post:17, topic:4474â]
Iâve turned on some verbose debug messages relating to GIO that I had turned off some time ago, because theyâre often too verbose. But they might help in this case.
[/quote]
How do I enable the verbose GIO messages?
I am now re-building the AppImage with a patched rawkit, I will post the new package as soon as available.
Those using snap are facing exactly the same problem. Here is a proposed solution for the Ubuntu tool snapcraft, which is not shell based but python based: Ubuntu Pastebin
While the python part where it patches the code is probably not relevant, but perhaps the environment variable part is. In any case, this part is foggy to me because I donât understand how by hardcoding the library name the program can actually locate it at runtime. My knowledge of Unix is a bit weak here.
To make those additional version checks and logging options, Iâve had to make some changes to the code since beta 7. If you donât mind install bzr then the easiest way to make a tarball with them is this: bzr branch lp:rapid cd rapid python3 setup.py sdist
The tarball will now be located in the dist directory. This is basically the 0.9.0 release, however Iâm reluctant to release it as the official release because if there is something I can add to the code to make the AppImage easier, then I think itâs worth waiting for that.
To quickly confirm what versions of various libraries are being used, run: rapid-photo-downloader --detailed-version
To see detailed logging output as you run: rapid-photo-downloader --debug
Perhaps. However, we canât rule out the problem is limited to when a user manually specifies their own directories using the env variable PYTHONPATH. It may not be picking up modules in ~/.local/lib/python3.5/site-packages
Just run with the --debug flag, or else you can always consult the log files in ~/.cache/rapid-photo-downloader/log (which have line numbers too).
Iâve not yet debugged the GIO problem, but it seems I have at least fixed the rawkit library loading issue, with this patch.
The new AppImage package can be downloaded from here, and the updated set of build scripts can be found here. The new build script is called bootstrap2.sh.
This is what the --detailed-version option gives on my build system:
This is how images are loaded in Qt / PyQt. I seem to recall Gtk+ has or is moving to a similar system to Qt these days too, for performance reasons. In any case, as Iâve been saying, something is wrong with the Qt environment.
