@gadolf Your local keywords are not included in the prediction. As soon as you have trained a neural network, the state will be frozen after the export and it will not learn anymore (there are certain special cases like lifelong learning or incremental learning, but I don’t want to go into too much detail). Precisely this implementation only passes data through a neural network to extract some important representations of your image.
The main idea of the implementation of photils is that I used ~ 1 million images as reference images with multiple tags per image. When you trigger the “get tags” method, photils extracts a feature (512 bytes) from your image and compares it to the 1 million images on similar images. As the last step, I select 30 images that are closest to your image and read the corresponding tags. The tags are sorted by the occurrence, which means that the tags are more common at the beginning, while later tags are less relevant but potentially not wrong. In other words, from a technical side, a simple similarity search like it happens when you reverse search images in google.
To describe your questions about the country. This is a really special case and more used in “place recognition” than keyword recognition. In other words, it could be, in all probability, that your place is not in my 1 million image dataset. I am not Google, I’ve limited resources and actually photil is more for simple photographic keywords (natural, minimal, abstract, macro a.s.o) than place locations.
Just to add more information to my case, photils got 173 tags from this image, among which these ones are place related:
scotland
norway
atlantic
northsea
spain
tasmania
australia
british
england
greatbritain
hampshire
unitedkingdom
wales
britain
northwales
hawaii
pacificocean
cornwall
florida
norwegen
minnesota
unitedstates
usa
america
california
france
iceland
I’d say that your image dataset is a bit First World centric
You are right
But this is maybe related to all which are interested to improve the plugin.
This is a tag list I’ve collected which filters the tags. Maybe we can extend or correct this list just to get better results.
I didn’t get it.
You mean I could add, for example, Brazil, or Rio de Janeiro, to that list?
What role does that list play on the pattern recognition process?
You mean I could add, for example, Brazil , or Rio de Janeiro , to that list?
Yes
What role does that list play on the pattern recognition process?
It has no influence during pattern recognition. But after I select the closest images I read all tags of these images. All tags are going through this tags_flattened list and each tag that is not part of this list is ignored.
That’s a good point, I would suggest only to use english and later I’ll think about translation (the current list contains German as well, I’ll cleanup this). But once we have a clean list in a single language we can easier translate it as a mixed list I think.
A more basic question:
if an image gets tagged with 173 tags, among which 24 geopgraphical tags spread all over the globe, what’s the practical use of such an application?
I’d still have to review the tagging, but this time to remove irrelevant or even plain wrong tags.
Add to that the network traffic and potential privacy concerns, and I think I’d rather not see such a plugin integrated into darktable as default (no problems if it’s an optional installation, in which case the user is responsable for its presence).
I did notice that the network version does not send the full image, but still, it sends content info, which is in itself information that can be linked to a user.
At least the first tags are usable and for me it helps a lot because I’ve not a huge, neat, tag base. Actually I’m too lazy to search in my tags which one could be useful or not and this plugin helps with suggestions.
But yes that’s your opinion and it’s up to everyone whether it’s useful or not. That’s my contribution because it makes MY life easier but sure you don’t need to install it if you have some security concerns or if the traffic is an issue for you.
Btw, the whole code is online so everyone can check what’s happen with the data…
It does not work…
Copied dtutils.lua in the right place. Entered reference in luarc.
Launching darktable -d lua obtain a lof of errors (lib/dtutils missing)
A feedback.
So far, my tagging strategy is mostly geographic and known people oriented, so, as it is, the plugin is not so useful to me.
However, regarding usual generic objects and things, like cars , clouds , birds , etc, it seems to work very well (although they’re not my main taggin interest).
So, good work!
Thanks for the feedback! Can you give me an example for tags you mostly use (geolocation is clear to me)? As I mentioned I’ll improve the plugin to be more generic and useful for a broader mass.
For now, just names of known people and dog
EDIT: My tagging strategy is having three main categories: What, Where, and Who. What is some action, activity, things, like Chrisrmas, Carnival, climbing, music and some generic ones for which I care, like clouds and plants .
The others are kind of self explanatory.
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27' not found (required by /home/mau/bin/photils-cli-linux-x86_64.AppImage) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.26’ not found (required by /home/mau/bin/photils-cli-linux-x86_64.AppImage)
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.26' not found (required by /tmp/.mount_photiloYluBS/usr/bin/../lib/libopencv_imgcodecs.so.4.3) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27’ not found (required by /tmp/.mount_photiloYluBS/usr/bin/…/lib/libopencv_imgproc.so.4.3)
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27' not found (required by /tmp/.mount_photiloYluBS/usr/bin/../lib/libopencv_core.so.4.3) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.26’ not found (required by /tmp/.mount_photiloYluBS/usr/bin/…/lib/libopencv_core.so.4.3)
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libc.so.6: version GLIBC_2.27' not found (required by /tmp/.mount_photiloYluBS/usr/bin/../lib/libgssapi_krb5.so.2) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27’ not found (required by /tmp/.mount_photiloYluBS/usr/bin/…/lib/libIlmImf-2_2.so.22)
16,513951 LUA ERROR command ‘/home/mau/bin/photils-cli-linux-x86_64.AppImage’ -i ‘/home/mau/Immagini/Archivio/2018-04 Venezia/_DSC6602.NEF’ -o ‘/tmp/lua_wa7b4S’ returned error code 256
16,514110 LUA use export to for prediction
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27' not found (required by /home/mau/bin/photils-cli-linux-x86_64.AppImage) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.26’ not found (required by /home/mau/bin/photils-cli-linux-x86_64.AppImage)
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.26' not found (required by /tmp/.mount_photilMmOoio/usr/bin/../lib/libopencv_imgcodecs.so.4.3) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27’ not found (required by /tmp/.mount_photilMmOoio/usr/bin/…/lib/libopencv_imgproc.so.4.3)
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27' not found (required by /tmp/.mount_photilMmOoio/usr/bin/../lib/libopencv_core.so.4.3) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.26’ not found (required by /tmp/.mount_photilMmOoio/usr/bin/…/lib/libopencv_core.so.4.3)
/home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libc.so.6: version GLIBC_2.27' not found (required by /tmp/.mount_photilMmOoio/usr/bin/../lib/libgssapi_krb5.so.2) /home/mau/bin/photils-cli-linux-x86_64.AppImage: /lib64/libm.so.6: version GLIBC_2.27’ not found (required by /tmp/.mount_photilMmOoio/usr/bin/…/lib/libIlmImf-2_2.so.22)
17,724132 LUA ERROR command ‘/home/mau/bin/photils-cli-linux-x86_64.AppImage’ -i ‘/tmp/lua_78Olhz’ -o ‘/tmp/lua_SB8Kae’ returned error code 256
Hi, Tobias,
I find your project very interesting. I let photils-cli evaluate some pictures from my portfolio.
local or hosted
It is fast, it is small and flexible. But I would like to have the choice if the comparison database is used on the internet or if the 500MB you mentioned are downloaded again.
If I understand this correctly, you can continue to train the database for your own needs, especially if it is local.
Accuracy / Tolerance
I consider this training very important. I exported images from my DT library without Exif metadata and ran photils-cli.
The result is very comprehensive and unfortunately only a very small part of it is correct. The tag “people” is generated even though there is not a single person in the image, it shows England, Germany, Holland, America, Belgium, India, Norway and many other countries.
The generated tags are very conflicting and I would like to be able to influence the accuracy. Is it possible to do this like with Face_Recognition (tolerance value)
Database
If the database is also local, it would be great to be able to train it with your own pictures. The hosted version should remain an option for the user to decide for himself.
Integration into darktable
The integration has been very successful. Thumbs up!
I will continue to follow your project with interest and hope that the points mentioned can be implemented. I also like testing
AppImage
I personally would prefer to get a description of how to translate photils.cli myself. I have reservations about AppImages and don’t like those “ready to use” things.
Hey Marcus, thank you very much for that constructive feedback!
I completely agree with everyone that wants an offline version. I also prefer this but it needs further investigations. The current used algorithm works more or less well with non-balanced data and that’s the reason for the usage. It allows it as well to update new data faster than the offline version but with the drawback to be online. From now I start working on a complete offline version which hopefully doesn’t need the 500mb reference data anymore.
The algorithm I use is more simple, you have not a confidence value that an object belongs to class x or something. But I think I can implement something that allows some manual adjustments. Meanwhile I’ve removed all location-based tags because this is really imprecise. All of you are right in this case.
Yes, definitely something to think about. Maybe nothing for the next update but for the near future.
thank you
I thought using AppImage is the most comfortable implementation for photographic users without a deeper understanding of *NIX systems. But sure, I will update the instructions for manual compilation. BTW. the .travis.yml file contains all required steps for manual compiling the source and all required dependencies a.s.o.
Again thank you! This helps me to improve the tagging functionality. If I’m ready with a new version, I would contact you for testing if are you ok with that?
and actually photil is more for simple photographic keywords (natural, minimal, abstract, macro a.s.o) than place locations.
