dt: *embedded.jpg files: where do they come from?

LateJunction · November 27, 2021, 10:49am

I recently posted a question about how to remove files named of the form <.embedded.jpg> from my windows install of dt. Although intrigued, I didn’t ask, at the time of that post, as to why these additional files suddenly appeared in my library of images. Subsequent to their removal, I moved my Linux install from very old hardware to less old hardware with a clean re-install of Mint 20.2 and restore of data from backups (which do not contain these embedded jpgs). Now, again I find many files of this same form, and also of the form <.embedded_1.jpg>, within my Linux copy of my Windows image library. This time I’m not merely intrigued - I’m concerned: what user error is it that causes these embedded jpgs to be written to file? I’m not aware of any dt function/feature that allows me to do that.

Claes · November 27, 2021, 12:03pm

Have you been using shotwell?

Have fun!
Claes in Lund, Sweden

rvietor · November 27, 2021, 12:05pm

And if not using Shotwell, why do you mention “Windows image library” when you tell us you restored data from backup?

LateJunction · November 27, 2021, 12:15pm

God help me, yes I have! But that is another issue, excluded from this topic because the shotwell file names do not contain the string ‘embedded’, do they? And it is pretty obvious where they came from.

The shotwell fiasco is now solved, after a ‘whack-a-mole’ type game this morning, which had the fans in my PC screaming their heads off for more than an hour. “Shotwell-in-foot” is now banished, eternally (I hope).

But this still leaves my original question open,

LateJunction · November 27, 2021, 12:23pm

Sorry: mis-understanding of my words. The restore from back-up applied only to data that was managed on my Mint install, during the transition from very old hardware to less old hardware. The Windows image library had been copied to my older Mint install a long time before this and before the problem with embedded files appeared in my Windows install.

Shotwell, in its current development level, is only viable under Linux; the Windows implementation I looked at was essentially non-functional, in my experience; it was very quickly removed. So the associated problem I had with shotwell files (see earlier post) applied only to my Linux install.

kofa · November 27, 2021, 12:35pm

Have you perhaps enabled some Lua script, e.g., this one, provided as an example?

github.com

darktable-org/lua-scripts/blob/56ec258cc7db60b58294bf19d8c72dafbb24a347/examples/multi_os.lua

--[[
    This file is part of darktable,
    copyright (c) 2017,2018 Bill Ferguson <wpferguson@gmail.com>

    darktable is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    darktable is distributed in the hope that it will be useful,
     but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with darktable.  If not, see <http://www.gnu.org/licenses/>.
]]
--[[
    multi_os - an example script that runs on linux, MacOS, and Windows.

This file has been truncated. show original

darktable does not extract embedded JPG images automatically. Maybe follow this search and look through the files:

Anything ending in txt, po or md is just text, not executable code.
There are the C and C++ sources with the word ‘embedded’:

LateJunction · November 27, 2021, 12:57pm

That was my understanding too; dt does not extract embedded jpegs automatically - hence my question: what did cause them to be extracted?

As to running a lua script: I doubt it; I wouldn’t, at this time, know how - unless a script can be invoked with out my knowing? Which would be fascinating as the only one I have, I think, is ‘purge-nonexistent-images’ (or words to that effect).

Thanks for the references to GitHub; all three return the result:
" We could not perform this search. The listed users, orgs, or repositories cannot be searched either because the resources do not exist or you do not have permission to view them."

Which does not make a lot of sense to me.

The stated reason for failure appears to fall into either 1 or both of 2 vastly different categories. This makes me react with thinking along the lines of “why can’t Github decide which is which?”.

rvietor · November 27, 2021, 1:15pm

Actually, I don’t know any way to extract and store the embedded jpegs with plain dt (be it automatic or manual). (no experience with extra lua scripts)

But the script @kofa referred to is supposed to add a button to the interface within dt.
From the comments in the script:

The 'urge_non_existing_images.sh script has to be run from the command line (afaik).

And indeed, replicating that search from @kofa does not give any indication dt can generate the files you mentioned (I also got an error when trying to use the links…)

LateJunction · November 27, 2021, 2:45pm

Thanks for the further insight into the script referred to by @kofa . I have checked both of my dt installs (win & Linux): neither has an extra button in the ‘selected images’ module and there are no presets for this module. So this mysterious situation continues to be so.

One other thought: having learned thay Shotwell created it’s own set of .shotwell.jpg files, could another application - rawtherapee, for example - have created these embedded files?

kofa · November 27, 2021, 2:50pm

Sorry, I copied the links from github, maybe they got messed up (they no longer work for me, either).
Another attempt:
https://github.com/search?l=C&q=org%3Adarktable-org+embedded&type=Code
https://github.com/search?l=C%2B%2B&q=org%3Adarktable-org+embedded&type=Code
And yes, I realise the Lua script whose link I posted requires manual installation and activation.

Have you tried the following?

create a new directory
copy a few raw files there
make sure there are no .embedded.jpg files in the directory
launch darktable
import the directory
check if .embedded.jpg files appeared
if not, open the raw files in darkroom
check again if .embedded.jpg files appeared
quit darktable
if there were no .embedded.jpg files generated previously, do they exist after exit?

If darktable is the one to create the files, it may be worth trying from a clean configuration and launching with debug logging enabled (see darktable 3.6 user manual - darktable).

To answer your question regarding RawTherapee – as far as I’m aware, it does not create such JPGs, either. Raw developers don’t need those files, so why would they create them?

It’d be essential to figure out whether it’s darktable that creates the files or some other software (see the experiment above), otherwise you won’t be able to get rid of them.

rvietor · November 27, 2021, 3:07pm

What also might help in tracking down what creates them:

look at the file timestamps: the timestamp tells you when the file was created/modified, so could give a hint if you remember what you were doing at that time ;
or inspect the metadata in those *embedded.jpg files: Some programs add or modify a field to show that particular program was used.

LateJunction · November 27, 2021, 3:59pm

Thanks for the ‘process’. Having followed it, it’s clear that dt on Windows (linux not yet tested) is not guilty.

From a post I read earlier today (on the subject of “why can’t we have a capability in dt to export the jpg embedded within a raw file?”; a post dating from Nov 2020, which created a lot of discussion, some of it quite ‘warm’!) it appears that Faststone Image Viewer is able to save an embedded jpg. Next step for me is to understand how this works - currently I can’t see any function in Faststone which will do this.

I followed the GitHub links - nothing in them referred to ‘embedded’ AND ‘jpg’ together.

The story continues…

rvietor · November 27, 2021, 4:41pm

In that case, it might be useful to tell us what other programs you use on your images besides dt, if you want useful replies from here. Otherwise, it’s going to be pure guesswork (and I’m having some trouble with my crystal ball lately)

priort · November 27, 2021, 6:45pm

I use faststone all the time with cm turned on as a viewer and have never seen any embedded JPGs saved… There are programs that will extract and save the previews…

LateJunction · November 27, 2021, 7:37pm

Yes: sensible suggestion. For the Windows install of dt , for the previous few months prior to and after finding these ‘embedded.jpgs’ I probably ‘touched’ the images with: Adobe Bridge (once, months ago), Lightroom 6.14, Photoshop CS6, (both very rarely), FastStone (regularly), XnView-MP (a few times), Win 10’s Photos app. (regularly), PhotomatixPro 6 (as appropriate with bracketed images), Photomechanic 5 (but not since dt 3.6.1 arrived), Nikon Scan 4 (probably irrelevant), Vuescan (probably irrelevant), Luminance HDR (rarely), Glary Utilities (probably irrelevant), Canon Digital Photo Professional 4 (Innocent, based on test just completed), GIMP 2.10.28 (Innocent; just tested), Hugin, Raw File Converter 3.0 (by Silkypix, only for RAFs; innocent; just tested).

For the Linux install: XnView-MP, Rapid Photo Downloader, Pix, G’MIC-Qt, GIMP 2.10.28, Hugin, Inkscape, Luminance HDR, Vuescan, XSane. Image Viewer

I’m a little concerned that I might be wasting a lot of people’s time on this, especially as the embedded files have no re-appeared in my Windows install since I deleted them, in dt. On the Linux side, deletion was a little more complicated, thanks to the ‘whack-a-mole’ games I had to play with Shotwell. That’s all wrapped up now, and I now no longer have any unexpected embedded files.

Should we just close this thread here, giving ‘attacked by throgs’ as a closing reason code ?

paperdigits · November 27, 2021, 7:45pm

We don’t really close threads. But you can abandon it