Solved: Auto import a directory when a file arrives?

Swapped code blocks around so that it waits an interval after detecting the file to import it.

Tried setting the sleep to 0 between display changes. It didn’t work. The lighttable display doesn’t finish loading before I tell it to go back to darkroom, so it goes to lighttable and stays. Reduced the wait to .5 seconds so it’s a little less noticeable and that worked. I’ll look at the API change, but the earliest it would make it in is 2.6.3.

Here’s an updated copy if you want to play with it.

Thanks,

Bill

filetether.lua.txt (6.3 KB)

will you submit it to https://github.com/darktable-org/lua-scripts ?

Yes, when I get it working well enough and fix the API so that I don’t have to flip back and forth from lighttable to update the darkroom image (actually I’m working on it right now - the API).

I now have dt.gui.views.darkroom.display_image(image). Now I just need to document, raise an issue and submit a PR.

Thanks Bill! I’ll give it a go tonight, I don’t have my camera with me. darix’s scripts don’t activate the image in the darkroom on this machine either so it is consistent with my remarks last night.

Just curious, are you able to query the file size from your script? If it is possible, the import can commence once a static file size is detected.

I looked a bit into changing the FlashairSync script to buffer to say /tmp and then move it after the copy is complete. But testing last night it won’t be bullet proof either. I dragged a test raw file I downloaded from a GFX Fuji and it corrupted; I believe the size was probably enough that it wasn’t complete when your program opened it. Depending on computers and files sizes it may not be predictable. Ditto on using a delay.

I’ve been thinking about ways to make sure the file is complete. I thought about checking the size, but didn’t go there yet. I’m also thinking about if the camera is attached to the computer and the computer recognizes the camera and can mount the camera as a disk, then I could just read the camera and copy from there. I’m curious to see if waiting the extra interval is dependable. I also made the api change so that I can change images directly in darkroom without having to go back and forth to lightroom. Now I just have to do the documentation and submit it.

1 Like

TBH I still feel the best approach will be the have the tool that copies the file to the directory to notify darktable via dbus. the tool will know when copying is done.

polling is always in danger of a race conditions. if you can play event based … I would recommend that. you can see houz’ watch.sh for the required dbus messages.

we could e.g hook it into tools like rapid-photo-downloader

1 Like

That works for llinux, but I’m not sure if it works for macOS and I’m pretty sure it doesn’t work for windows. I just looked and dbus is a maybe for both but takes some work. How about a pipe? The tool that copies puts a message on a pipe and a darktable script pulls it off and does the import? It doesn’t require anything special and it’s event based.

Hi Bill,

I just tested and it still corrupts. But I may have given you bad info. I timed the transfer and it is actually 13 seconds not 7. It feels slower tonight; not sure if it’s just me or if it really is inconsistent.

The other thing that is weird is I just tested and I can’t open the GFX50R files on my machine at home (ver 2.7.0+764) but they open at work (not sure the version). So the corruption last night may not have been with your program.

Kev

Try increasing the interval to something like 20 seconds and see if it still corrupts. Since you’re running 2.7 I’m assuming you’re compiling. If you would like the updated darkroom.c file with the API change and the changed version of filetether.lua that uses it, let me know.

P.S. One thing I am wondering about is when it reads the file early and it says “failed to read white balance” why does the error not go away once the file is complete? Is it stuck in the database that darktable can’t read the file?

Try reimporting the whole directory and see if that clears it. If it’s still there, select the image and do a remove, which deletes it from the database but leaves the file. Then, try a directory import again.

One more thought. If the remove/reimport doesn’t clear it remove it again then make sure the xmp file is gone too, just in case it had the error in it.

I looked at houz’s watch script. The good thing I found was the inotifywait, which of course works fine on linux but not macos or windows. Houz was using it to trigger sending the dbus messages, but I could use it to tell when the write was complete and then just import the file. Now, if I can figure out MacOS and Windows solutions…

Hmmm… ran into a bit of a problem. I updated darktable (now at ver 2.7.0+764) and your script is not working consistently anymore. It grabs the first shot and then after that I have to stop and start and it brings the files that it missed importing. Then it misses the first shot and any I shoot until I stop/start again. I used the -d lua switch and got this when it sensed the file (and it seems to crash):

150.724914 LUA importing /home/me/Pictures/01]OOC/FlashAir/226LEICA/L1009312.DNG
150.735993 LUA L1009312.DNG imported with id 3797
150.738243 LUA L1009311.DNG.xmp
150.744970 LUA L1009313.DNG.xmp
150.750880 LUA New
150.759022 LUA importing /home/me/Pictures/01]OOC/FlashAir/226LEICA/New
150.760413 LUA ERROR : …dmin/.config/darktable/lua/flashairtether/filetether.lua:93: field “filename” not found for type dt_lua_film_t

stack traceback:
[C]: in ?
[C]: in metamethod ‘__index’
…dmin/.config/darktable/lua/flashairtether/filetether.lua:93: in upvalue ‘start_polling’
…dmin/.config/darktable/lua/flashairtether/filetether.lua:169: in function <…dmin/.config/darktable/lua/flashairtether/filetether.lua:166>
[C]: in ?

It appears that it tried to import a file named New that wasn’t an image.

Oops, now I seemed to have crashed darktable with this:

721.911223 LUA number of files is 29
723.911445 LUA view in darkroom is true
723.911536 LUA L1009311.DNG
723.946635 LUA importing /home/me/Pictures/01]OOC/FlashAir/226LEICA/L1009311.DNG
723.954908 LUA L1009311.DNG imported with id 3796
723.957524 LUA L1009312.DNG
723.981323 LUA importing /home/me/Pictures/01]OOC/FlashAir/226LEICA/L1009312.DNG
723.986190 LUA L1009312.DNG imported with id 3797
725.070101 LUA L1009311.DNG.xmp
725.097587 LUA L1009313.DNG.xmp
725.134369 LUA New
725.180276 LUA importing /home/me/Pictures/01]OOC/FlashAir/226LEICA/New
darktable: malloc.c:2868: mremap_chunk: Assertion `((size + offset) & (GLRO (dl_pagesize) - 1)) == 0’ failed.

I’m not sure what the file named New was or where it came from, but it appears that the script and darktable didn’t care for it. Perhaps I should check that it’s an image before I try and import it :smile: I’m going to rework it and use inotifywait to watch the directory for changes then react when inotify signals a close_write event.

New is a directory I created to play when I was playing with the failed images. Good bug to catch!

Tried inotifywait, but it blocks so that was a no go. Found a way to determine if the file is closed, so I check it and if it’s still open I keep checking every 2 seconds until it closes. Also put a check in for entries that don’t resemble a filename. Here’s a new version if you want to play.

Bill

filetether.lua.txt (7.5 KB)