How can I interpolate or crossfade a series of images?

I have some series of shots of polar lights and am looking for a way to interpolate or crossfade them to create smooth(ish) videos in Linux.

The following three images are from a series of about 800 exposures, eight seconds each.

I’ve tried slowmoVideo, but that results in stars sometimes moving - they get pulled out of position by the moving polar lights.

I then tried to use basic crossfading with ImageMagick’s convert command and its “morph” option as per here, but that gets killed after quickly using all my RAM and swap even if I set resource limits.

$ convert source-images-1080/IMGP*.jpg -delay 10 -morph 10 -limit memory 32MiB -limit map 64MiB imagemagick/%05d.jpg
Killed

I’ve also tried this bash script which uses ffmpeg, which seems to be promising. The resultant video shows each of the source images for a second or two after each crossfade, though, and I’ve left a question for its author on how to remove that as I don’t understand the commands used.

It looks like G’MIC might be able to provide for the crossfading, but its documentation scares me!

Does anyone have any ideas or guidance on how I could go about this? Many thanks.

There’s two main options I would think:

  1. Buy @David_Tschumperle a hot chocolate and press him for a simple G’MIC script (or any of the G’MIC filter authors for that matter).
  2. Possibly doing something in Blender? I’m not sure how best to handle crossfading so many sequence of images, but there might be a way (I’m not near my main blender machine atm to test it).

The problem with imagemagick is that it will try to load all of your images into memory(!) before it can start to create output. Have you considered doing it in batches maybe? Like, every pair, then every pair, etc… until you reach the end? (I used to have to do this until David fixed it for me in G’MIC).

1 Like

That’s an interesting problem, I’ll try to do something about that.
I’ll let you know.

OK, I’ve done a quick hack, maybe you can test and tell me if that works for you.
It tries to minimize the memory usage, and compute a temporal fading of a serie of image, in a streamed way.

First, you need to update your command definitions (only once):

$ gmic -update

Below is the command line that works for me with the bash, if you are on Windows, there should be some differences (no need to backslash the * I think). I’ve not tested on Windows though.

$ gmic -w -fade_files IMG\*.jpg,30,0,-1,1,output.png

The parameters of this command are, in this order :

  • Filename pattern
  • Number of fading intra-frames.
  • Indice of the first frame
  • Indice of the last frame (-1 means ‘last frame available’)
  • Frame step (1 is default)
  • Output filename (increasing numbers will be appended, like output_0000.png, etc…).

If you don’t want to have the visualization window, just drop the -w option when invoking G’MIC.
Hope this will be useful.

5 Likes

You could try loading them in a non-linear video editor and applying either temporal blur or a framerate-doubling algo.

@David_Tschumperle its great that you’re working on something with this :slight_smile:

Wow - thanks for that! It seems to be working well for me in Ubuntu 14.04; I installed G’MIC from the Ubuntu repository and then followed your instructions. Here’s my test result:

https://youtu.be/sYixTgSZw28

After a bit of experimenting, I decided to go with 7 intra-frames into 60fps video for my eight-second exposures. It’s not perfectly smooth of course (I suppose I should try to reduce the exposure length in future), but I’m pretty pleased overall. I’ll probably tweak the tones and darken the night sky slightly before adding some Sigur Rós music. :wink:

What is the frame step?

The commands I used:

$ gmic -w -fade_files source-images-1080/*.jpg,7,0,-1,1,gmic-7/nl1.jpg

$ ffmpeg -r 60 -pattern_type glob -i 'gmic-7/*.jpg' -qscale 1 nl1-gmic-7-60fps.mp4

Where can I contribute to the hot chocolate fund?

8 Likes

Wow! Awesome result! It can definitely fit some sigur ross.

Try here at paypal. :smiley:

1 Like

The frame_step parameter can be used to ‘skip’ some frames for the computation, like considering only one image out of 3 in the image sequence. It is not that useful if the image sequence is a list of image files, but can be if your input image sequence is given as a video file (using command -fade_video).
I’m happy to see this is working as expected! Your resulting video is truely amazing :slight_smile:

http://slowmovideo.granjow.net/ might be worth trying too. It’s calculating optical flow and then morphing the images. With that said, you video does look very good already! :slight_smile:

That is something G’MIC should be able to do also. Give me 5 minutes…

I do remember seeing the optical flow estimation at some point. It would be really cool if we could get G’MIC to do that! :slight_smile:

Actually, command -morph already does that, but it’s a bit memory consuming right now (because it keeps all interpolated frames in memory). So, I’m working on a ‘streamed’ version of -morph.

Thanks for the feedback. I’ll let you know when I’ve got a proper edit done which I think will contain some other sequences, too.

I see - thanks.

That’s the one I initially tried, but I found that sometimes some stars would be moved along with the lights and then snap back into place on the next “keyframe”. I’d definitely be up for testing a streamed version of G’MIC’s morph!

OK, so I’ve added another set of commands -morph_files and -morph_video which basically works like fade_files and fade_videos, with some extra parameters for the morphing algorithm.
The morphing is done between two consecutive frames by estimating both the forward and backward motion vectors (using an optical-flow like method), then interpolate the frames temporally using these two motion vector fields.
No need to say this can be very time consuming ! :smiley: (even if like me you have 24 cores!)

How to use it?

  • First, update your filters, and check the commands are recognized:

    gmic -update gmic -h morph_files

     ...(help should display here)...
    

An example of use, using image files as the input (here again, using bash on Linux, may be slightly different on Windows):

$ gmic -w -morph_files input_images\*.jpg,20,0.1,4,0,-1,1,output.png

(you may want to replace output.png by output.avi to gets an .avi video file as the output).
And that’s almost the same if you have an input video file, instead of a sequence of images:

$ gmic -w --morph_video input_video.avi,20,0.1,4,0,-1,1,output.png

The smoothness parameter is important to set correctly for the morphing algorithm. Basically if the frames you want to interpolate can be well registered by a rigid motion (translation,shift,…) then the smoothness can be high (like 1 or 1.5). For non-rigid motions, try a lower value (0.1 is medium, 0.01 is low).

2 Likes

After using the update command, the morph_files command doesn’t seem to be recognised.

$ gmic -update
[gmic]-0./ Start G'MIC interpreter.
[gmic]-0./ Update commands from the latest definition file on the G'MIC server.
[gmic]-0./ End G'MIC interpreter.
$ gmic -h morph_files
  gmic: GREYC's Magic for Image Computing. 
 
        Version 1.7.3, Copyright (c) 2008-2016, David Tschumperle. 
        (http://gmic.eu)

[gmic] Command 'morph_files' has no description. Try 'gmic -h' for global help.

hum. Isn’t it because of some caching problems ?
Try to get the update file directly from http://gmic.eu/update174.gmic, and put it in your $HOME/.config/gmic/ folder (replacing the older one).

Here is an example of the (subtle) differences you get with commands -fade_video and -morph_video.

Not sure the morphing here adds a lot (except for the computation time of course :wink: ).

1 Like

Adding that file to G’MIC’s config directory has unfortunately not solved the issue. I also tried adding cli_update174.gmic.

That’s a nice comparison video. The morphing really does help the smoothness in many places.

Ah yes, sorry but if you have version 1.7.3 of G’MIC, you need to rename this file $HOME/.config/gmic/cli_update173.gmic. It should work then (but in this case, don’t invoke $ gmic -update, otherwise the cli_update file173.gmic will be overwritten.

I see - thanks. I’ve now downloaded the following files and replaced ‘174’ with ‘173’.

gmic -h morph_files is still not showing help for the command, and gmic --morph_files gives:

[gmic]-0./ Start G'MIC interpreter.
[gmic]-0./ *** Warning *** File '/home/davido/.config/gmic/cli_update173.gmic' is not a valid G'MIC command file.
[gmic]-0./ Input file '--morph_files' at position 0
[gmic]-0./ *** Error *** Unknown command or filename '--morph_files'.

It seems like I’ll need to update G’MIC in order to test?

Edit: should the morph_files command be preprended by one or two dashes?