Timelapse photography and darktable

Yes, definitely, thatā€™s actually done in darktable already (when we read those params). I even think that if you didnā€™t enable the XMP compression in preferences, they are already stored in clear.

Reading your whole post, I wonder actually how much work it would be to just re-implement the feature from scratch in dt. Matching white balance and average brightness is not rocket science.

2 Likes

I just tried with switched off compression, the values are still not really stored in clear but (assumed) bases64

Thanks for your supportive attitude.
I do admit, backwards compatibility could be a pain. Another good reason, to pack such things in fresh modules only, to not hurt, what we have already.

There was some communication on github about how to decode the xmps, maybe have a look there (donā€™t remember which issue or pr). I still donā€™t think special modules are needed to interface to lrtimelapse, itā€™s all there.

1 Like

I didnā€™t read the whole thread, but I have a more or less working open source timelapse workflow. Darktableā€™s exposure module does already do a great job for deflickering and equalizing the exposure. Then itā€™s just a question of tuning filmic (filmicRGB now). A good starting point is the picture with highest dynamic range of the whole sequence. Then edit as usual (I like local contrast & tone curves) and copy the history stack. Be careful with masks, however, parametric masks can be helpful to keep the highlights in check. The whole process is a bit more involved and I mean to write an Howto, essentially documenting my whole workflow, but Iā€™m drowning in work atm.

The final piece (actually this should be done before all of the editing) is smoothing out the white balance if you shoot your timelapse in AWB (*), like I do. Iā€™ve written a Python script which reads the WB coefficients from the RAW files, applies a smoothing filter (google Savitzky-Golay) and writes it to the XMP files. The params in the XMP file are HEX-packed binary structs, for WB thatā€™s only 4 doubles. Pretty easy to interface with Python once youā€™ve figured it out. However, my script is CLI only and not very polished but Iā€™m satisfied with the results.

Iā€™ve recently extended the script to allow interpolation (not smoothing!) of exposure and graduatend based on keyframes. That means, pick some files of your sequence, tune exposure (very rarely necessary) and / or graduatend and automatically interpolate all files in between. I still to hacky for a release though.

Here are some Videos I processed roughly following my workflow & using the script:

10 Likes

Nice, I especially the Burg Rabeneck movie. How many images per minute do you usually shoot?

Thank you! :blush:

Depends on the subject. Astro ususally with 30s intervals, so I can expose long enough when itā€™s dark. But itā€™s also subject to the 500 rule for astro shots :slight_smile:
For clouds I find ~4s good, depending if I want a sunrise or sunset transition that needs to be longer.

Day-to-Night and Night-to-Day is difficult, hence the term ā€œholy grailā€. At the moment Iā€™m experimenting with fixed 4s intervals (Shutter speed mode and a variable ND filter) and 2s pauses. Deflickering can be done in SW, so I donā€™t care about my aperture changing. The upside is that I can use f/16 for a nice sunstar and once the sun has set the aperture will open automatically. The downside is (potential) flickering and that itā€™s getting to dark for the VND later at night, so youā€™d have to remove it, which means touching the camera, which is dangerous :wink:

Btw I also found that you want to be careful about the lens correction module. Different apertures mean different vignetting correction. I now only do distortion & TCA, which seems to work fine.

Looks really nice!
Would you mind to put you python script on GitHub, even itā€™s Ā«too hacky for a releaseĀ»? It could be a useful starting point for me anyway, Iā€™m looking for something since a while and didnā€™t have an idea where to start. :blush:

Iā€™m looking forward to this!

1 Like

Here you go :slight_smile:

6 Likes

Thank you very much!

Iā€™d really love to see this integrated into dt. @anon41087856 any change you will have a look into doing this, or would it be faster if I would start learning c now?

Not going to discourage you, but darktable is a vast C++ code base which is more or less a framework around modules. Even if youā€™re going to learn C++ now itā€™ll take you some time until you understand darktableā€™s code base which is (afaik) pretty undocumented. In addition, this is not a standard module in the sense of take pixels from the pipeline, modify pixels and write them back, but rather read information from a group of images and apply it to a group of images, a process darktable was never designed for. So prepare for some hacky solutions here. I wouldā€™ve loved to this in LUA though but the API isnā€™t there, thatā€™s why I ā€œreverse-engineeredā€ the XMP interface.

Apart from that, god speed to you! :slight_smile:

Iā€™ve seen the script and all it does is calculating and applying smoothed out WB across set of images (sorry for trivializing actually great piece of work). Technically it can be either an operation in DT in ā€œselected filesā€ module (so youā€™d have to learn C) or LUA script that would do the same :wink: So you can either learn C or LUA. I think lua script can also call Python script that @jchnkl done (lua scripts can call external binaries to do processing). So ultimatelly options are - introduce this as an op in C, introduce it as an op using LUA (total rewrite in lua) or intruduce it as an op in lua dependent on python script.

My stack of tasks is pretty much full for at least the 2 next months, so if you are able to learn C in less than 2-3 months, chances are you will be faster than I am.

1 Like

Well, you could do a ā€œmoduleā€ for the lighttable (src/libs/copy_history.c is close to what you do here), code a SQL request to fetch exposures/WB for the selected list of pictures, unroll your interpolation of parameters, save the SQL in database and refresh XMP files.

3 Likes

Great idea and doesnā€™t sound to hard either.
Iā€™ll be gone till mid-march soon, but Iā€™ll keep that in mind! Thanks for the hint!

2 Likes

Now that darktable has arbitrary module ordering, making a toolchain of module invocations in lua would be a powerful way to process multiple images.

I recently completed a batch capability for my own software, and it has surpassed my expectations for improving the efficiency of my workflow. For instance, after Iā€™ve produced my proof JPEGs after a dayā€™s shooting, if I find something like a white balance correction that applies to a series of images, I can open one of them (which opens the raw file and re-applies the toolchain to make the JPEG), change the white balance, then I delete the other proofs i want to apply the change to and kick of the batch dialog, which will process new JPEGs with the modified processing. Itā€™s not lua, but the concept is the same: have a scripting language that exposes the same image processing API as the GUI program uses.

Food for thoughtā€¦

No, the modules that you can reorder are the pixel pipe ones (where ā€œmoduleā€ means a GUI widget and an image operation filter, located in src/iop). Those are not accessible from Lua. Lua let you play only with pipeline I/O and the lighttable modules (where ā€œmoduleā€ means only a GUI widget to do arbitrary stuff not affecting pixels, those are located in src/libs)

Okay, some of what Iā€™m getting at is already in darktable-cli. From my initial read of the doc, one needs a previously-saved XMP with the desired history stack. Iā€™m not a large fan of history stacks, Iā€™d rather just have a ā€˜toolchain stackā€™ of the parameters I finally settled on, not all the less-than-desirable iterations it took me to get there. Okay, further reading reveals ā€œcompress history stackā€, which gets rid of all but the essential information needed to process the image with the final editing decisions. That gets me thereā€¦

I still prefer scripting my toolchainsā€¦ :smiley:

Just to be complete her, I found this video ā€œTimelapse stabilised with Hugin toolsā€:

And here is the code:
https://bitbucket.org/brunopostle/panotools-script/src/eee8f84f40fda3d6d1d5954fa7bb20848b8041fc/bin/nona-deshake?at=default&fileviewer=file-view-default

1 Like