Thanks for the speedy answer!
It’s standard CR3, not C-RAW. Do you need a file for testing?
Regards
Hajo
I’ll try the ones on raw.pixls.us first; both kinds are there.
Unfortunately, I think you have to wait until LibRaw releases support for the R5 and R6, unless you want to convert your raws to DNG with the Adobe DNG converter.
Hello CarVac.
I have not used filmulator for more than 30 minutes, but after over One year of trying to make darktable work for me, I had to concede that getting to a basic simple image, which I could edit further in darktable was a bit of a hit and miss, and I was never sure of how any image would look using the fundamental photo development tools in darktable.
I then went back to using Adobe Photoshop Express, Sony Imaging and Capture One and instantly in every case, straight from camera raw, I was presented with a much better starting off image, for further edits.
I then tried out your tool today, further to a recall of its mention in dpreview. Within a few minutes I was very impressed with the default results. In darktable, you have to spend a few minutes or a few hours to get to the same kind of results as I achieved in filmulator, without having to do anything.
Do you think there would be value in incorporating certain aspects of filmulator into darktable?
Darktable has nothing similar that produces instantly pleasing results. Nothing.
A marriage of darktable and filmulator would be fantastic, and I think it would open up huge opportunities for the work you have already achieved.
I have not tested filmulator extensively, only used it for 30 minutes and I immediately spoke to the darktable community (who are some a bit resistant to change) that it would be wonderful to have some of what you have accomplished in darktable.
What do you think of this.
It does not impede your own development of filmulator, only adds more opportunity to see the excellent work you have done, serve a much larger audience.
Until then, I will change my workflow to develop my initial images in filmulator and continue further edits in darktable.
I would definitely like to hear from you and hope that you can work with the darktable community to achieve a much needed enhancement to darktable.
In zero seconds, in filmulator, I achieved what could take hours of tweaking in darktable, and when I compare the results, no contest, I am pleased with what filmulator gives me straightaway - default.
If we could add filmulator to darktable - which is my preferred workflow editor, that would be amazing. like Filmulator being like the Adobe Camera Raw component to Lightroom, filmulator can be the same for darktable.
1 Like
I did once implement a Filmulation module in darktable but it really runs against the way darktable’s pipeline runs.
The Filmulation step must run using the entire image together in order to accurately reflect the final result, whereas darktable’s modules try to constrain the calculated input region of interest when you zoom in to minimize computation.
Filmulator’s pipeline instead always calculates the full image extents and caches heavily to maintain interactivity.
So in the end it really kneecaps the performance and doesn’t fit with the way the way you work in dt. On top of that, while I remember some people liking the output, at least back then I couldn’t get it to look right (or anything like the Filmulator default output) even with the Filmulation module.
Dear CarVac,
Thank you ever so much for your prompt and kind response. I truly appreciate this. I have invested so many days and nights, over the last year, getting to learn darktable and photography, and especially the dream of being able to take advantage of post processing of raw files, to improve upon some of what is possible via in-camera jpegs.
I will admit, it has been a bit of a frustration with darktable, the tool idea is great - flexibility, and the user interface and general stability of recent versions on Windows has been outstanding. In theory it works, but one has to judge by the final output, the images.
Not wishing to repeat what you already know, but just to have some context.
There are two or three key modules which I have used to transform raw files in dt to a visibe image, which one can revise further with other modules. These three are the base curve module, the various recent versions of the filmic-rgb module - aka filmic, and strangely there is also a crude method for generating this image using LUTS, in the 3DLUTS module.
LUT’s. :
The challenge with all these methods has been, a relatively inconsistent success rate, of any of the methods. The LUTS based methods, obviously requires images to conform to some of the methods for capturing video, such as sticking to the recommendations for IRE, skin tone - etc =, which are pretty strange and not readily applicable to photo cameras, which do not have any indicators or meters for the kinds of things monitored in video cameras. But with the right image strangely sometimes all you need is a LUT, and youhave something decent. some of my best image development in dt has been with using LUTs. strange but true but the method is not assured, too many rules that the image must conform to, and some uncertainty in my mind about the implementation of the 3DLUT module, related to colourspaces - which is not adequately covered in the user manual - very little information about it there.
Base Curves :
Using the base curve generally works, once you have an image that is definitely not overexposed, and ideally a bit underexposed, or dare I say it - properly exposed, according to the input ideal for the base curve. When I started using dt, I invested about a month with at least an hour a day, hand creating a set of base curves - conforming to different image styles, and exposures, for the digital camera which was my main camera at the time. Sure I have truly thrown away lots of time on darktable, cos in theory the whole solution is a great idea if it works. So I had a set of different custom base table curves and would try out different ones, as a starting point for each photo, and pick the most suitable and proceed from there.
The challenge with base curve approach is that from my laymans understanding, modifying luminosity leads to inconsistent changes in chromaticity or maybe its vice versa, anyway any attempt to keep things consistent then leads to a loss of colour and the image starts to look dull. The typical work around then is to allow the colour to become excessively saturated by turning off the luminosity preservation in the module - set this to none, and then attempt to use another module further down the process pipe, to reduce the oversaturated image. Clearly the end result of this approach is not consistent, across images. Definitely not point and click. It’s point click and pray, and up to 30 minutes of going back and forth between different modules trying to tame the beast that ran away and put it back in the cage, the beast of over exposure, created by turning off the luminosity preservation in base curve.
FILMIC
This has similar issues, by default, you get somewhat desaturated unnatural colours, but if you turn off the luminosity preservation option in this module, its similar to the base curve, colours become over saturated and you have to suppress this behaviour down the line bu undersaturating. Not a consistent approach cos the image also becomes unnatural. The filmic also has some issues with strangely modifying the sharpness of certain portions of the image. The way I express it best is to think that it destroys the natural contrast gradient in locally contrasted areas, and information or detail is lost. And it is impossible to then recover t
It was these inconsistencies with the most important aspect of photo editing, the initial “development” of the photo that I found really problematic, and this should be the easy part. An image editor should present me with a decent image, that is “high quality” from the start, without me having to first fuss over it. Then I can go further to edit to taste. Almost like buying a car, then I have to push it to start it - every single time. Trust me I have once had a car which behaved like that for a few days, when the electrical/battey charging system went bonkers !!!
All this was what led me to filmulator, I opened a few images and was impressed from day one. Definitely something I could work with, and develop further.
I appreciate that filmulator encapsulates what would be equivalent to several of darktables pipeline modules, into one tool, and implementing this in darktable would require fragmentation of the code into probably several modules…
I have read you response and here are my thoughts.
Especially in the open source world, its best not to reinvent the wheel.
For example you have a librtprocess library that is reused in rt, and in some other apps, , and this is also reused in filmulator, which is an excellent idea - develop once. From a brief look this module addresses the demosaicing of raw files.
Your app also does other things, provides a user interface, and organises and generates tiff and jpg files. On that note one of the major issues I found out when I gave your filmulator app a real workout, was the inability to determine the properties of the generated image file. I could not set things like jpg compression, bit depth, of the exported image file, nor could one address things like the gamut/colourspace of the final product. These are not particularly weaknesses of filmulator, cos the real strength of the app was to demonstrate its “film like image development”, which it does in a very promising manner.
I apologise if I am very direct, considering the time in which filmulator has been in development, it may take a while for the app to add all the bells and whistles, such as being able to change the settings for exported jpg files.
The beauty of something like darktable is that it already has stable code for these “housekeeping” functions like creating and generating and setting options for the kind of file to be generated., But it is not the best for presenting a simple high quality image to the end user as a starting point for further editing of the photo, which can be quite discouraging, and that is where your skills and what you have achieved could be the saviour of darktable.
I.e you being able to add much needed improvements to darktable, and your inputs can take advantage of existing house keeping code that already exists in darktable.
Lets take it one step after another.
Your filmulator app, appears to have what I would call several key aspects.
-
Debayering/Demosaicing. Which is possibly different from that in darktable. imagine you took that aspect of your code, and developed an alternative debayering/demosaicing module, or your code/library became available as an option in the preexisting module in darktable, so that one could choose settings or simply use the defaults that you already use in filmulator (should it not be expedient to expose such settings to the end user who could end up doing more damage than good - one does not give knives to children - cos they may harm themselves!).
-
Your app has an excellent approach to establishing the “film curve”. and you could create a module for this feature that darktable, in spite of all its efforts so far, has not excelled at, that encapsulates the key aspects of this film curve as an alternative to the filmic or base curve. I took one look at the image generated by default in filmulator and said yes - this is what we need in darktable.
So the idea is to pick the battles carefully, to focus on one or two modules that you could add to darktable, so people like me can finally enjoy darktable.
It does not have to be encapsulated in a single module. I’m sure you understand what I am trying to say.
Filmulator is a great idea, but it will take a lot of work to add all the user oriented features that would complement it, such as the generation of different image types, which is already included in darktable.
I nevertheless think adding the key aspects of filmulator to darktable, would be the most rewarding investment of your available energies.
- You would get a huge user base of existing users - a huge one, to challenge and test all your ideas. In recent times I have had to go back to using tools like Capture One Express for Sony, cos like filmulator the image development by default is just better.
I appeciate that one of the things you are considering, is the inclusion of a noise reduction module, and if you have alternative ideas about this, that could also be a new module in darktable, which users can optionally choose. This is the beauty of darktable, the potential flexibility for the users and also for the developers, so each of us concentrates on what we do best.
Those who know how to develop code for image processing can concentrate on those aspects, while others who enjoy improving user interface can also concentrate on that, and everybody benefits.
Please give my suggestion another thought, and I do hope I can encourage you to adopt the suggestion. Sure it will take some work, to conform your app components to dt’s , in separate modules, but especially as you do mention that you have done some of this before in dt, it should be a bit easier for you.
I look foward to a positive response and the hope that your existing efforts can become one of the next step forwards for darktable.
I acknowledge - nothing progressive is easy, but the effort is worth it.
Thanks for thinking through my suggestion.
I was especially appreciative of the excellent and not excessive sharpening in your filmulator tool. dt needs such quality of image sharpening. Lots of ideas from your app that could enhance dt, if implemented in appropriate new modules, as alternatives to some of the existing modules in dt.
Regrttably I have had to stop using filmulator, cos I cannot use the exported jpg, as I cannot define its quality, size, etc, or export into other image types like png, and all these things are already in dt. As you can see - merging what you have already achieved with what has already been achieved in dt, is the best way forward for both apps. - the features of filmulator and the excellent processing infrastructure of dt.
Hope that’s all we have. And we live only once.
These are planned to be released in the next version of Filmulator, together with noise reduction.
Interestingly, there is no conventional sharpening step in Filmulator, it comes entirely from the film simulation.
What I had initially hoped to do was “develop” in filmulator and bring in the result as a tiff file into darktable to continue any final edits.
ii.e in filmulator - let the image look good, which it already does achieve, minor fine tuning, set white balance if needed, then export as tiff.
Fortunately it does look like I can still achieve my intention with the current version. I checked the tiff format, its a 48 bit file. and full resolution, same as the original image.
That definitely meets my needs for now, kindof like how Adobe Camera Raw is the development “engine” for Adobe Lightroom and Adobe Photoshop, I can thankfully use your current version to “develop” the raw image, save as tiff, which is already an included capability, then continue my edits in dt.
Thank you for all your work. I have a solution with the combination of Filmulator and Dt, which works ok for me.
Is it possible that when you add the noise reduction, could you make it optional, so that if one were to continue photo editing in another app, one does not end up with too many layers of noise reduction, in different apps, which could reduce the image quality.
In dt there are a couple of ways for noise reduction, which I will not go into here, but to mention one which I found interesting - a raw denoise module, which is NOT enabled in the pipeline by default, but which I usually use now in all my photo edits, cos I found it to be the most appropriate, so I do not bother to use any other noise reduction except this one.
What’s interesting about this module is that it comes pretty early in the pipeline, even before the demosaicing. It would be interesting to discover where in your pipeline, your noise management is handled - before or after demosaicing, and especially if its optional in the app, I can then decide where (in which app) to do my denoising.
One more question. I have not been able to figure out where you store your image settings, in Filmulator. What I know is that when I edit an image, the settings are saved, and whenever I close filmulator and come back to it, the settings are saved, but I have no clue where this “state” of the image has been saved to. I am aware you have a database installed with filmulator - is this where recallable image settings are saved into, which would need me to ensure I maintain the same directory structure and also backup this database - if I move to another computer.
Then one more thing. There is a bug in Filmulator. When you use the white balance picker tool, rather than set a new white balance, it also overwrites the “saved” white balance and you cannot recall this saved white balance.
Ideally if I set and “save” a white balance for example so I can use the same white balance across many images, and then change the current white balance either through teh white balance picker or manually, I should be able to revert back to the “saved” white balance, by clicking on the appropriate button on the right of the screen, for this “restore” of the saved white balance.
At this time, any change to white balance using the white balance picker, erases and immediately overwrites, any “saved” white balance and there is no way to recover from the loss of the saved white balance.
And one more, I have now accidentally while learning your user interface - set the font size so big that I am no longer able to change it, cos the fonts are now so large after a restart, I can no longer get to the button that saves the new font setting, which should then be enabled after a restart. I can no longer save the revised user interface settings.
What initialisation file do I go to to - to restore things back to the default for user interface font size? The only thing I can think of now would be to uninstall, and reinstall, but then if the settings are in the database, I cannot access these as an end user via an editor. Please let me know how to resolve this. I am on Windows 10. 20H2.
1 Like
Good point on the white balance UI. I’ll change that for the next version.
Noise reduction will 100% certainly be optional.
Processing parameters are stored in a database right now. The database doesn’t get touched by uninstalling and reinstalling.
The settings location depends on your OS: for Linux it’s ~/.config/Filmulator/Filmulator.conf and ~/.config/Filmulator.conf, while for Windows I think it’s in the Registry under HKEY_CURRENT_USER\Software\Filmulator\Filmulator and HKEY_CURRENT_USER\Software\Filmulator\OrganizationDefaults.
Thanks - will check and get back to you. I am away for a few days so it may take a while.
Keep well.
I resolved the user interface issue by deleting the following key:
HKEY_CURRENT_USER\Software\Filmulator\Filmulator\ui\uiScale
And restarted the app. The app on restarting set the value to 1 in the interface on the settings “page”.
Please check that all the settings actually resort to their default values, when the reset button is used in the app.
In the case of the UiScale for resizing, the reset button does not work, and return this to the default value, if default value is changed. i.e unless you had a user who was really observant they do not know what the default value is and it would be impossible for them to go back to the default value
In the case of the UserInterface Scale setting on the settings page, definitely the reset button is not working as expected.
But I’m ok now, thanks for your help, at least I can use the app for now.
Please let me have your thoughts on what should be the best way to manage the non-destructive settings for each image which you have stated are stored in a database.
Are these settings stored in the database for each image, using relative links, or other kind of reference to the image file names.
If one moves the image files to another location or to another computer and wishes to retain the same settings, even if I copy the database to the new computer, how does one manage this since the new location of the file has no link to the previous location of the file on the old computer?
Obviously lots to think about.
I ran into a similar issue with darktable where I backed up my image files but did not backup the database. Fortunately darktable also generates .xmp’s for each image or virtual image duplicate, so I did not lose everything.
Lots to think about for your next software update. Happy to help you test the beta versions. If you let me know when they are available for download. I already test some key software for some well known apps, but cannot state which it is, cos of NDA.
Pretty familiar with software testing, but I cannot say much more about this.
I wish I had not stopped developing software, which was once my day job a long time ago, but its too late to go back to that. Life is too short. I do still code here and there but nothing major, just for fun., and only for myself, on very rare occasions, where I really cannot find an app that does what I need. Much better to reuse, and use software written by others.
1 Like
Hello CarVac -
One more thing (maybe I will change my username to OneMoreThing) !!
Is it possible to allow the end user to enter values for things like White Balance. Using only the slider, it is impossible to set an exact White Balance like 5500. If I could enter this directly, that would be ideal, as well as having the option to use a slider.
2 Likes
@CarVac I spent some time tonight with Filmulator. You really have done a nice job here. I used it on my Pixel 3 dng files. Despite all the hype I don’t really like the jpg files. On the phone they can look not bad but the skin tones are funny sort of pale and they crank up the blue so the images are too cold but I think it works with their tone mapping to give the impression of more contrast…In any case I have not yet mastered the controls but even so without too much fiddling and truly knowing the ins and outs I got some nice images as I would rate them.
For me the noise reduction will be a nice add but 2 things I really miss are something like ctrl-z. I was trying to be systematic and I went to tweak the shadows slider from the default value. I didn’t take note so when I decided to revert my change I could only guess or reset to the "hard " default of 250 not the one determined when the file was processed. I think that would be a nice add. If you had the ctrl z ctl y pair you could go back and forth is you wanted. I am not sure how hard that is to code so I don’t know but I think there is good value.
On that same note the thing that I really miss is some form of comparison to the starting point, in other words toggle off the edit and back on or a split screen / snapshot type comparison. Some form would be extremely useful.
Great work. I could see running a few editing sessions and seeing how often I find the need to do more. Thx
PS I second the call for adding manual numbers its nice sometimes but again I don’t have to write the code for it so I don’t know how much effort there is for each ask…
1 Like
Hidden trick in the documentation: right-click on the name for any tool with a continuous slider, and you get an iPod-style scroll wheel for more precise input. Great for fine tweaking.
I think I can still add the ability to type in numbers, but not having thought about it too hard, it might be more complex than it appears at first.
The current behavior of the settings values was intentionally “reset to what it was when the settings were last saved” rather than to the default. Which, now that I check, is clearly in violation of what the tooltips say it does. So I’ll fix that, but I’ll also add a single “reset all settings” button to the top left so that no matter if you’ve set it to 4x scale, you can still rescue it by pressing that button (probably twice) and then restarting the application.
I appreciate your receptiveness. After a good nights rest, a quick one.
I hope its ok to be open, and state that as part of any tests or review, I am obviously comparing Filmulator with other tools. Winter is only winter, because we have other seasons like summer.
I noticed that the pixel size changes from the original.
A original raw file captured in Sony Nex-F3 digital camera as a 4912 x 3264 image, becomes saved by Filmulator, in both tiff and jpg formats as a 4920 x 3276 pixel image.
But Filmulator is not alone in this “slight” alteration of the aspect ratio.
I had always felt that there was something not quite right about the portraits which I processed in darktable, and now I can clearly see why, from an objective opinion, the same image, and in fact all my images from this camera are saved by darktable as 4928 x 3276.
The in-camera jpgs are exactly the same pixel width and height as the raw image. And when I check the output of Capture One 21, which is free to download (but needs logging in to get an account/license) it also generates images in jpg at the same size as the raw image.
All knowledge is positive. But this makes it more difficult to directly appreciate any differences between the “colour science” of the different apps, cos as you switch from one image to another in the image viewer, you eyes also have to adjust to this instantly modified aspect ratio.
This is a pretty interesting observation, and in all the brief discussions I have been priviledged to read online, or watch on youtube, this issue has not ben raised with respect to darktable, which has already a good number of users.
When I go into the various photo raw tools, and check the image proprties in the tools, each of them displays a corresponding set of width and height pixels that corresponds to their jpg and tiff outputs/exports. So there is consistency between what the app “thinks” and what it generates, but there is no consistency between the apps, and the two open source apps that I have checked - Filmulator and Darktable, deviate, in different ways, from the original image pixel size, which is adhered to and not altered by the three other apps I have tested with - Adobe Photoshop Express, Capture One Express for Sony, and Sony’s own Imaging Edge app - all these closed source apps are free to download.
Clearly as the open source photo apps rely on some common libraries which are most likely not created by the developers, but reused, I deduce that the this deviation is coming from the image processing libraries, e.g libraw or rawspeed.
Cosmetically - not a big deal, but if one had a forensic kind of use case, or different human editors working on a large pool of images taken on the same camera, but using different photo editing tools, this slightly varying pixel size results, would be something one would wish to avoid.
I appreciate that it may not be something you can do anything about, cos you most likely did not develop all of the raw-image library which you use, but it gets even more interesting.
I also checked with Raw Therapee only because it is supposed to use the same image processing library for extracting the image from the raw file, as Filmulator, and in Raw Therapee it gets it absolutely right, and maintains the identical pixel size as the original raw file. So it is possible for an open source app to get this absolutely spot on.
But we go one step further, cos when I look at the exif data via Raw Therapee this deviates from the original size to 4928 x 3276 neverthelesss Raw Therapee still correctly displays in the image viewer, and generates jpgs in the original pizel size of the original raw file, using some other method (could be there are two aspect definitions in the exit data - I have not checked the exif using other tools) or a companion database of “corrections” which RawTherapee uses. !!
So we use an Exif viewer and the reasons for some of the above, become a bit clearer or easier to understand.
I used Exif Tool Gui, which relies behind the scenes on some other open source console app. Much easier to use a GUI.
In the exif, we have different values.
- ImageWidth 4928
- ImageHeight 3276
then we have another set of values
- SonyImageWidth 4912
- SonyImageHeight 3264
So it appears that the correct info is there, but each app may have a choice in how it decides to use their image raw library to determine the size of the image.
From an accuracy prspective, I do not think there should be any discrepancy or difference between apps, on something as objectively accurate as the pixel height and width. And if the app were to deviate from the exact size of the original raw image, for whatever reasons, it would be good to let the user know or give them a choice.
Especially when taking photos of products, and people, you definitely want to get the aspect ratio absolutely right, otherwise, you have inadvertently distorted their image, irrepairably.
I tell you no lie, a few weeks ago, I was uncomfortable about the image of my wife, whom I obviously know well enough, as I processed some portrait shots in darktable, just did not look right, now I can appreciate one good reason for this. The image had been “distorted” and I had absolutely no clue that this had been done, cos I trusted the app.
I am sure this “bug” is something you can fix in the next release. Best wishes.
I have done, within a few minutes, this morning an extensive comparison of the images generated by different image processors, but this will need a thread of its own, and it may take me about a month or two to find the time to present my thoughts, in a way that allows others to make their own objective and subjective conclusions. That should be another pretty “revealing” thread. As we are speaking of images, the pun(joke) on revealing is intentional.
I hope everyone would agree with me, imaging apps, should not make their own mind up about the pixel size of the image, without the permission or consent of or explicit notification to the end user…! Hopefully unlike the subjective issue of color science, we can all objectively agree that pixels should be presented with the utmost accuracy size wise, especially in a digital domain where this accurate information is available in the raw file.
Nothing gets distorted by having these different pixel counts.
The only difference between them is cropping at the edges.
And no, the information is not available in the raw file. Filmulator uses whatever LibRaw gives it. RawTherapee has a database of pixel sizes in camconst.json that it uses. darktable uses its own way to determine what’s valid image data.
@OK1 As I also have commented in the other thread, you are mistaken about what’s going on. @CarVac is correct in saying that the image is not distorted, just cropped slightly differently.
And I refuse to believe you are able to distinguish between three images that differ in aspect ratio only to the third decimal place. 
(1.504901961, 1.501831502 and 1.504273504).
1 Like
I believe this is common that a small portion of the theoretical pixels around the border are not processed for purposes of noise of some other reason but I am sure that this can be part of the process…
EDIT From here Developing a RAW photo file 'by hand' - Part 1
The photo we will develop was taken with a Nikon D7000 using this Sigma 17-50mm f/2.8 lens. The aperture was f/11, with 1/200s of exposition time, ISO 100, handheld, and with Vibration Reduction (or Optical Stabilization in Sigma terms) turned on. The target is directly illuminated by the sun at 10:22 AM. The raw file format is “14-bit Lossless Compressed Raw” and is named _ODL9241.NEF with 19.1 MB of size.
The picture has “formally” 4,928 x 3,264 pixels. We say formally because physically we will find raw data for an image of 4,948 x 3,280 pixels. Probably, this is because the pixels close to the borders have not enough neighbors to make the interpolation, so 10 pixels at both vertical ends and 8 pixels at both horizontal ends do not count for the final picture when using regular development software.
This does help improve my understanding. Thanks.
1 Like