I trained a small neural network for demosaicing on X-Trans sensors. On my dataset, it quickly outperforms Markesteijn (the OpenCL implementation from darktable). The model was trained on a Mac, and inference takes about 0.6 seconds for a 40MP image.
I believe the model can be further improved with more data and synthetic inputs. It could also be extended to jointly perform demosaicing and denoising.
Checkpoint takes ~630Kb for float32 precision.
I am not sure what to do with this model. I was thinking to either make a pull request to integrate it to darktable or rawtherapee. Or make a standalone native mac app for raw processing (x-trans demosaic was actually the most difficult part).
Yet, this week Apple announced new native raw processing api, so not sure that open source app is still a good idea.
I don’t know if you’ll find many people interested in using it, u less it gets integrated into a well used raw editor.
Most of the Fuji owners are not pixel peeping, especially with the current generation of sensors. I was interested when I had my X-T1, but not since I’ve upgraded.
really interesting, seems to me there is a nice improvement with false colors. for now it seems a bit difficult for me to try, i hope it will become an option for art-rawtherapee-darktable and free software in general. Does it also take care of chromatic aberrations that currently are not autocorrected with xtrans in free software?
This would be nice to have in darktable. Especially since the checkpoint is so small, can be locally trained, etc, seems like a no-brainer and easy to implement. Neural network demosaicers+denoisers seem to be the future and a lot of commercial software are using them as well by now.
@hatsnp I saw it, but haven’t looked into it deeply yet. Thanks for sharing!
I will try to find some time and train a joint demosaic and denoise model for both x-trans and bayer.
Also, I checked apple’s new raw processing and I don’t like the plastic feel it gives.
I have started a similar project and written about it in this post.
The idea is to train using pytorch and then run it on Rust wgpu shaders.
I found your post here and I like your approach. I was hoping to pick your brain a bit when it comes to neural architectures for demosaicing.
Would you mind sharing if you have done any more experimenting since your post and maybe give some thoughts around how you arrived at the architecture you used?
Do you have any general recommendations to me and others and have you given any thoughts to extending your architecture to include different CFA patterns?
