Fast neural demosaic for x-trans

Processing
,
1

I trained a small neural network for demosaicing on X-Trans sensors. On my dataset, it quickly outperforms Markesteijn (the OpenCL implementation from darktable). The model was trained on a Mac, and inference takes about 0.6 seconds for a 40MP image.

Code and a pretrained checkpoint are available on GitHub: GitHub - danylo-kelvich/neural-demosaic: A neural network for X-Trans demosaic · GitHub

I believe the model can be further improved with more data and synthetic inputs. It could also be extended to jointly perform demosaicing and denoising.
Checkpoint takes ~630Kb for float32 precision.

comparison
comparison1160×560 1.04 MB

4 Likes
2

Nice, maybe talk to X-veon: a better demosaic for X-trans (100% less worms!) to combine efforts :slight_smile:

1 Like
3

I saw this work. It actually inspired me to build something similar. There are a few issues with his architecture that I addressed in my model.

1 Like
4

I am not sure what to do with this model. I was thinking to either make a pull request to integrate it to darktable or rawtherapee. Or make a standalone native mac app for raw processing (x-trans demosaic was actually the most difficult part).

Yet, this week Apple announced new native raw processing api, so not sure that open source app is still a good idea.

1 Like
5

I really like the simplicity of this approach. All it takes is 70 raw images and 15 minutes to train it on a macbook.

2 Likes
6

I don’t know if you’ll find many people interested in using it, u less it gets integrated into a well used raw editor.
Most of the Fuji owners are not pixel peeping, especially with the current generation of sensors. I was interested when I had my X-T1, but not since I’ve upgraded.