New filter `Colors / Transfer Colors [Multi]`

[David_Tschumperle (G'MIC staff)]

Hi everyone,

I wanted to write a specific post to describe you a new filter I’ve added to the G’MIC-Qt plug-in, for the next major release (4.0.0) of G’MIC.
Let’s get right to it: It’s the Colors / Transfer Colors [Multi] filter.

Context

This filter lets you transfer colors from a “style” image to one (or more) “content” image(s) to be processed.

This filter therefore requires at least two input images (placed on two different layers) to work : “style” and at least one “content”. Let say we use these two images for instance:

Here I will use the strawberries image as the “style” image, because it exhibits a limited set of different main tones (green, red, gray, and a bit of yellow), so that’s a good way to see if the color transfer algorithm is able to correctly map any color from the content image into one of these different style colors.
With these two images selected, the filter appears as this, in the G’MIC-Qt plugin interface:

The proposed set of filter parameters allow to choose which layer is considered to be the “style” layer (top or bottom layer), which transfer algorithm you want to use (Histogram-based, PCA-based or Variational Registration), and a few other shared parameters between all these methods, like brightness preservation, number of steps for spatial regularization, and so on. It also allows to preview different things (like color distribution in the 3D RGB cube, see below).

Unsupervised color transfer

The three color transfer methods proposed in this filter are :

• Channelwise Histogram Matching, as its name suggests, tries to modify the “content” image so that each of is R,G,B histograms match that of the corresponding color channels in the “style” image. Here’s what it gives with our current example:
• PCA Matching modifies the content image so that the co-variance matrix of the joint R,G,B distribution matches that of the “style” image.
• Finally, Variational Registration tries to directly align the shapes/colors of the 3D color distributions, using a (quite smart ) multi-scale shape registration algorithm performed directly in the 3D RGB cube.

Actually, I’ve spend most of my time these three latest weeks to finalize this variational registration algorithm for color transfer, implement it to make it work in G’MIC, and write a descriptive paper about it.

Here is a comparison of the color-transfer results obtained with these three different methods, along with a 3D view of the color distributions of the respective images:

With this particular pair of style-content images, it is clear that the variational method works best and create a color-transferred image whose color distribution is closer to the “style” image.
This is actually a quite “extreme” example: here all methods return very different color-transferred results, but it’s not always that contrasted (see more examples below).

As I said, I’ve started writing a scientific report with some dear colleagues, describing precisely how this “variational” algorithm is designed and how it is implemented.
It draws on a lot of what I’ve been able to experiment with in image processing over the past 20 years (in particular, the knowledge I have gained in designing variational methods and solving PDEs in discretized 2D or 3D spaces), so I have to say I’m pretty pleased to have come up with something that’s been neatly implemented in G’MIC and runs relatively quickly if you have enough CPU cores (the variational registration algorithm scales quite well to parallel processing), and that does not rely on some machine learning approaches (so the algorithm itself relies only on a few numbers of parameters).
Believe me, setting all of this up from scratch was quite a job!

Technically speaking, developing this algorithm had also interesting side effects on G’MIC, since it led me to completely rewrite the displacement command, which estimates a 2D or 3D displacement field between two images.

Here I present a few results directly extracted from our scientific article (we are currently finalizing), to give you an overview of how this new algorithm performs and how it compares to other modern color transfer techniques (including a deep-learning-based method, Modflows). In the figures below, PCA, IDT, ROT and Modflows designates different color transfer methods we compare our variational method with (displayed on last column).

Overall, we’re pretty happy with what this color transfer filter can do, all completely automatically and in just a few seconds (sometimes less depending on your CPU! ).
And the fact that it will be freely available in the next version 4.0.0 of G’MIC is a good thing: we hope digital artists will be able to do some cool things with it!

Guided transfer

Another interesting point of our variational algorithm is the fact it can be somehow guided by the user. Indeed, sometimes the color transfer algorithm assigns a color from the “style” image to a color in the “content” image that is not the one we intended.
Here’s a somewhat extreme example: Imagine you have a pair of images of cars:

and you want to transfer the colors of the Formula 1 image to the image of the Beetle.
What you get with the automatic result in this case is actually not that different from the input “content” image itself :

That’s because most of the colors in the Beetle image are also present in the Formula 1 image, so the algorithm does not have to perform complicated color pairing for the transfer
(in this case, it basically find a nearest color that approximately matches).
For instance, The color red can be seen on poppies or on the road markings in the Formula 1 image. In that case, why would the color transfer algorithm choose a color other than this red to perform the transfer?

Our new variational algorithm for color transfer allows you to manually define color correspondences so that you can “guide” the color transfer with additional (hard) color matching constraints.
For the Beetle → Formula 1 pair, here is typically how you would do that using our G’MIC-Qt filter, setting a value different of 0 to the Color Correspondences parameter:

From a theoretical perspective, these types of color-matching constraints fit naturally into our variational formulation and lead to a transfer function that forces the selected colors (and their neighbors) to map to the specified colors in the style image.

Again, I propose you some excerpts from our paper that illustrate what guided color transfer can do if you want to enforce certain color matches.

Isn’t that just really cool?

Limitations

Of course, since an algorithm is never perfect, there are sometimes situations where color transfer doesn’t work as intended, even with user guidance. This is typically the case when attempting “contrast inversion,” such as assigning a light color to a color that was originally dark, and vice versa.
Also, the result of the algorithm ultimately generates a 3D color LUT for the color transfer, so that two points of the same color in the “content” image will always be mapped to the same color in the transferred image.
Since our algorithm does not rely on neural networks, it is not inherently capable of “understanding” the objects in the images or assigning them a “semantic” meaning that could be used to somehow guide color transfer.
On the other hand, this allows us to have a fast algorithm with few parameters that runs perfectly well on the CPU alone.

I hope you find this filter interesting and that it will be useful to you in your image processing tasks.

Cheers !

[Mushy (Martin)]

Excellent filter! Sounds like you put a great deal of effort into this one! Cool…
I’m using a tablet with just 2GB RAM, so it going to be slow, I tried defining a couple of source/target colours in the command (using the CLI), and with c. 1700x1000px + source and target with the Variational Registration transfer type it took only 12 seconds:

Source:

Target:

Variational Registration result:

Also tried the PCA Matching type, the result was amazing too, but took a lot longer (a minute or more…) but like I say this was from the command-line on a low-spec Android tablet!

Thanks for furnishing us users with another great tool. David - this is a great filter, I will certainly be trying it out some more too, but first impressions >

[afre (Alan)]

Very promising. From your results, the new method preserves the colour structure considerably better.

@Mushy’s may be lacking compared with the PCA variant because Variation needs sensible user guidance in that case.

[Reptorian]

I can see myself using this in conjunction with “Luminosity” Blending Mode.

[Soupy (Tim)]

Great tool. I can see this being useful to emulate film or painting colours.

[Mushy (Martin)]

@afre

…@Mushy’s may be lacking compared with the PCA variant because Variation needs sensible user guidance in that case.

Yes, with the first of my results I only tweaked a couple of colours, mostly because I was trying it out quickly, and to see whether it made any difference with using the default parameters or not.
My first command, employing Variational Registration was:

gmic v 0 2887.jpg 2427s.jpg fx_transfer_colors 0,2,0,0,35,2,0,0,0,0,256,0.01,1,1,0,0,0,0,0,0,0,255,255,255,255,255,255,255,0,0,255,0,0,0,255,0,0,255,0,0,0,255,0,0,255,255,255,0,255,255,0,87,72,59,49,64,72,74,71,65,16,31,48 k[1] o tc_5.png

Every parameter value except the last 2 RGB Source/Target values are default I think, so maybe better results would be had by encoding as many colour values to transfer as possible.
The same command but specifying PCA Matching was used for my second result above.
I have tried the Channelwise Histogram Matching option but I get the error:

[gmic] *** Error in ./fx_transfer_colors/_fx_transfer_colors/*if/*foreach/transfer_histogram/ *** Command 'transfer_histogram': Invalid argument '.,0,srgb'.
[gmic] Command 'transfer_histogram' has the following description:

  transfer_histogram:
      [style],_nb_levels>0,_color_channels

    Transfer channel-wise histograms of the specified style image to selected images.
    Argument 'color channels' is the same as with command 'apply_channels'.

    Default value: 'nb_levels=256' and 'color_channels=all'.

    Example:
      [#1] image.jpg 100,100,1,3,"u([256,200,100])" +transfer_histogram[0] [1]

Not sure what to make of that though.

[afre (Alan)]

@Mushy

Not sure what to make of that though.

I have not tested it… Have you tried not using the . shorthand, using [#] or # instead? Replace # with the actual index number.