RawTherapee and Pentax Pixel Shift


(system) #1

RawTherapee and Pentax Pixel Shift

Supporting multi-file raw formats

What is Pixel Shift?

Modern digital sensors (with a few exceptions) use an arrangement of RGB filters over a square grid of photosites. For a given 2x2 square of photosites the filters are designed to allow two green, and one each red and blue colors through to the photosite. These are arranged on a grid:

Bayer pattern on sensor

The pattern is known as a Bayer pattern (after the creator Bryce Bayer of Eastman Kodak). The resulting pattern shows how each RGB is offset into the grid.

Bayer pattern on sensor profile

Each of the pixel sites captures a single color. In order to produce a full color representation at each pixel, the other color values need to be interpolated from the surrounding grid. This interpolation and methods for calculating it are referred to as demosaicing. The methods for accomplishing this vary across different algorithms.

Bayer Interpolation Example
The final RGB value for the initially Red pixel needs to be interpolated from the surrounding Blue and Green pixels.

Unfortunately, this can often result in problems. There can be chromatic aliasing problems resulting in odd color fringing and roughness on edges or a loss of detail and sharpness.

Pixel Shift

Pentax‘s Pixel Shift (Available on the K-1, K-3 II, KP, K-70) attempts to alleviate some of these problems through a novel approach of capturing four images quickly in succession and by moving the entire camera sensor a single pixel for each shot. This has the effect of capturing a full RGB value at each pixel location:

Pixel Shift shifts the sensor by one pixel in each direction to be able to generate a full set of RGB values at each photosite.

This means a full RGB value for a pixel location can be created without having to interpolate from neighboring values.

Advantages

Less Noise

If you look carefully at the Bayer pattern, you’ll notice that when shifting to adjacent pixels there will always be two green values captured per pixel. The average of these green values helps to suppress noise that may have been interpolated and spread through a normal, single-shot raw file.

Pixel Shift Noise Reduction Example
Top: single raw frame, Bottom: Pixel Shift

Less Moiré

Avoiding the interpolation of pixel colors from surrounding photosites helps to reduce the appearance of Moiré in the final result:

Pixel Shift Moiré Reduction Example
Top: single raw frame, Bottom: Pixel Shift

Increased Resolution

This method is similar in concept to what was previously seen when Olympus announced their “High Resolution” mode for the OMD E-M5mkII camera (or manually as we previously described in this blog post). In that case they combine 8 frames moved by sub-pixel amounts to increase the overall resolution. The difference here is that Olympus generates a single, combined raw file from the results, while Pixel Shift gets you access to each of the four raw files before they’re combined.

In each case, a higher resolution image can be created from the results:

Pixel Shift Increased Resolution Example
Top: single raw frame, Bottom: Pixel Shift

Disadvantages

Movement

As with most approaches for capturing multiple images and combining them, a particularly problematic area is when there are objects in motion between the frames being captured. This is a common problem when stitching panoramic photography, when creating image stacks for noise reduction, and when combining images using methods such as Pixel Shift.

Although…

The RawTherapee Approach

Simply combining four static frames together is really trivial, and is something that all the other Pixel Shift-capable software can do without issue. The real world is not often so accommodating as a studio setup, and that is where the recent work done by @Ingo and @Ilias on RawTherapee really begins to shine.

What they’ve been working on in RawTherapee is to improve the detection of movement in a scene. There are several types of movement possible:

  • Objects showing at different places in a scene such as fast moving cars.
  • Partly moving objects like foliage in the wind.
  • Moving objects reflecting light onto static objects in the scene
  • Changing illumination conditions such as long exposures at sunset.

All of these types of movement need to be detected to avoid the artifacts they may cause in the final shot.

One of the key features of Pixel Shift movement detection in RawTherapee is that it allows you to show the movement mask, so you get feedback on which regions of the image are detected as movement and which are static. For the regions with movement RawTherapee will then use the demosaiced frame of your choice to fill it in, and for regions without movement it will use the Pixel Shift combined image with more detail and less noise.

Unique to RawTherapee is the option to export the resulting motion mask
(for those that may want to do further blending/processing manually).

The accuracy of movement detection in RawTherapee leads to much better handling of motion artifacts that works well in places where proprietary solutions fall short. For most cases the Automatic motion correction mode works well, but you can also fine tune the parameters in custom mode to correctly detect motion in high ISO shots.

Besides being the only option (barring dcrawps possibly) to process Pixel Shift files in Linux, RawTherapee has some other neat options that aren’t found in other solutions. One of them is the ability to export the actual movement mask separate from the image. This will let users generate separate outputs from RT, and to combine them later using the movement mask. Another option is the ability to choose which of the other frames to use for filling in the movement areas on the image.

Pixel Shift Support in Other Software

Pentax’s own Digital Camera Utility (a rebranded version of SilkyPix) naturally supports Pixel Shift, but as with most vendor-bundled software it can be slow, unwieldy, and a little buggy sometimes. Having said that, the results do look good, and at least the “Motion Correction” is able to be utilized with this software.

Adobe Camera Raw (ACR) got support for Pixel Shift files in version 9.5.1 (but doesn’t utilize the “Motion Correction”). In fact, ACR didn’t have support at the time that DPReview.com looked at the feature last year, causing them to retract the article and re-post when they had a chance to use a version of ACR with support.

A recent look at Pixel Shift processing over at DPReview.com showed some interesting results.

Sample Image Raw
The image used in the DPReview article. ©Chris M Williams

We’re going to look at some 100% crops from that article and compare them to the results available using RawTherapee (the latest development version, to be released as 5.1 in April). The RawTherapee versions were set to the most neutral settings with only an exposure adjustment to match other samples better.

Looking first at an area of foliage with motion, the places where there are issues becomes apparent.

For reference, here is the Adobe Camera Raw (ACR) version of a single frame from a Pixel Shift file:

Pixel Shift Comparison #1

The results with Pixel Shift on, and motion correction on, from straight-out-of-camera (SOOC), Adobe Camera Raw (ACR), SilkyPix, and RawTherapee (RT) are decidedly mixed. In all but the RT version, there’s a very clear problem with effective blending and masking of the frames in areas with motion:

Pixel Shift Comparison #2

Things look much worse for Adobe Camera Raw when looking at high-motion areas like the water spray at the foot of the waterfall, though SilkyPix does a much better job here.

The ACR version of a single frame for reference:

Pixel Shift Comparison #2

Both the SOOC and SilkyPix versions handle all of the movement well here. RawTherapee also does a great job blending the frames despite all of the movement. Adobe Camera Raw is not doing well at all…

Pixel Shift Comparison #2

Finally, in a frame full of movement, such as the surface of the water.

The ACR version of a single frame for reference:

Pixel Shift Comparison #3

In a frame full of movement the SOOC, ACR, and SilkyPix processing all struggle to combine a clean set of frames. They exhibit a pixel pattern from the processing, and the ACR version begins to introduce odd colors:

Pixel Shift Comparison #3

As mentioned earlier, a unique feature of RawTherapee is the ability to show the motion mask. Here is an example of the motion mask for this image

Pixel Shift Motion Mask
The motion mask generated by RawTherapee for the above image.

Also worth mentioning is the “Smooth Transitions” feature in RawTherapee. When there are regions with and without motion, the regions with motion are masked and filled in with data from a demosaiced frame of your choice. The other regions are taken from the Pixel Shift combined image. This can occasionally lead to harsh transitions between the two.

For instance, a transition as processed in SilkyPix:

Pixel Shift Transition SilkyPix

RawTherapee’s “Smooth Transitions” feature does a much better job handling the transition:

Pixel Shift Transition RawTherapee

In Conclusion

In another example of the power and community of Free/Libre and Open Source Software we have a great enhancement to a project based on feedback and input from the users. In this case, it all started with a post on the RawTherapee forums.

Thanks to the hard work of @Ingo and @Ilias Pentax shooters now have a Pixel Shift capable software that is not only FLOSS but also produces better results than the proprietary solutions!

Not so coincidentally, community member @nosle gave permission to use one of his PS files for everyone to try processing on the Play pixelshift thread. If you’d like to practice consider heading over to get his file and feedback from others!

Pixel Shift is currently in the development branch of RawTherapee and is slated for release with version 5.1.


This is a companion discussion topic for the original entry at https://pixls.us/articles/rawtherapee-and-pentax-pixel-shift/

(Ronald E Chambers) #2

The images here are terrible. I will reprocess if you contact me.
RONC


(Pat David) #3

In what way? You’re invited to share some new ones with us! :smiley:


(Ronald E Chambers) #4

Some of them have compression artifacts that are really bad. Resolution
one needs better one. I’m sending Ingo one where I shot handheld Pixel
Shift and it needs COMPENSATION badly. I have resolution I need to locate
and will send to Ingo.
Ingo is sending me file name and locations.

Good hearing from you. Don’t hesitate hollering my way.
Regards,
RONC


(Pat David) #5

I feel you’re looking at them here on the forum, you’re seeing a compressed version from the forum software. For articles and blog posts you should look at the actual website post:

Hopefully that looks better! :slight_smile:


#6

If any of the examples needs replacing, the zoomed-in image of the bricks is the one. It’s somewhat out of focus, so it’s extremely hard to tell the difference between the non-pixel-shift and the pixel shift one.


(Morgan Hardwood) #7

Agreed that the sample images could be improved:

  • The histograms could be stretched to go from black to white, especially in the “less noise”, “mess moire” and “hard transition” examples,
  • the difference in the brick wall is hard to see on a smartphone,
  • and the comparison using leaves could be tweaked using curves so that the RT version matches SilkyPix more. I expect readers of this article who are unfamiliar with RT to think that the flat tones show a problem with the software when in fact it’s a feature they’re unaware of - that RT can show the actual neutral image with no behind-your-back tweaks as most other programs do.

I can help if you like.

It would probably also be good to split these comments from the actual article.


(Ingo Weyrich) #8

Yes, please :slight_smile:


(Morgan Hardwood) #9

I will need the raw files.