Computational DOF and Bokeh

The Brenizer method sucks. You need to have a static subject and a static background and some time to shoot your frames array. Then you need to stitch those pictures. Then you loose a lot of pixels and storage space.

The results are sometimes nice, but it’s an awful workflow and you really need to be lucky with the right conditions. It’s really random.

Don’t get me wrong, it’s a clever hack, but I won’t advertise it as a sensible workflow and bend a software to speed it up.

3 Likes

Yeah, I know about that and it’s great for static subjects but not so much for live events.

The thing that prompted this post is that I’ve seen our countries presidential candidates tweeting photos from their events that were captured by some 3rd person (campaign leader) of them giving speeches and stuff.

And it pains me to admit it, but with all the flaws it looked awesome (or good enough).
So if it’s good for our current president why wouldn’t it be good for me to use some sort of fake DoF.

The thing is, those are all moving subjects :confused:

1 Like

I’ll check the paper, thanks! until then, I remain skeptical

I thought about this too an see at the moment different (theoretical) possible ways to get depth maps.

  1. Create a lens with a prism and sprit the image into 9 images.
    Pro:
    A prototype is ready for tests: https://www.k-lens.de/
    @houz works for K-Lens and the software they use looks very familiarly :wink: : https://www.k-lens.de/en/produkt-02#&gid=1125898543&pid=3
    Con:
    The image size is reduced. (K-Lens expect to get ~2/3 of the original image size.)
    You need this extra lens.
  2. Use the pixel shift that some cameras use.
    Pro: The Hardware is ready
    Con: I have no clue if it works for creating depth maps
  3. Adding a LiDAR to the camera like a flash:
    Pro: LiDARs are getting much cheaper at the moment because they are needed for autonomy driving. (https://www.blickfeld.com/ https://www.livoxtech.com or Bosch will Laserradar entwickeln | heise Autos)
    Con: I haven’t heard form someone even trying to use this for photographic.
    Needs synchronisation with the camera
  4. A camera array.
    Pro High image resolution
    Con Needs some kind of synchronisation
    Expensive

I played a little bit with the demo images from k-lens and posted it here:
(The quality of the depth map was not perfect, but it was one of the first test images)

If you are interested in getting the depth map from a Samsung Gallaxy S10. I blogged here the exiftool command how to extract it and import it in GIMP :

If you have a focal length on your camera similar to your phone and shot the same scene with both, could you map the depth from one onto the other?

Also Lidar is well withing financial reach: https://www.amazon.com/MakerFocus-Single-Point-Ranging-Pixhawk-Compatible/dp/B075V5TZRY/ref=mp_s_a_1_3?keywords=lidar+sensor&qid=1578772604&sr=8-3

I think no hardware approach should be taken but the lidar is definitely interesting.

Now, can lidar damage a camera?:

There are some suggestions that, depending on the wavelength, lidar can damage eyes. You can always buy a new camera, but…

A depth map should be able to be produced from Canon’s dual-pixel cameras. Canon has a patent for quad pixel autofocus which would make it even better.

Combined with machine learning I think high quality depth maps and artificial bokeh will be possible.

Ok, I never thought of the fact that most cameras already have pretty accurate depth sensors for AF. Dual pixel improves the image coverage by a mile, but still that data is not available for the user.

The k-lens approach is essentially light field processing, which can do refocusing/artificial bokeh without actually computing a depth map. Of course 3D (ok, 2.5D) imaging is also possible with such a lens.

@Tobias I’m not sure if pixel shift would work, the base line is very small, but that’s just gut feeling.

In theory it also should be possible to calculate a depth map by depth-from-defocus, which requires blur estimation. The current focus detection mode is based on edge detection if I’m not mistaken, but its gradient classification might be a base for blur estimation. I think our local hero @anon41087856 also has fought with deconvolution sharpening at some point, that might also be a way in.

Does anyone know if it is possible to get the AF information with magic lantern? (I have no Canon.)

If you want computational bokeh you need to do it in the camera. Just as the phones use multiple exposures or even the multiple lenses to help calculating the depth map. I think doing it post mortem from a single photo might not get you as good results as the phones.

Apparently it’s already contained in the raw file:
"During Dual Pixel RAW shooting, a single RAW file saves two images into the file. One image consists of the A+B combined image data and the other only the A image data. This means the Dual Pixel RAW files contains both the normal image and also any parallax information, which can be measured and subject distance information extrapolated. As Dual Pixel RAW images contain two images they are therefore double the file size of normal RAW images. "

source:

Dual Pixel RAW

At least in past it was a problem that one raw frame is Image A while the next raw frame is Image A + B, which are encoded using the same bit depth, means that A + B is often clipped and B can not be recovered completely…

As anything else, it’s just a matter of getting the maths right and respect lights transport models.

I’ve just stumbled across a video which illustrates how Dual Pixel AF works.

Basically it takes two slightly shifted/parallaxed images and then calculates the depth of the subject.
Now, I don’t have an EOS R or 5dmk4 but I presume that Dual Pixel RAW saves both images in a raw file.

This could be the vector the industry needs to attack the computational DoF problem.

EDIT: except Canon states that the two images A and B that are shot are not how they are saved. They say that first image is A image data and B image data combined and the second image is just B image data. The thing that worries me is that they are combining both image data in the first image.

Now does this means everything falls apart or can a depth map still be generated from A+B and B image data?

If you know B, and A+B, then you can calculate A = (A+B) - B.

But the A+B image might be lossy-compressed, or something. If you can link to such a raw file, we can take a look.

I’d say it’s not the fault of the reduced depth-of-field cause by wide lens aperture, but the angle of the focal plane (planes perpendicular to the lens axis). It is aligned with her spine, which is not useful for this seated pose. That is not really annoying foreground blur. It is lens-subject misalignment.

I don’t own a camera that can produce Dual Pixel RAW images but a guy that I’m following on twitter has an EOS R so I’ve DMd him if he wouldn’t mind to send us one here. So I’m hopeful that we can get a sample :slight_smile:

That’s the fault in Canons dual-pixel approach. If A is not clipped and B is not clipped, but A+B is clipped, this does not work.

It’s silly because they could have made Dual Pixel Raw contain A and B image and do A+B in Canon EOS Utility or in post in general. Why they decided for this approach blows my mind.
Then again, maybe it’s not an issue at all since we don’t have a sample file yet.