Cinema DNG Raw compatibility Update / Potential help

Software RawTherapee
1

I know that this issue has been raised before, but I would like an update with the status of fix, and to help with the fix of the Cinema DNG magenta noise raw misinterpetation problem that has been previously documented, if need be. Moderaters, please do not delete this as a duplicate as I have some new information / potential help I could bring.

First of all, I need to give a shout out to the developers of Rawtherapee (Skip this paragraph if you want to cut to the point). Between RL deconvolution, Amaze demosaicing, wavelets, and false color supression filter, I prefer the results from Rawtherapee for raw processing raw video over Davinci Resolveā€™s raw engine, based on sharpness of result, minimizing aliasing and moire, and subjective feel for footage from the Blackmagic Pocket Cinema Camera. I am able to get good results despite pushing the BMPCC files hard, and if there is issues with maze artifacts (fault of mediocre sensor), I can simply solve the issue with vng4 demosaicing. So the fact that I care to want to figure out a serious video workflow with Rawtherapee (previous discoveries from Adobe DNG converted tests), is a HUGE compliment to the Rawtherapee dev team, for features listed above and my positive experience with Rawtherapee stability.

Now, I have been using Adobe DNG converter for tests and my first video shot featuring a friend with the BMPCC (still not finished editing yet), but this is unacceptable for serious video. First of all, it is an extra cumbersome and time consuming step for what is already a borderline impractical workflow, from file management, to the time consumed in processing the conversions. Second, the original files are useful in terms of metadata that Resolve uses, such as color science, and keeping both when in the middle of a video project that would require use of both raw processing engines would be excessively cumbersome and hard drive intensive.

Of course, the biggest issue with Adobe DNG converted files are that they have any highlights near clipping has a strong magenta cast that renders the Rawtherapee HL recovery capabilities useless. Yes, I did confirm with original and converted DNG files loaded to Resolve that the issue is Adobeā€™s fault, not that of the BMPCC or Rawtherapee.

I am a university student that would be willing to try to find a computer science major to volunteer, as well as friends who may have coder professional networks that might be useful, but I would like to know some details to help this go smoothly.

Is this issue already fixed in a development version of Rawtherapee that someone knows about?
Have any developers already started working on the issue?
How much work is likely needing to be done (Keep in mind I donā€™t understand programming myself)?
What online technical resources should I send the person so that he/she could work on it, as well as contacts needed?
Is there any specific programming languages the person have to be versed in, or is the task something any photo knowledgeable programmer would be able to solve?

Keep in mind that I havenā€™t found anyone to commit yet, so having a sense of how much work likely needed is helpful. Also, let me know if there is any way I can help. I have already uploaded a sample CDNG file from the BMPCC to the Rawtherapee raw upload , and but can do it again, or do a Google Drive link. Any other things I can do to help solve this issue?

Edit: Had a programmer friend that I thought would be doing it, but just found out his knowledge is decades out of date, so transitioning to references to a male friend to gender neutral language, and a general volunteering to recruit ANY programmer to fix this.

2

Hello & welcome!

We donā€™t really do thisā€¦ this isnā€™t reddit :stuck_out_tongue_winking_eye:

You probably want to keep an eye on this issues in the RT bug tracker: Support for JPEG-compressed DNG files Ā· Issue #4285 Ā· Beep6581/RawTherapee Ā· GitHub ā€“ looks like it is still open.

If you can bring coding resources, that would be sublime, the code is open so people are free to get it and hack away!

1 Like
3

I am not a coder myself, and am primarily trying to get a sense of how much work is involved and recruit someone who can commit to a project of said size, and looking for info on the kind of background knowledge, programming languages, ect is needed to be able to do this, as computer science can be very diverse.

4

Youā€™d need to know C++ and the dcraw package, as well as have knowledge about image decoding, and DNG files.

1 Like
5

Thanks for the coding related required knowledge for my recruiting. Do you have a time commitment estimate for the average coder who knows these things?

6

Thatā€™d be heavily related to exactly how much they know and how much they have to figure out.

7

Also, if someone is strong on C++ and knows basic theoreticals of image decoding, but is unfamiliar with dcraw (correct me if Iā€™m wrong, but dcraw sounds like something pretty niche to the FOSS photography software world), would it be pretty easy for them to pick up understanding of dcraw?

Thanks for the specifity needed for me to start recruiting.

8

Iā€™d think picking up dcraw would be the same level of difficulty as getting into any unfamiliar code base. Keep in mind that this person doesnā€™t have to do it alone, there are several chat rooms for developers and quite a few people could provide guidance.

9

What Iā€™ve found regarding this is, if the highlights are exposed at and above the sensor saturation point, the data stacks up there, as illustrated in the histogram. When white balance correction is applied, it spreads the three channels appropriately, but the stacked saturated data splits into three peaks, one for each channel. As you then move your data to do, say, positive exposure correction, you move each of those peaks to the display white bound, where they again start to stack up. If the green peak is leftmost, the highlight takes on a magenta cast because itā€™s green component is not at ā€œwhiteā€.

If you increase the exposure correction enough, all three peaks eventually stack up at white and the magenta goes away. The difference between softwares is how aggressive is the automatic scaling for display.

Iā€™ve been dealing with this specific thing in coming up with a batch workflow. For that, I want aggressive scaling so the proof image Iā€™m making in batch looks nice. But if process ā€˜by handā€™, I want to see it happen and deal with it manually.

10

At first, I thought it was an issue like you are describing, first the green channel clips, and then the green channel is lowered in white balance, yielding a magenta cast, but then I observed that the issue only occurs for the Adobe DNG converter files, not the original files when both are compared side by side and developed by Davinci Resolve.

11

Actually, I think itā€™s the green channel that clips last.

Does the Adobe DNG converter have a scale or black-white point setting you can alter?

12

I made some progress recently. You can follow up here

13

I just merged Blackmagic and Canon Magic Lantern Cinema DNG support into dev.

14

Here are some overexposure tests for white levels of the BMPCC, as well as some documentation of technical issues with the BMPCC that might complicate the issue (or might not, this is not my expertise as I am doing this at the directive of a developer).

Here are the the raw files for white point calibration, shot with a tungsten light source for smooth spectrum lightsource, filming while moving the light closer and further away to change exposure. I shined it in a way to make a slight gradient in light level across the white paper I was shooting for this test. The subject was out of focus, so that any fine detail could be attributable to camera sensor and image pipeline artifacts.

Although the request is for one 1-2 stop overexposed raw file, I decided to give you multiple, because I noticed a few issues that might complicate figuring out white points. First, I noticed a weird fixed pattern grain in highlights very close to or at clipping (not 100% shure as I donā€™t have Rawdigger, the most accurate way of assessing), and it is observable even when it is safe to say the files are being correctly decoded (camera original files pixel peeped in Davinci Resolve). Note this is not an issue that warrants trying to do lots of trouble shooting to fix, as it is only visible at 100% or more with excessive sharpening, and is likely pretty easy to fix in most pro NLE video editors. I also noticed that the rgb channel levels, according to the waveforms in Davinci Resolve (Again, disclaimer, this is not Raw Digger), seem to reach a saturation point (Example file labeled 4 Saturation light.dng), with an observable variance of the levels, depending on the pixel. As the overexposure rises further, the flat lines of levels rise just a little bit, as well as getting just a little less variance in values (5 Saturation Heavy.dng).

https://drive.google.com/open?id=11gbxOLWAj7mMo5AXGX9VpcgFfGADXUM3

I also posted this on the Github issue.

15

Using demosaic ā€˜Noneā€™ in RawTherapee you can see the raw values (after black-level subtraction) here:

grafik
grafik1920Ɨ1080 202 KB