I am getting this artifact from a Rawsie dng file and I am curious of what it can be. I got two other samples sent to me (Rawsie is only working in OS X) from a R5 but at ISO 1600 and 6400. Much worse because not only the optical black field was affected but also the shadows in the picture.
Rawsie looks to be closed source, and DNGs, ad we know, are a minefield. You should bring this issue with their DNG file up to them.
And of course: please don’t delete your original raw files. They’re artifacts from your camera and should be treated as such. This goes quadruple for closed source software that makes a claim but offers no source so you can see what its doing to your files. Too many horror stories of “oh I used adobe DNG converter but it turns out that years later there was a bug in it and my original raw files are long since deleted.” This rawsie is in the same category.
What do you think is the cause of the pixels in darktable but not dcraw?
I already know the story of the first version of Adobe’s DNG converter that erased optical black area. Or the last version that changed the colour matrix for Leica Q2.
Hi Peter,
I am not addressing your original question but just wondering why you are converting the canon raw file to DNG. Is the R5 file not supported in your editing software?
I don’t own any R5 and I don’t own any computer with OS X. I own a 1Ds that I thought RawTherapee supported, but that is not the question here but in the other thread.
I do like to compare things. For example Canon raw vs craw is easily seen in the optical black field.
What I found this far is that black/white level changes and it also looks strange in clipped highlights. And of course the dots, but that may be a darktable thingy.
unprocessed_raw dump to PGM/TIFF from the LibRaw package also does not show them…
However, reading in Python using tifffile does show some non-zero data there as well.
I now also see LinearizationTable in this file - could it be that RawSpeed is passing the entire buffer through the table which makes these pixels more visible, while dcraw/LibRaw only applies it to the active area? @LebedevRI
It looks like it’s not that LUT application either - I just passed the tifffile uncompressed array incl. the black areas through it and the hot pixels are not there. Here’s a new comparison of
dt (RawSpeed) w/ black level set at 0 and crop removed, demosaic in passthrough:
(and the same can be shown for LibRaw’s unprocessed_raw output)
There’s clearly non-zero data in the black areas in both cases, but hot pixels only w/ RawSpeed. @LebedevRI So potentially something might be up w/ the LJpeg decompressor or elsewhere in the DNG decoder?
Does the issue go away it you completely disable DNG opcode processing?
I’m aware of an, err, confusion as to what the DNG codes should do wrt the crop,
should they apply to the whole image, only the active area (i.e. crop), or whatnot.
“Indeed, we use the lossless JPEG coder already present in the DNG SDK. Coding is done over the native bit-depth of the camera (DNG supports up to 16 bits). The way that we achieve better results is that we calibrate each individual camera, at each iso and most shutter speed, in the lab. From that we generate custom coding per camera and per iso. The quality is not bit-for-bit lossless, but the error generated with respect to the noise is approximately 1/10th of that of lossy JPEG at the same file size.”
Eh, no, i guess that is not it?
I was thinking it was due to the dithering during the LUT application,
but that does not seem to be the case. I think, we are on a wild goose chase.
AFAIU, that “software” is trying to minimize the filesize, by smartly re-encoding
the data, in non-lossless way. But do we know that they are trying to preserve
the whole image, or only the active area of the image? If you are optimizing
for size, it makes little sense to preserve the black areas, so i’m not exactly sure
that those artefacts in the black areas aren’t intentional.
