Looking for samples - smartphone DNGs with embedded GainMap

A text file linked from the main page would probably be sufficient. Or a thread here on discuss where those that decide give the feedback. After submission one could just check every couple of days until there’s feedback (which is recognized by date of submission, camera model (if known), and maybe file name), just an OK or a “rejected because wrong firmware revision - could you update and resubmit?” etc.

It’s more of an issue that it is more workload for those that decide about the rejection, such that they have to balance between workload and benefit. I see no need to code something.

Yes more burden for those already doing. Anyone can do what you’ve suggested. Its admin work though, and that is boring and thankless.

Either I don’t understand, or you. I am no native English speaker, so it’s probably on my side, but I really want to understand your issue with my posts.

To clarify: There is somebody (I think Roman was mentioned earlier) who decides if a submitted raw file is sufficient for the purpose(s) of raw.pixls.us or not. That means, I cannot do this job, as I do not understand based on which criteria to decide. As I wrote, I am not even able to submit a raw that is sufficient, how could I decide? It’s not admin work as far as I understand admin work – it’s more engineering task, understanding the raw file, testing with rawspeed etc.

If you did not get it, I am trying to contribute: I submitted 2 raw files of missing cameras (I don’t even need support for them, I am submitting to the benefit for other people) the same day, one was silently rejected despite there is need for this camera documented on raw.pixls.us. I am willing to resubmit a better sample, but I already followed all advice on the site but probably made some mistake I don’t know. All I wanted to discuss is if it would be beneficial to document rejection reasons such that people that are contributing can do it better/again/whatever.

I mean, there is somebody that already put effort in rejecting the sample I submitted, and the camera is still marked as missing. How could it be bad to report the rejection reason? IMO it would be beneficial for all parties:

  • Public examples of dos and don’ts would improve the sample quality of future submissions → less work for those who check the samples
  • Faster resubmission of missing samples
  • More cameras supported

Only drawback: The people who check the samples (I expect this is a lot of work) additionally would copy-paste the submission metadata to the forum and add one sentence regarding the rejection reason.

Where the heck is my misunderstanding?

The issue is you’re asking for more work from people who are already doing work.

If you’d like, you can watch the submissions and post here or write a text file that we can post somewhere.

But likely other here are in the same situation as you: not enough time, so suggesting more work isn’t that helpful.

I am by no means asking for something. I only highlighted that the approach of no feedback is inefficient and that it’s worth to think about. The person that decides saves 30 seconds for documenting the rejection reason, while the 1 hour it took to make the setup, photograph the color target, check, copy and select the files, upload, etc., are totally lost because I get no feedback what was wrong with my submission.

Plus that I have no chance to upload a better sample. I would love to help out, I have the missing camera (which I personally don’ use for raw), I can upload again, but uploading the rejected files over and over again makes no sense. And, it is probably not the file content which is the issue, as it is a well exposed shot of a color target.

If it stays like this, I am fine, no problem. I just wanted to highlight the issue, I am not the one who decides. And I even get the impression that you do not understand what I am writing, maybe language barrier on my side.

As far as I understand, the submissions are not public, only if they are ACCEPTED they will show up. So what should i watch. I really don’t understand. Even if I could watch all submissions, I would not know about a rejection, and even if I would, I would not know the reason. I am really puzzled!

If you want to understand it like this, it’s OK, but that’s not what I did. Over all, I even suggest less work for all, by a little tweak of the process. This, or I still misunderstand the whole process.

3 Likes

These statements conflict. You are asking for people to do more work and then claiming you’re not asking for anything. You may see it as “not a lot” but I don’t think that’s true.

If you had read the page, no where do we ask for a color target. I don’t understand why this took you am hour, its a 5 minute job.

I don’t see how writing a rejection reason and posting it is less work than not doing that. Your statements are quite confusing.

I’m sorry if you feel you’ve wasted your time.on the shots.

They don’t. I am not asking for something, I am making a suggestion. At least for me this is a difference. If it is not, then sorry, my English is obviously not good enough for this discussion.

I – again – suggest that people could invest a couple of minutes extra to save probably more time later by having way less uploads that have to be rejected. Net this could be less work, it of course depends on the actual number of submissions, number of rejections, and effort that goes into one rejection.

I assume that analyzing a submission may take 15 minutes. Posting a rejection may take 30 seconds. If only every 100th submission is rejected, the extra work stays extra work, but is overall negligible. If every 5th submission is rejected currently, but after some feedback of rejection reasons only every 10th submission is rejected, the saving is real, 15 minutes saved for 30 seconds invested per 5 pictures, and even a reduced effort of 15 seconds per 15 minutes because less rejections have to be documented.

I understand that it hardly makes sense to do such calculations, as life is never as straight and there’s a lot of gut feeling involved as well. However, the vehemency with which you are reacting to my suggestion – not asking for something but maybe desiring something because of the impression that it would make sense – made me feel to have a need for a more practical example.

This is among the responses that I was looking for. Simply saying: “I am the one who does the job and it would be tremendously more effort to put the rejection reason somewhere” is totally OK. As I am not sure if you are the one who does this job still, as said, Roman Lebedev was mentioned before, but as more people might be involved I did not want to ask directly but add this to the running discussion.

In some thread I am after all this fruitless discussion too lazy to look up, there was mentioned that a color target is not required but beneficial. Furthermore, it makes a good subject (it is no person) and it increases likelihood of proper exposure.

I had to revive a camera that I was not using for several years. It took quite some time to install CHDK on it (or find the old SD card where the suitable CHDK version was already installed) and make it ready. It was probably more than an hour. Still, even with a camera that I regularly use, I would expect this task to take more than 5 minutes. But maybe I am doing it too perfectionist.

You (I mean those that do the job) are investing a little bit of work to the benefit of saving a lot of work themselves later, as explained above. As said, judgement is by those that actually do the work, I might be wrong by my assumptions, but I thought this is a proper reason to discuss these things. However, I still do not understand what is so difficult to understand about this.

As I submitted 2 cameras, at least for one there is now a proper raw sample in place. I am still willing to contribute the other missing sample, but as already stated, I need to know what was wrong with my first submission. I don’t care about the time I wasted, I care more about my inability to provide a better sample for the camera that is in the list of missing cameras.

2 Likes

Seems he is busy at the moment. There are a couple of raw samples uploaded that haven’t shown up. For example D3s and R3.

Will this work with DJI Mavic Pro?

CC0
20220512DJI_0027.DNG (23.2 MB)
20220512DJI_0030.DNG (23.3 MB)
20220512DJI_0031.DNG (23.2 MB)

These are valid DNGs and should work with any application that fully supports the DNG spec or uses the Adobe DNG SDK. But they are not supported by the GainMap implementation in either ART or Darktable master.

The typical smartphone DNGs that are supported by those apps have the GainMaps in OpcodeList2, which are applied to each color channel separately before demosaicing, and correct both vignetting and color shading at the same time.

But these DJI DNGs work differently - in OpcodeList3 (to be applied after demosaicing) they have two separate opcodes. There is FixVignetteRadial which fixes the vignetting and has no effect on color, and there is a GainMap which only affects color shading.

2 Likes

Thanks for the explanation! Seems impossible to correct them with Lensfun due to colour shading.

Could someone modify your DNGpreprocess project to target that data and apply it the what you did to std DNG before it was merged into DT??? Or would it be a start from scratch project??

It would be a simple one line change to have the Adobe DNG SDK also perform the OpcodeList3 processing in addition to the OpcodeList2 processing, but that would mean the Adobe DNG SDK also has to demosaic the image in between. I’m not sure how it performs demosaicing but it doesn’t appear to have any sophisticated algorithm like AMaZE or RCD, so there could be some loss of quality.

As for supporting an image like this in Darktable - it probably would be worthwhile to add support for the FixVignetteRadial opcode as it’s also used by some iPhones. Some of the code from the recent change to support GainMaps could also be reused to make other DNG opcodes available to other modules, so the lens correction module could use the FixVignetteRadial data for vignetting correction instead of lensfun.

The GainMap in OpcodeList3 could be trickier - one possibility is to add a separate GainMap implementation either in the demosaic module or in a new module that comes after it. Another possibility is to take that GainMap in OpcodeList3 and transform it into equivalent OpcodeList2 GainMaps that have approximately the same effect but are applied before demosaic. It might not be exactly the same result bit for bit as if it was applied at the correct place, but would probably be close enough.

1 Like

Thanks for taking the time to explain it. I noticed in ART that you can select embedded for lens correction and it seems to apply one. I was going to ask you if it was a similar thing is grabbing from the meta data…

Hi Paolo,

Here are a couple of .dng files taken with my Fairphone 4 - they appear to have Exif entries for Opcode List 2 and Opcode List 3, so hope they might be of use regarding embedded DNG Gainmaps. Kind regards, Paul

IMG_20220409_101408.dng (22.9 MB)
IMG_20220413_170247.dng (22.9 MB)

Thanks! These have the typical smartphone GainMaps in OpcodeList2, which are supported by ART and the current Darktable master. The OpcodeList3 has a WarpRectilinear opcode for distortion correction - Darktable does not support that and I’m not sure what other apps do.

Adobe used AHD, Adaptive Homogeneity-Directed, as demosaicing method last time I checked.
Possibile to check by using this Bayer moire | LibRaw

It might be hard to see but here is your image…note dark corners…
image

Now with gainmap applied…

image

Yes, I can see the second image is definitely lighter.

How do I know if the dngs from my smartphone are using the GainMap in darktable? I checked that exiftool shows that they have gainmaps:

Opcode List 2 : GainMap, GainMap, GainMap, GainMap