Using camconst.json in Filmulator

CarVac · November 21, 2020, 1:35am

I’m planning on using RawTherapee’s camconst.json as a source of data for Filmulator, specifically for white saturation points for the time being.

Is the dev branch of RawTherapee on GitHub the canonical place for it, if I have a button auto-download it?

darix · November 21, 2020, 2:26am

I am not saying that you have a library which could be the canonical shared source for it.

heckflosse · November 21, 2020, 9:41am

In RawTherapee the dev branch on GitHub is the canonical place for it

CarVac · November 21, 2020, 1:08pm

Now to figure out how I’m going to make use of it…

I might have it be loaded into the SQLite database for easier querying instead of reading through it at startup…

Thanatomanic · November 21, 2020, 2:02pm

You probably already figured out it is not strict JSON, as there are comments in the file, which is not allowed. Something to take into account when parsing the file…

CarVac · November 21, 2020, 2:43pm

Actually I had no idea, because I have never dealt with JSON before.

I was hoping I could just use Qt’s library for parsing it, but maybe not…

ggbutcher · November 21, 2020, 3:29pm

I include camconst.json with rawproc, and I use Dave Gamble’s cJSON library to read it.

It seems to ignore the comments just fine…

Edit: Oh, that’s also what RawTherapee uses…

CarVac · November 23, 2020, 6:37pm

Happened upon this on Hacker News…

“To be safe allocate 5 bytes more than you need”

ggbutcher · November 23, 2020, 7:18pm

I don’t use that function, I think it’s only needed if you intend to print the data structure.

Here’s my code:

github.com

butcherg/rawproc/blob/master/src/CameraData.cpp#L56


			result = fgets(buf, 255, f); 
		}
		break;
	}

	fileorder.push_back("dcraw.c");
	camdat_status["dcraw.c"].append(string_format("<li>load: %d additions (%d replacements), %ld entries total</li>\n",i,d,camdat.size())); 

}

void CameraData::parseCamconst(std::string filename)
{
	int i = 0, d = 0;;
	FILE *f = NULL;
	long len = 0;
	char *data = NULL;
	size_t result;

	f = fopen(filename.c_str(), "rb");
	fseek(f, 0, SEEK_END);
	len = ftell(f);

CarVac · December 31, 2020, 9:53pm

I’ve gotten the parsing working now, but now I need a little advice on how aperture scaling works.

Do I scale the white saturation point before I subtract the blackpoint, or do I scale it afterward?

CarVac · December 31, 2020, 11:29pm

Actually, I might not bother with aperture scaling. The whitepoints in camconst.json seem pretty close to the limit as-is.

In other news, it now works! No more purple highlights on 80D pics!

Also I discovered that while the 80D whitepoint was set very unsafely for intermediate ISOs, my 60D’s whitepoint was set extremely conservatively by LibRaw, and at some ISO values I have exposed a decent amount of highlight headroom!

CarVac · January 2, 2021, 2:54am

Today I discovered that some Nikons have the camconst white level specified as if they were 14-bit, not 12-bit…

Yay for more complexity.

ggbutcher · January 2, 2021, 3:02am

Not sure if it’s the greatest thing to do, but in rawproc I use an instance of the blackwhitepoint tool to scale white, and I have an option named ‘data’ that scales it to the maximum value of the channel with the largest maximum. There’s a option with ‘data’ to scale to the minimum-maximum, which just blows clipped pixels to white. So, essentially I scale ‘per-image’. I need to work this into the librtprocess hilight_recon() behavior, which I ran out of brain cells to do prior to 1.0…

CarVac · January 2, 2021, 3:07am

I just detect when a) the camera is a Nikon, b) the LibRaw whitepoint is less than 4096, and c) the CamConst whitepoint is greater than 4096, and then convert the 14-bit value to the 12-bit value per the comment in camconst.

It’s a pretty simple fix, all things told. It does, however, assume that the LibRaw whitepoint is never less than 4096 for a Nikon 14-bit image…

Also, I wonder what’s gonna happen for future 16-bit raws (I think the a7s3 can write 16-bit)…

CarVac · March 11, 2021, 9:40pm

One thing of note: Fuji data in RawTherapee has no aperture scaling listed, instead saying “likely negligible”, but on this LibRaw page I stumbled across someone saying that the X-Pro2 has tons of aperture scaling, by a factor of 1.4 for f/1.2.

https://www.libraw.org/articles/Canon-5Dmk2-headroom.html

https://www.sylvainphoto.com/p901179378/ha0037831

That’s a large enough change to make it definitely worthwhile to ask for people to gather data on, I’d say.

Unless the camera itself throws out the raw data, given that it tops out at 16100?

Thanatomanic · March 12, 2021, 8:18am

@CarVac It may differ greatly based on specific camera model. For the FUJIFILM X-T3 people supplied sample images for me to determine that there was no significant aperture scaling, unless I have not done my analysis properly which is entirely possible. I may have wrongly extrapolated this notion to other Fuji models…

CarVac · March 12, 2021, 4:52pm

In terms of highlight clipping, there may be no need for compensation for aperture scaling even if the camera does apply it. After all, the white point for the cameras is quite close to 2^14.

If it’s true (that person said they did testing of the photon transfer curve) then Fuji is just throwing away a half stop of dynamic range at f/1.2. I wonder how strong it is at f/1…