Rawproc: yet another raw processor, aptly named

Processing
1

Probably more to generate discussion about the plethora of raw processors than any other purpose, I’ve put my source code out on Github at:

I do want the authors of the good softwares to take heart in the fact that I am a serious user of G’MIC, Raw Therapee, and GIMP, and I’m not looking to replace any of those tools in my kit. But my interest is railroad photography, and I wanted a way to use limited resources in the field to do what I’ll call ‘contact sheet’ processing - get a decent JPEG out of a Nikon NEF for posting, and later use as a reference for ‘real’ processing with the good tools.

I use a Microcenter Winbook tablet for this work, it is indeed cheap and slow, and rawproc works pretty well on it. I also have been using rawproc on a Ubuntu desktop with a 23" monitor, quad-core AMD and 8GB memory, and it works really well there. I envision a workflow where I shoot RAW only, then review them in the evening in Windows File Explorer. For the ones that look interesting, I open them in rawproc and quickly make a decent TIFF. It’s the presence of the TIFF file that constitutes ‘image management’ for me, and the basis for subsequent cropping and resizing for posting to OneDrive for others to regard. rawproc has only been usable in that regard for about a week, so I’m still feeling out the concept, but I’m encouraged - I can quickly and easily open a NEF, apply gamma and a curve to get a good TIFF, in say, a minute or so.

I posted the code to github now primarily to facilitate synchronizing my Linux and Windows development work, but I thought I’d also point to it for discussion here. I’m not so much interested in others using it as I am in discussing yet a different approach to RAW processing. In that regard, I’m not going to offer a lot of support for rawproc as a product; you need to be able to compile code to use it, and you’ll have to wait for me to make fixes and enhancements on my own compunction, or fork it.

2 Likes
2

Thanks for sharing your code, I’ll have a look and try to compile it as soon as I manage!

Meanwhile, since you own a Nikon camera and you are interested in having a “quick starting point” for your edits, you might be interested in this other answer where I give a couple of resources that explain how to precisely match the in-camera Jpeg colors and tones using free RAW converters: Can I match RT to out-of-camera JPEG - #5 by Carmelo_DrRaw

1 Like
3

Carmelo,

I actually stepped through your tutorial a few months ago, while figuring out to what extent the camera processing went to produce its JPEGs. Very informative. The conclusion I came to was that, like in using the Zone system to expose and develop film negatives, I wanted to start with as close to the raw digital data as practical and lay on each manipulation step with full knowledge of what it was doing. In that regard, I drive manual transmission cars… :smiley:

Since I’ve resurrected my interest in photography (previous: the '70s, 35mm Pan-X, etc), I’ve wandered through the internet resources marveling at the extent to which the digital medium has been simplified and abstracted for folks to concentrate on producing images. Conversely, I’ve seen some of this simplification lead to misconceptions about what’s actually happening in the process, and sometimes this misconception leads to frustration when a photographer wants to achieve a certain result and does not understand why the slider doesn’t behave like they believe it should. And that’s why I’ve subscribed to post here, because here is where I’ve found the best confluence of creative and technical considerations for my taste.

1 Like
4

I compiled your program and had a quick test. My impression is that there is no color conversion (yet) of the RAW data to some working colorspace (sRGB or the like). I think you will need that for a reliable color reproduction in your initial TIFF, because the RGB values that get directly out of your sensor need to be “interpreted” by an input camera profile in order to be meaningful…

Other than that, the program works fine and is definitely quite fast!

5

A screenshot would be nice.

6

You should add a LICENSE file to your github repo to make it clear which exact version of the GPL you are using.

7

Done. Since two sources of contributed code are GPL2, I chose that license until I can sort out the impact of using GPL3.

8

Here’s two screenshots, both based on “open source-ing” a file named DSG_3890.jpg, which had a UserComment EXIF entry of:

rawproc-0.1 DSG_3890.NEF gamma:2.2 curve:16.0,0.0,64.0,109.0,131.0,194.0,220.0,256.0 saturation:17

The Open Source… menu entry retrieves this command from the selected file, opens the file specified in the command, and applies the manipulations listed, in order. No sidecar file…

1200×778 1.21 MB

Pretty simple; a big display pane, a manipulation list pane to the left, as well as a parameter pane for the currently selected manipulation. In the top-left of the display pane is a thumbnail that can be toggled between a pan manipulator (little white box that you can drag around the display when zoomed in), a display histogram, and no-thumbnail, using the ‘t’ key.

The second screenshot shows the histogram, and the curve manipulation. Curve manipulates the RGB tone, no separate channels or other colorspaces. What it does do that is important to me is handle 16-bit values in the appropriate range, 0-65535; while the curve range is 0.0-255.0, it’s calculated in floating point (c++ double) so I pull values every 0.003891 or so for a 16-bit lookup table.

1200×778 1.18 MB

The histogram is of the 0-255 8-bit display image. That bothered me for a bit until I came to realize that a 16-bit histogram displayed in the same space wouldn’t look much different, because the thumbnail is only about 200 pixels wide.

In this screenshot, notice that, while the curve tool is selected, the checkmark is on the saturation tool, so the display is based on the cumulative application of all the tools. You can select any of the tools to display, and fairly quickly step through the manipulations to see the differences. Well, not so quickly on the cheap tablet, so I fixed the histogram so it would only calculate if toggled-to, which sped up display-switching considerably.

There are still a few ragged edges; the README on the github page lists them. I’ll be cleaning them up in the next few weeks, but I’m having to interleave that effort with taking on a new position at work. Doggone day job…

Carmelo, I’m pleased you were able to compile and run rawproc; my build system is pretty hard-coded to my two development machines. Posting the code compelled me to clean a few things up in that regard, which also helped me out as I was previously just using “thumbdrive-hub” for source version control. You can imaging the synchronization issues with that approach; indeed, I still have to work out a tablet-desktop diff as the histogram generation change I mentioned above is not in the github code yet.

All, thanks for looking…

9

Carmelo,

That’s probably the first thing I plan to consider after cleaning up the manipulation tools. Right now, FreeImage uses the libraw library, and the only libraw parameter they expose is the choice of image, either demosaiced raw, thumbnail, or preview. You’ll note that I light up radio buttons on the open diaolg to choose one of them if a raw image is selected, but getting to the other parameters, including the output colorspace, will require a modification to the FreeImage source code. At this time, I believe the default colorspace in libraw is sRGB, so that’s what rawproc is starting with.

I’m toying with an option to use dcraw to produce the demosaic-ed image, and similarly employ exiftool to migrate EXIF to output images, but I’d much rather be able to handle such with the libraries.

Note to Moderator: I just saw the “Consider replying to several posts at once” missive in the preview box; I’ll start doing that after this post…

10

Why not just use LibRaw? (I know, “just”, but if your intention was a raw processor…)

11

CarVac, THAT is a great question. I started to do just that, with dcraw, but I found I didn’t want to batch-convert all of the NEFs in a directory from a day’s shooting. So, to pick, you need to look…

I actually considered just using the camera-produced JPEGs for “contact sheet” PP, but at the time I didn’t trust the camera to do more than what I wanted to consider regarding exposure. I also wanted to look, tone-adjust, crop and resize for posting all at once, and no single tool easily and quickly supported that. So, here I am…

I may yet abandon my coding effort as I learn more, but really, that was the essential point of doing it in the first place: LEARNING! And, writing curve look-up-table, saturation, and even just FreeImage->wxWidgets display code taught me a whole bunch about digital imaging.

12

It’s pretty easy to use LibRaw.

For example, here’s all of the code Filmulator needs to get images from LibRaw into our own matrix format (which we use instead of plain arrays)

13

Thanks, no question it’s straightforward. What I have to decide is if I want to dig into FreeImage to make the mods, or use libraw directly. I’d prefer to just use FreeImage, but that would mean either providing a patch to the FreeImage developers or including the FreeImage source with rawproc.

You’ve got me thinking about using libraw directly; I’d have to write a imread.cpp equivalent for creating a FIBITMAP, but I already have done such in converting FIBITMAPs to wxImage for display. In my source tree, FreeImage16.h is a set of routines I’ve already written on top of FreeImage, mainly to do 16-bit processing where FreeImage only did 8-bit. CarVac, thanks for the observation…

14

@ggbutcher @patdavid shall we move this into a new “rawproc” software category?

15

No, it’s fine here I think.

16

I’m fine here. I wouldn’t want my hack code to be construed as a software product that people should rely upon… :smiley:
I just appreciate the opportunity to discuss the stuff I’ve been wrestling and learning about PP since I’ve re-engaged photography as a hobby…

1 Like
17

Here are two screenshots of the same image, one obtained with rawproc in “Display” mode and the second one with darktable using the default Adobe matrix.

My conclusion is that LibRAW is definitely doing some color conversion, but seems to be using something different than the default adobe matrices: the rawproc result is a bit darker and more saturated than DT.

With photoflow I obtain the same result as DT.

2560×1324 1.13 MB

2560×1452 2.44 MB

Personally I always shot RAW+Jpeg, and use the Jpeg to judge which pictures to consider for further editing.

In general your camera applies some custom tone curve and some saturation boost when generating the Jpeg, to give you a more “pleasing” result. With Nikon cameras, this can be tweaked using the “Picture control” settings.
Personally I find the “Normal” setting of my D300 a bit too strong in terms of contrast, while the “Neutral” setting is too flat. The best compromise is IMHO the “Portrait” setting, which is only available in post-processing with Cature-NX (or the corresponding ICC profile loaded into PhotoFlow). This is very often a rather close match of the final rendering I’m aiming for most of the pictures… of course you can mimic one setting or the other with curves and saturation adjustments (or at least get very close to the in-camera results), but that’s an additional work.

18

Carmelo,

I just spent some quality time with the libraw code that comes with FreeImage, and I’m not finding specifically where it sets the output_color parameter. All I have to go on right now is that libraw strives to be dcraw-compatible, and dcraw’s default colorspace is sRGB, if -p is not used to invoke the camera-specified colorspace.

Eventually, I’m going to want to control this in rawproc input, as well as the choice of demosaic algorithm and the white balance parameters. With regard to using the camera JPEG as a baseline, I did that at first but found (mostly due to my ignorance) I was fighting the Camera Control settings in my real-time selection of exposure. I may yet go back to that approach, but for right now I’m enjoying the challenge of taking my raws and seeing what it takes to make them look appealing.

19

For some things, LibRaw doesn’t use the default Adobe matrices.

Last summer I rented a 6d to try out, and then I updated to the latest version of LibRaw and the white balance changed: they changed the matrix because of, apparently, complaints about color rendering for that camera. Long story short, but the way Filmulator handles white balance made that matrix change have a very large impact on previously imported files, and I was surprised that they’d do that.

So it definitely could be LibRaw having a different matrix.

@Morgan_Hardwood, I’ve seen at some point it said that RawTherapee uses different matrices than (I think) dcraw. Why, which cameras, and how are they different? Or is this just me not remembering something properly?

20

Some of dcraw’s matrices and white/black points are of poor quality and off, resulting in poor colors and magenta in highlights and clipped shadows. dcraw is also slow and Dave wrote that he is “extremely reluctant” to apply ready-made patches which would significantly speed it up. I hear rawspeed does things differently.