rawproc 0.6 at GitHub - x86_64 AppImage Posted

Software
1

I feel a bit funny posting at this time, I don’t want to take anything away from the recent darktable port to Windows. If darktable would have had a Windows port back in the day, I may not have done any of this programming… Kudos and thanks to the darktable team, I’ll be trying it out.

Right now though, I’ve just tagged the 0.6 version of rawproc in github:

If you happened to have tried previous versions in Linux, I think you’ll find this version to be workable; I just previously didn’t have the wxWidgets chops to keep from doing things GTK didn’t like. Now, works fine with GTK2; I’ve compiled and tried GTK3 and it seems to be okay, but I haven’t tested that combination with the same fervor…

The big things I want to bring up here are:

  1. Colorspace tool: After the profiles discussion from a month or so, thanks to @Elle’s kind feedback, I decided to just add a Colorspace tool that can be inserted anywhere in the processing chain. I don’t know if this arrangement will stay, but for now it give me the ability to do apply/assign colorspaces wherever and see what happens. I haven’t calibrated my camera or monitor yet (too busy programming :slight_smile: ) and the dcraw ProPhoto space is working well enough so I’m going to stay there for the time being. But, I intend to wrestle understanding of color in the coming months, and this tool seems to be what I need to experiment.
  2. Redeye tool: I mention this one for @David_Macek, based on his recent thread. It doesn’t find eyes, but it does a decent job of converting the red to a realistic looking rendition.

There are 32 and 64 bit Windows installers and zipped executables at the release page. For compiling, 0.6 now has autoconf tools. The old Makefile is still there, my advice is to do an out-of-tree configure-based build or the old Makefile will be overwritten. Building rawproc is still an adventure, requiring building gimage first:

I’m going to study up on AppImage, so you may see one of those packages in the near future.

Things I’ve found useful:

  • Easy access to dcraw wavelet denoise. Just put a input.raw.libraw.wavelet_denoise parameter in Properties, try a value of 500 to start. Excellent denoise performance, doesn’t add much time to raw conversion.
  • Per-channel curve: I put this in to provide tools for tinting grayscale images, but most of you probably know it has much more utility. I’m now also using it for white balance correction, but a WB tool is contemplated for rawproc 0.7.
  • img, the command line program, now does wild card processing, so you can do a batch of images like this:
    img "*.NEF" gamma:2.2 blackwhitepoint resize:640,0 sharpen:1 ""album/*.jpg"
    and you get a bunch of equivalently named JPEGs in the album subdirectory with rudimentary processing. This is really the tool that enabled me to shoot RAW even for family snapshots; I used it on our recent vacation and was able to post decent pictures for family consumption every evening. Later, I go back and pick the ones I want to process more specifically with rawproc. Really, I’ve found that I don’t want to keep intermediate images for PP; what I really needed was a JPEG sized for general viewing and posting. 250k images are quicker to post than 2-3MB images, and look just fine in web browsers. Any different PP starts back with the NEF file.

So there, I think I’m going to take a programming break now, and shoot some images…

4 Likes
2

I just posted an x86_64 AppImage to the release page:

I made it with Ubuntu 16.04, may not be old enough according to the instructions. I also snarfed the libraries with a perl script I wrote that uses the AppImage excludelist, which didn’t exclude all the things I thought it should, e.g. X libraries.

Anyway, having never done this before, feedback is welcome.

3

It works for me. And now I have a chance to test rawproc more intensively :slight_smile:

4

This morning, I realized that putting rawproc.conf in the AppDir/usr/bin
directory wasn’t such a smart move. rawproc.conf is critical to using the
program in more than just a non-color-managed mode, and sitting in the
AppImages doesn’t let the program save the settings between runs. rawproc
startup logic is to look for rawproc.conf in 1) the executable’s directory,
and 2) the OS default for appinfo. So, I’m going to remove it from the
AppImage and test finding it in the ~/.appinfo directory. That’d require
downloading rawproc.conf separately from github. I’ll post the revised
AppImage this evening (MDT).

The help file describes using the rawproc.conf parameters and maintaining
them with the Properties dialog. Of interest:

  • For color management, I’m using @elle’s profiles, and the dcraw-supplied
    output profile as my working profile until I can make a camera profile. Set
    input.cms to 1 to turn it all on. Set cms.profilepath to a directory
    containing your icc profiles so you don’t have to enter a full path for
    each profile.
  • All of the libraw dcraw parameters are available through
    input.raw.libraw.*; the help file has the definitions.
  • You can mess with the multithreading using tool.*.cores: 0=all available
    cores; n=specific number of cores; -n=subtract n from the number of
    available cores.

There’s a bunch more settings to occupy your time; the help file is your
friend…

5

It is interesting that one can drag the histogram around… Not sure if that’s desired or not.

6

It is interesting that one can drag the histogram around… Not sure if
that’s desired or not.

Interesting thought; the only pane I made the conscious decision to drag
was the parameters pane; when cropping or rotating, making the tool image
bigger helps.

One thing that comes to mind; dragging all the dock panes out of the
application window leaves all that real estate for the display image.

7

Okay, an updated AppImage is located on the release page. You need to download a copy of rawproc.conf from the github source tree:
https://github.com/butcherg/rawproc/blob/master/rawproc.conf
and install it at ~/.rawproc/

8

0.6.1 is now on github:

Particularly, I implemented the input.raw.libraw.cameraprofile parameter. It lets you specify a camera profile to assign to a newly-opened raw. There are a few fixes for things I found after releasing 0.6.

If you’re going to try rawproc color-managed, get @Elle Stone’s well-behaved profiles and put them in your directory tree somewhere:

https://ninedegreesbelow.com/photography/lcms-make-icc-profiles.html

  • Set cms.profilepath to their location.
  • Copy your camera and monitor profiles to the same directory.
  • Set display.cms.displayfprofile to your monitor profile name (don’t need a full path if it’s in the cms.profilepath)
  • Set the output.*.cms.profile parameters to appropriate profiles for JPEG and TIFF.
  • Set input.raw.libraw.cameraprofile to your camera profile and you’re ready to go.

I know, that’s a bit of work, but I’m more into controlling the process than abstracting it.

9

Hi @ggbutcher

I downloaded the 0.6.1 appimage and made it executable, and also installed sys-fs/fuse-2.9.7. Unfortunately I couldn’t get the appimage to run. Instead, this complaint appears in the terminal:

./rawproc-0.6.1-x86_64.AppImage
rawproc: symbol lookup error: /usr/lib64/libpangoft2-1.0.so.0: undefined symbol: hb_font_funcs_set_nominal_glyph_func

Saving the indicated ./rawproc file from an earlier comment didn’t solve the problem, nor did using the --appimage-extract option.

Best,
Elle

10

Thanks for trying…

I’m curious, do you have any ‘pango’ library in /usr/lib or /usr/lib64? This may be an artifact of Gentoo’s display architecture vs Debian and derivatives.

I’ll update the AppImage to include both the fuse and pango .so files and repost.

11

The pango libraries are in /usr/lib64.

12

Hi @ggbutcher - I was able to run the rawproc appimage following a suggestion from this page:

using these commands:

./rawproc-0.6.1-x86_64.AppImage --appimage-extract
find squashfs-root/ -name harfbuzz -delete
squashfs-root/AppRun

Responding to this comment from RawTherapee restriction on user-chosen output profile - #25 by ggbutcher :

Following your instructions in the RawTherapee thread that we hijacked :slight_smile:

  • Assigning the custom camera profile from disk produces expected results - the colors look correct, matching colors produced by other raw processors.

  • Adding a colorspace layer and selecting “Rec2020-elle-V4-g10.icc” as the ICC profile produces unpleasant and oversaturated colors. The default (upper) checked box is the “apply” box, which I would suggest changing the label to “assign” as that’s what it seems to be doing.

  • Unchecking the “apply” (upper) box and instead checking the “assign” (lower) box produces the correct colors that should result from converting from the camera input profile to “Rec2020-elle-V4-g10.icc”. So this box’s label should be changed to “convert”.

Using the default rawproc settings (removing “rawproc.conf” and then restarting rawproc), I saved the resulting image to disk as a tiff. Opening the resulting tiff in GIMP-CCE, the image was in dcraw’s incorrectly-made ICC sRGB profile with the D65 white point. The image colors look to me as if:

  1. Somehow “Rec2020-elle-V4-g10.icc” is being used as the camera input profile, instead of the dcraw camera-specific profile made from the adobe_coeff table.

  2. Somehow a curve is applied to the image file to produce altered tonality, by dcraw? maybe the Rec709 curve? maybe the sRGB curve? Maybe via an ICC profile assignment somewhere?

  3. The resulting “wrong colors and wrong tonality” image is converted to the built-in dcraw sRGB profile and saved to disk.

Something other than 2 above could be happening to produce the final image tonality, but I’m pretty sure - just by looking at colors in the raw file displayed in the rawproc image window - that somehow “Rec2020-elle-V4-g10.icc” is by default being used as the camera input profile, instead of the appropriate “standard” dcraw ICC profile from the adobe_coeff table.

As an aside, may I suggest making the user interface backgrounds (119,119,119) by default? A pure white background makes it difficult to judge colors and tonality.

Also, is it possible to use black point compensation when sending the image to the monitor screen?

13

Very good feedback. Now, to get it straight, “convert” would be to apply cmsTransform() to the image, transforming its colorspace from it’s current profile to a specified output profile? Then, “assign” would be to simply replace the assigned image profile with the specified one?

I might also have the logic reversed, will check that first. I wll change from the “apply” verb to “convert”.

Hmmm, removing the rawproc.conf should have turned color management off, and no profile should be saved with any output image. Will check that.

When you open an image, right-clicking on the entry in the command tree and selecting “Image Information” will display a dialog containing, among other things, the colorspace profile assigned to the image. When the colorspace tool is selected and “apply” (soon to be “convert”) is selected, the image is transformed from the assigned profile to the one specified in the tool.

Yes, and Yes, in the next dot-build (0.6.2…)

14

Yes. Except that when you first interpolate a raw file to make an RGB image file, technically there is no assigned profile at all, until a camera input profile is assigned.

The only thing needed to fix the colorspace layer in rawproc is to change the labels for the check boxes. The upper checkbox label should be “assign”. The lower checkbox label should be “convert”. The actual code seems correct.

15

Hmm, my apologies, trying again with a different image and with no rawproc.conf , I can’t replicate the"looks like Rec2020" behavior I described above. Instead the rawproc image looks pretty much like rawtherapee output using the “neutral profile” with the standard adobe_coeff matrix.

Well, the image I opened this morning using rawproc and then saved to disk did have the dcraw sRGB profile embedded. What do you mean by “turned color management off”?

16

input.cms=0 turns off color management. It’s checked at every file open, and applies until the next open. I’m not in a position to check, but that should be the default in the code, so no conf file should be 0.

17

Yes, input.cms=0 if there’s no rawproc.conf.

In Help->About, the configuration file rawproc opened at startup is shown, or (none) if none was found.

At startup, rawproc looks for a .conf first in the current working directory, then in whatever directory is specified by the OS for user configuration. In a Linux distro, that should be ~/.rawproc

I appreciate the exchange, I’m getting a sense of what would be important to put in a “quick start” guide…

18

@elle, I found the problem with the colorspace tool; the profile wasn’t being updated for the display, so the previous profile was being used against the converted image. That, and a few other things, are fixed and in a 0.6.2pre AppImage posted in the 0.6.1 release page:

https://github.com/butcherg/rawproc/releases/download/0.6.1/rawproc-0.6.2pre-gentoo-x86_64.AppImage

md5: 8b5a94fd44df17e94f20ce5cf3a436d4

Note to others, this AppImage has a library removed, libharfbuzz, that Gentoo didn’t like. It should work okay with Debian/Ubuntu distros, YMMV for others.

So, there are two approaches to do color management in rawproc. Both require the following settings in Properties:

input.cms=1
display.cms.profile=yourdisplayprofile.icc
cms.profilepath=/path/to/your/profiles

Option #1: dcraw profiles

input.raw.libraw.colorspace=prophoto, or one of the other dcraw options
input.raw.libraw.gamma=prophoto, or one of the other corresponding gamma presets

This will allow opening a raw file with the image converted from the adobe coeff, to the dcraw output profile and TRC applied. That’ll then be the working profile for the rest of the processing chain.

Option #2: Proper Working Space Profile

input.raw.libraw.colorspace=raw
input.raw.libraw.gamma=linear
input.raw.libraw.cameraprofile=yourcameraprofile.icc
input.raw.default=colorspace:one_of_your_working_profiles.icc,convert

This at open will assign yourcameraprofile.icc to the unconverted raw linear image, then input.raw.default contains tools to apply when a raw file is opened, and here a colorspace tool is specified with a working profile and ‘convert’ for the operation. When the open and default processing is done, the image is available for further work in the working colorspace.

To your original challenge, getting a TIFF with the camera profile assigned would be:

input.raw.libraw.colorspace=raw
input.raw.libraw.gamma=linear
input.raw.libraw.cameraprofile=yourcameraprofile.icc

Open the raw, then save as a TIFF.

1 Like
19

Okay, I’ve now used the above-described fixes on a couple of computers, and they don’t work well on slower Windows devices. I’m guessing it’s a thread-based race condition, not getting to working the queue of the wxWidgets Refresh() routine before something else kicks in…

@Elle, thanks for trying things, but you might consider not wasting your time on it right now. I’ve got a better fix for the colorspace tool committed, and I’m going to make a few other changes like multithreading and adding rendering intent to the parameters, then I’m going to just use them on the available computers for maybe a week to make sure I have it working correctly. So, 0.6.2 in a couple of weeks.

1 Like
20

Hi Glenn. My apologies for not having already tested the rawproc changes. Recently my “computer time” has mostly been occupied with dealing with digiKam and with multiple copies of various images and other files scattered over nine different hard drives. Bleh. Well, also I made time to edit a couple of images. I’m very curious to see the latest developments in rawproc, but per your advice will wait for the next release.