rawproc 0.9 Is Released, and rawproc2 is Contemplated...

Software
1

Since I’ve been using screenshots of it in various places, I thought it was time to put a bow on the release. See the Release page on github for particulars.

I’ve gotten to the point with this code base that 1) I’ve learned a lot about how to better organize such a software, and 2) this code base probably dead-ends a lot of that learning. The group tool made that realization poignant; I like this tool very much but the 0.9 implementation was basically hacked upon the original tool chain organization. So that, and some of the data structuring around librtprocess has me considering re-starting (or, what do they call it in the movies, re-booting? :smiley: ) rawproc development. Some of the things I’m considering:

  • A whole new image library organization. I’ve already explored some of this in a gimage2 repository over at github. The main improvements would be 1) reorganize the internal image array around librtprocess data conventions; 2) collection all of LittleCMS, Exiv2, and Lensfun as first-class members of the gImage class; Oh, if you look at gimage2, ignore OpenCV, I’ve pretty well decided not to use that.
  • Move to the Qt toolkit. I’m really content with wxWidgets, but I need to consider a high-bitdepth display architecture.
  • Localization/Internationalization. All here of other-than-English speaking have been very polite to me, need to pay that back…
  • CMake building. Geesh, I’ve just figured out the GNU autotools, now this… :smile:

I recently semi-retired (half-time, working from home!), so I now have the time I’ve wanted for a few years now to do more of this. That’d be more rawproc, and more participation in the other projects. Oh, and getting out with the camera…

8 Likes
2

Thats the most important one! If you don’t have photos to process, why do you need to write software to process photos?! :wink:

4 Likes
3

Hi Glenn, it’s kind of you to announce this. I’ve often wanted to try your software.

It’s taken a bit of doing to almost get there:

installing librtprocess and wxWidgets were both tricky … gcc complained and I had to get around a couple of issues. Finally I got to make rawproc. But upon running it, I get this:

rawproc: error while loading shared libraries: librtprocess.so: cannot open shared object file: No such file or directory

yet when I whereis librtprocess I get

librtprocess: /usr/local/lib/librtprocess.so

Is there a command-line option to set the path for librtprocess? Should I do something different at configure time?

It’s probably something dead obvious, but I’mfeeling a bit brain dead at the moment, so I’d appreciate a point in the right direction.

4

With Ubuntu, I had to modify the library search path to include /usr/local/lib. Once that directory is added to one of the files in /etc/ld.so.conf.d, then run ldconfig

Edit: The AppImage might be a more expedient approach. Right now, it is the same executable that you’d make from the source checked out from master.

5

Thanks for the tip; I can start the application now, but get a lot of exceptions thrown once I have opened a pic.

6

Hmmm, paste them here so I can see what’s going on…

Edit: Oh, and tell me how you’re running it: Operating System, number of cores, amount of memory, that sort of thing.

At home, I run it in two environs: 1) Ubuntu/AMD 4-core/8GB memory, and 2) Windows/Intel 2-core/2GB memory, so it’s good to see what it does in other houses… :smiley:

7

uname -a

Linux BARADDUR 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

running on an AMD A12 -9800 (4 CPU cores, 8 GPU cores) with 16GB DDR4 RAM

edit: running a plasma desktop

(rawproc:10055): Gtk-WARNING **: 16:57:14.259: Unable to locate theme engine in module_path: “adwaita”,

(rawproc:10055): Gtk-WARNING **: 16:57:14.264: Unable to locate theme engine in module_path: “adwaita”,

** (rawproc:10055): WARNING **: 16:58:32.341: Invalid borders specified for theme pixmap:
/usr/share/themes/Breeze/gtk-2.0/…/assets/line-h.png,
borders don’t fit within the image

(rawproc:10055): Gtk-WARNING **: 17:15:18.411: Unable to locate theme engine in module_path: “adwaita”,

(rawproc:10055): Gtk-WARNING **: 17:15:18.424: Unable to locate theme engine in module_path: “adwaita”,

** (rawproc:10055): WARNING **: 17:15:59.800: Invalid borders specified for theme pixmap:
/usr/share/themes/Breeze/gtk-2.0/…/assets/line-h.png,
borders don’t fit within the image

(rawproc:10055): GLib-GObject-CRITICAL **: 17:16:31.360: g_object_unref: assertion ‘G_IS_OBJECT (object)’ failed

(rawproc:10055): Gtk-WARNING **: 17:16:31.381: Unable to locate theme engine in module_path: “adwaita”,

(rawproc:10055): Gtk-WARNING **: 17:16:31.393: Unable to locate theme engine in module_path: “adwaita”,

** (rawproc:10055): WARNING **: 17:16:36.562: Invalid borders specified for theme pixmap:
/usr/share/themes/Breeze/gtk-2.0/…/assets/line-h.png,
borders don’t fit within the image
/usr/local/include/wx-3.1/wx/strvararg.h(445): assert “(argtype & (wxFormatStringSpecifier::value)) == argtype” failed in wxArgNormalizer(): format specifier doesn’t match argument type
/usr/local/include/wx-3.1/wx/strvararg.h(445): assert “(argtype & (wxFormatStringSpecifier::value)) == argtype” failed in wxArgNormalizer(): format specifier doesn’t match argument type
/usr/local/include/wx-3.1/wx/strvararg.h(445): assert “(argtype & (wxFormatStringSpecifier::value)) == argtype” failed in wxArgNormalizer(): format specifier doesn’t match argument type

The exception wxFormat etc - is thrown in lots of 3 times - e.g. when I try to edit EXIF data, and I’m told there is no exif editor defined, I first get 3 of these exceptions.

edit: Is there an assumption I am running gnome, or a dependency on adwaita which won’t work using plasma?

This message pops up before the application window every time:
image

edit: Obviously I don’t have the documentation zip file; however the make install ran fine. Again, I couldn’t see a separate download for the doc zip file on your github page.

1 Like
8

@martin.scharnke, thanks for trying…

The GTK-Warnings started to show up when I switched to GTK3 in WxWidgets. I chased a few, but it would seem GTK3 is less tolerant of the WxWidgets assumed defaults than was GTK2. I ignore them, for the time being.

For the EXIF dialog, you need to have Phil Harvey’s exiftool installed somewhere. Then, go to the Properties dialog (Edit → Properties), look for the exif.command parameter, and enter the full path to exiftool there.

For the documentation, do ‘make doc’, and you’ll get a rawprocdoc.zip file in the same directory as the rawproc executable. Ah, I’ll need to modify ‘make install’ so it depends on doing ‘make doc’, and copies the rawprocdoc.zip to /usr/local/bin. For the time being, you’ll need to copy that file to /usr/local/bin by hand. Thanks for helping me find that.

9

Hi Glenn,

thanks for the tips … I ran make doc followed by a make install, but - yes, as you said - the doc file is not installed correctly. On moving it manually however, suddenly things worked - the main window comes up instantly! (Previously, as well as complaining about the absence, a very long delay - more than 30 seconds was the go).

I managed to Edit → Properties, no items defined, and added exiftool and the the full path. Error thrown “Unable to write config file.”

Again, however, load of an image is gratifyingly fast - when it works, which is mostly. Previously measured in minutes, now it takes a few (<10) seconds.

I’ll keep plodding along … just think of me as a beta tester, and I’ll get back to you when I’m stumped!

10

Ah, rawproc.conf didn’t get copied anywhere. In your home directory, make a .rawproc subdirectory and copy the rawproc.conf from build/src to it. That’ll be a persistent place that any copy of rawproc (AppImage, /usr/local/bin, dist if someone ever packages it) will find.

I think that, without a found rawproc.conf, the program is trying to write it to /usr/local/bin, for which the permissions probably are preventing it.

This is good; I spent a lot of time thinking through the wininstallers and AppImage, but not much consideration for folk who would compile from source…

11

OK, Glenn; no rawproc.conf to be found anywhere, so I created a blank one in .rawproc
Now I can save the exiftool.command entry, and rawproc successfully calls exiftool

12

Oh gee, I forgot about the “make conf” target; I have all this scripted, and haven’t actually typed this recently… I’m concentrating on remembering our up-coming wedding anniversary, don’t have the brain cells for recalling much else… :smile:

I try to keep the help file current and informative, not as expansive as RawPedia but should describe all essential rawproc functionality. One thing you will probably have to consider sooner than later will be setting up a directory for ICC profiles. What I did was simply to clone @elle’s GitHub profiles:

and set my cms.profile path to there/profiles/. That gave me a rich set of working/output colorspaces with a range of TRCs with which to experiment. In that directory I also put my display and camera profiles. Every property that requires a profile flie name will first look in that directory. rawproc doesn’t use the OS-specified display profiles, so you’ll have to put a copy in this directory and refer to it there.

1 Like
13

Hi Glenn, I found rawproc a few months ago and have really liked it. I was looking for something to use to pixel bin (half_size=1 / dcraw -h) and your tool is perfect. I had found rawproc v 0.62 and hadn’t realized there was a newer version that supports white balance. I just installed 0.9, but when I load a file, it’s tinted green now (I’m using Sony A7RIV files). The camera profile wouldn’t have existed in libraw from the time of 0.62 but it worked perfectly anyway.

Are there properties I need to set now in 0.9 to get the effect of 0.62’s default rendering? The only properties I had changed in 0.62 was colorspace=adobe;half_size=1;use_camera_matrix=0 (I might be wrong, but it seemed like use_camera_matrix=0 turned on cm and =1 turned it off somehow).

I noticed in the bottom right is a message: CMS:xform_error

Thanks! And thanks for building this!

1 Like
14

Oh, your post is timely, I’m working through such things for cameras I don’t have, trying to get to 1.0…

0.9 was a radical change, full independence from dcraw processing. It’s still there, but if you put in an input.raw.libraw.rawdata property and set it to 1 or ‘crop’ (1=full raw image, including borders, crop=borders removed), you start with the raw data right out of the file. Other changes include a basic set of pre-demosaic tools that have ‘camera’ options, where the tool will look for the relevant data in either the metadata, what libraw provided from the file, or from camconst.json and dcraw.c.

With that, I set input.raw.default to this:

colorspace:camera,assign subtract:camera whitebalance:camera demosaic:half blackwhitepoint:data

and this works for most cameras if the data is available.

This means that the display colorspace transformer doesn’t have either an input profile or display profile. display.cms also has to be 1, and is the property I put in to turn on/off color management with respect to display. So, if you have a display profile set in display.cms.displayprofile, then it’s likely there’s no camera profile. ‘colorspace:camera,assign’ in the above default processing will make one from either 1) dcraw.c’s adobe_coeff table, 2) RT’s camconst.json data, or 3) libraw data.

Prior to 0.9, if you used dcraw processing you likely got a profile from it, but it’s probably sRGB after dcraw does the conversion.

I know, this is a bit complicated, but it becomes more straightforward after you’ve grokked 0.9 and get it to work. To that end, if you could post a Sony A7RIV file, I can use it to 1) give you a good input.raw.default string, and 2) debug any foo I might have in rawproc. Right now, I’m doing just that for Fujifilm cameras… @@%!!!

15

Hey @ggbutcher https://raw.pixls.us might be useful for raw files :wink:

16

Not working:

This site can’t be reached

raw.pixls.us refused to connect.

17

Thanks for the reply! That’s good news!

I had originally thought to build my own raw converter (a very simplistic one, just enough to bin the file), but was delighted to find yours and to find it worked exactly as I was looking for (plus it has many other features).

With the high photosite counts of cameras like the A7RIV and GFX 100 (which have the same sensor design), pixel binning is finally a decent option, and so far in my tests seems visually superior to any interpolation method, which is what I’d expect. I don’t own the A7RIV yet, so I’ve been using raw files from review sites. I normally shoot film for personal work, and wanted a digital option with better color than interpolation allows, and I’m in the process of buying the RIV as I sell off some other stuff.

I’m using the raw file that can be downloaded from here: Sony a7R IV sample gallery: Digital Photography Review
I tried a number of raw files with 0.62, and all had reliable results. That one in particular has a lot of skin tones and some green leaves in the sun, which seem to benefit a lot from binning and show more dramatic differences.

The data must not be available for the RIV because I get an error that the primaries weren’t found, when the input.raw.default is set to that string (colorspace gives that error with each of the options it has).

I have not set a display profile or camera profile yet. That makes sense that dcraw would use sRGB.

1 Like
18

Hmmm we’ll have a look at that!

19

At the link you provided, I downloaded DSC02139.ARW, 62MB (Geesh!). Opened it in rawproc-dev (github master branch), and Libraw 19.5.1 identified it as ‘ILCE-7RM4’. The camera primaries also came from Libraw , and they are:

4913,-541,-202,-6130,13513,2906,-1564,2151,7183

I’m posting them here because in rawproc 0.9, you can copy that string and paste it into the filename box of the colorspace tool, and rawproc will turn it into a D65 profile. rawproc 0.9 was built with Libraw 18.something, which isn’t likely to have those numbers for this new camera. Doing this will give the display color transformer something to work with.

Also, if that version of Libraw wasn’t able to find the black subtract, that value is 512,

rawproc makes a copy of the working image for each tool, and this file choked my 8GB desktop. To mitigate this, for input.raw.default, use this toolchain:

group:colorspace:camera,assign;subtract:camera;whitebalance:camera;demosaic:half blackwhitepoint:data

The first four tools are combined in a ‘group’ for which only one copy of the image is made. New to rawproc 0.9, saves a boatload of memory…

Hope this helps; when I get closer to 1.0, I’ll make some release candidates with current libraries; might make this easier…

20

haha and that’s a compressed 12-bit raw instead of 14-bit, which is about 125 mb; you don’t want to see the file size of the gfx 100 for 16-bit (it crashes rawproc on my computer). fwiw, I’m on a 10 year old PC with 24 gb ram running win 7 64 bit and it doesn’t seem to struggle much with the A7R files (even the 14 bit files), maybe a 10 second delay if I don’t preempt with half_size (I only have about 2 gb free ram, and the cores are often running at 100C; I should probably do something about that eventually haha).

thanks, I tried those color primaries/black subtract and the toolchain, but it doesn’t seem much different, so maybe I’m doing something fundamentally wrong. After the white balance it becomes cyan instead of green, and after demosaic:half it becomes monochrome.