Script Processing in rawproc

Ah, forgot that wxfilename doesn’t populate the path by default. My scripts to date have all been in the cwd.

I’ll get it fixed over the weekend…

Now, I got multi line scripts to work just fine, windows line endings. Linux, will have to wait for the weekend to try.

Hah, I knew I was being dumb, why I didn’t think to put that in the current dir I have no idea!
I’m already thinking about options to pass metadata into images in the g’mic stack. This has loads of potential even as it stands…

1 Like

Just returned from cabin-in-the-woods, where I was able to code up an initial implementation of the ‘script’ tool. I used the ‘gmic’ tool code as a starting point, so it follows the same fundamental behavior: take the image from the previous tool in the chain, pass it as input to the script, and take the output of the script and present it to the next tool in the chain. ‘script’ does it with TIFFs, and runs the script program as a child process.

Using G’MIC and Imagemagick as examples, I wrote the tool to be configurable to an arbitrary number of script programs. All that’s required to use a particular scripting application is the definition of three rawproc propertries:

  • script.[scriptprogram].command: The absolute path to the script program.
  • script.[scriptprogram.commandstring: This is a template of the script command with fill-ins, looks like this: [program] [infile] [script] output [outfile],float This is my current template to run a gmic script. The [ ] are fillins, where rawproc replaces them with the values needed in the appropriates posistions of the script invocation.
  • script.[scriptprogram].channelformat: 8bit|16bit|float|unboundedfloat

The [scriptprogram] part of the property name is ‘gmic’, ‘imagemagick’, whatever you want to use to identify the script program. rawproc uses these to populate a drop-down selector in the parameters pane"

rawproc-scripttool

To use the tool, one just selects one of the script programs in the drop-down, then selects a script file using the file selector presented by clicking “Select File…”. The selected file is used to populate the commandstring template, which is then executed as a child process. rawproc waits for the script process to terminate, then retrieves the result image and stores it as the result of the script tool.

Checking the ‘auto update’ checkbox makes rawproc automatically run the script whenever the script file is changed. That way, one can work the script and save it, and rawproc will update without any other action.

The script tool disposes of the temporary TIFFs by default, but they can be retained by checking the ‘retain files’ checkbox. The image files will be overwritten in the next invocation of the script, so if one really wants to retain either or both, they need to move them before this happens.

The TIFF saved by the script tool for input to the script invocation is the image being edited, at the state of the previous tool. If the TIFF format is float, the value range will be 0.0 - 1.0; if the format is 8bit|16bit, the data will be scaled proportionately to the integer boundaries.

When the output TIFF is read back into rawproc, it’ll be used to replace the input image. The original metadata and ICC profile associated with the input image is retained (except for image width and height, those values are updated). If the image was modified with respect to any of this data, the user is responsible for manually updating it in the toolchain.

For anyone wanting to try it out, here are my relevant properties:

script.gmic.command=C:\msys64\home\glenn\ImageStuff\gmic\src\gmic.exe
script.gmic.commandstring=[program] [infile] [script] output [outfile],float
script.imagemagick.channelformat=16bit
script.imagemagick.command=C:\msys64\mingw64\bin\convert.exe
script.imagemagick.commandstring=[program] [infile] [script] [outfile]
1 Like

Looking forward to testing it later today, I suppose it’s more flexible and SSD/cache is probably fast enough that passing through filesystem won’t be dreadful.

1 Like

Going to test it myself on Ubuntu later today. 0.0 - 1.0 encoded floats seemed to pass back and forth just fine, same as libgmic. Let me know what you run into…

@garagecoder, a few things from running in Ubuntu:

  1. my config properties in the previous post is missing script.gmic.channelformat=float.
  2. The default for channelformat should be 16bit
  3. There should be a way to view the stdout/stderr for the script execution

So, there’s a new commit to do #2 and #3. I still get an occasional segfault at script invocation, so I’m now going to debug that…

1 Like

Well the good news is it works - I assumed the “script” branch and pressing “Run Script” works fine. I haven’t noticed it being too slow.

The “auto update” doesn’t seem to work - it detects that it changed, but from that point on does not seem to run the script at all. The Output button looks promising, although the default output went to “error”:
program_output

Once again I’ll have to leave tinkering for now, others things to attend to…

Thanks, I’ll look into the auto update…

FYI, there’s a race condition between updating the internal image and the status bar information population. After selecting a script, if you don’t move the mouse over the display panel you can avoid it, looking for a way to fix…

Edit: RE output vs error, that’s where G’MIC spits out its processing progress… :crazy_face:

1 Like

I would prefer you just output everything (stdout and stderr) or better yet have the option for a console popup to display it (like PhotoFlow does for all its processes - inelegant but I like it).

Okay, I’ve added a shell option. Set script.[scriptprogram].shell to a shell and whatever parameters it needs, and the script tool will use a shell program to run the command string. Add this to properties:

script.gmic.shell=wxcmd

and G’MIC scripts will be run in the single-command wxcmd shell that comes with rawproc. Haven’t tested it with other shells, but it works nicely with wxcmd. Note that the rawproc display image won’t update until the shell is dismissed. Also note that the shell program needs to be in a $PATH directory or the absolute path needs to be specified in the property.

Of note, Imagemagick convert doesn’t produce stdout or stderr output unless something goes wrong, so a shell isn’t helpful. I changed the Output button logic so it isn’t enabled unless output was generated by the script invocation.

BTW, I’ve noticed that I should have named all the script properties tool.script…, so this will probably change before I merge this with the master branch.

@snibgo for IM stuff, which I don’t remember very well and esp. v7, which is new to me.

ImageMagick uses stdout to read or write according to the command. For example we can read an image from “-” or write text information to “info:”.

As @ggbutcher says, IM writes to stderr only to report problems.

IM v7 has been out for some years now. It is better than v6 in many respects. The standard program is “magick”. The program “convert” is available for backwards compatibility with v6.

I’m going to let the script branch stew for a bit in the latest commit, seems to work reliably enough with the various non-nominal parameter actions I threw at it last night. As always, I’ll take whatever feed back folk might offer to clean up the current implementation, then I’ll merge to master AND THEN change the property names to be prepended with tool. That’ll keep the script branch set up to original expectations…

Now, back to 3D printing of HOn3 locomotive parts… :laughing:

1 Like

Only one thing spotted so far: if the current raw filename includes spaces, I couldn’t figure a way to get g’mic to use it (and the script therefore doesn’t run).

I’m wondering if script entries could be treated like a plugin system eventually. Other than that, it’s amazing how much time this saves - really appreciate this addition, thanks!

Yeah, that’s going to be a problem; early on, I settled on the space character as the tool delimiter in rawproc, sort of like G’MIC command lines were constructed. That has shown its limits in ways like this. I did include a simple name=val parse alternative when I reorganized the toolchain processing with the vision of eventually storing a JSON string in rendition images. Let me think about it some…

I’m glad it’s working otherwise…

@ggbutcher A separate issue here, but didn’t want a new thread just for it. I get this error sometimes:

free(): double free detected in tcache 2
Aborted (core dumped)

Steps to reproduce:

  • Add an exposure step
  • Add some other step after it, e.g. filmic
  • Untick the exposure (disable it but leave in steps)

Side note: I know there are other people eagerly awaiting builds with the script processing :wink:

Is this with the script tool, or gmic?

I’ll be a few days before I can get to it, owe my wife a weekend… :smiley:

I’ll also be doing the help file adds then; with that and a LittleCMS update, I’ll release 1.3 and associated builds. Late next week…

1 Like

No nothing to do with the script tool - as I say, I was just avoiding a separate topic :slight_smile:

There’s no hurry for me either, I can simply avoid pressing the untick…

1 Like

1 Like

Not sure whether it is relevant to you with grandchildren. (I think you mentioned you had some.)