Lots of raws to process -- How to minimize impact on computer?

Software ART
1

My wife and I stayed in a mountain-top cabin overnight (350 miles away) and I shot a sequence of more than 1,000 images for a time lapse.* However before shooting I forgot to change the format from raw to JPG. So I have a LOT of raws to process. Yikes!

* This is really an academic exercise since the images are no great shakes, but …

I’m on Windows, so rather than endure the pain of batch file loops, I have a batch file with an art-cli.exe line for each file. (Using a text editor it’s simple to edit with search / replace.) For example:

"C:\Program Files\ART\art-cli" -o C:\data\images\_stage\processed -s -j80 -c C:\data\images\_stage\IMG_3534.CR3
"C:\Program Files\ART\art-cli" -o C:\data\images\_stage\processed -s -j80 -c C:\data\images\_stage\IMG_3535.CR3
"C:\Program Files\ART\art-cli" -o C:\data\images\_stage\processed -s -j80 -c C:\data\images\_stage\IMG_3536.CR3

So anyway, it works fine but I’m more than a little antsy about the prospect of my (good-enough-but-not-a-beast) laptop with its fan roaring – and CPU cooking – for several hours as it chugs through all these files.

  • Any suggestions on how to minimize CPU impact?

  • Anyone ever done this with so many files?

  • Is there a sleep equivalent for Windows? I could put that in every XX lines to give the laptop a (five minute?) rest for as long as it needs.

Or am I just being a worrywart? I don’t want to do something I’ll regret, know what I mean?

Thanks.

1 Like
2

I’ve never really worried about this - I suppose on the assumption that the laptop will have enough self-preservation instinct… :wink:

Having said that, a few thoughts come to mind.

Can you set performance level on the laptop? Many have options (either in Windows power setting or a proprietary app) to tradeoff between performance and battery life - I suppose if you set it up for max battery life it should logically run cooler (and slower…).

Prop the laptop up at the back to allow maximum airflow. (Check where the inlets and outlets are).

Put it in the fridge??? :wink: :cold_face:

3

timeout is a quick and dirty way, giving you an option of delaying further execution by a maximum of 2.77 hours. I do not know if you can tie it to temperature analytics… :thinking:

Probably the best way is to schedule the processing so that it happens whenever you are sleeping and not using the laptop.

Last thing is to remember to redirect the standard output and error into a log, so you can keep track of what is being done and the errors that may surface.

Edit: Does ART have a built-in batch processing feature/tool? Calling the app repeatedly seems inefficient…

4

If you have decent jpg previews in your raw files you could extract them…

2 Likes
5

Yeah, maybe so. I know I can change process priority in Task Manager, but that would have to be set for every image (each is a new PID). But apparently one can (or at least could, in the past) run a command with:

start /low command line ...

I’ll run some tests on that when I have time. Hopefully soon, maybe…

[ Unrelated details ] I’ve been out of the loop for a bit. We just returned today from a Fri-Mon 1500 mile drive. My oldest sister in Kansas City passed away in January, but they were having subzero (Fahrenheit) highs for days / weeks, so any kind of gathering (requiring travel) had to be postponed. It happened Saturday evening and was really good. But we drove 750 miles Friday, 400 Sunday and then 350 today – with 80% of it in the rain. And 300 miles from home we had a tire start shaking, which fortunately we were able to get replaced promptly this morning (it was starting to split). Anyway, all’s well that ends well, but I’m in catch-up mode for a while now… And Wednesday we’re getting all our home window panes replaced, so there goes that day! :slight_smile:

Thanks.

1 Like
6

Well, there it is. Thanks!

Yes, it can be done in the GUI, but I was hoping to avoid the unneeded overhead of the UI (GTK, etc.) by using the command line. One advantage I suppose of using the GUI is that I could lower the priority using Task Manager. Offhand I don’t know if it spawns subprocesses, though…

Hmmm…

7

They’re full-res, but I want to raise the chroma a tad, add lens correction, etc. so I need to process. However if I could extract the thumbnail from the raws after applying my edits across the board that might work. I wonder if that would take less CPU than processing to a JPG?

And how is that done, by the way? :smiley:

8

I was thinking of this:

image
image1539×550 30.5 KB

and this (from the “change plan settings” in the above screenshot
image
but I haven’t really looked into it. Will depend on hardware too.

Now that’s a road trip!

9

You can have a range of sizes in DT …not sure of the quality but there are some settings around that as well… in ART I would have to go digging…

10

There are other settings there that one could explore. I tend to resort to that only when resources are already low and I want to smooth the throttling and fans a bit more.

@lphilpot Sorry for the loss. Glad you returned home safely.

1 Like
11

Yeah, that’s really at the OS level although by default it would cap anything art-cli was doing. Ideally I’d like to constrain only art-cli, but I’ll have to look into it.

After working on Windows in IT for so long, it became second nature on every new (and often, not so new) PC to go into power settings and disable every sleep, limit and otherwise “green” feature we could find. Not that we wanted to waste power, but those settings always seemed to cause issues.

1 Like
12

Thank you.

13

Mm. Yes. I was kind of working on the assumption that you’d leave it running overnight, so you wouldn’t be doing anything else…

14

unless your laptop has cooling issues, you should be fine to run it.

if you’re really that worried about the laptop, you can probably spin up a VPS on a cloud provider for a few bucks, rysnc all the files over, process them, and rsync it all back without too much issue.

2 Likes
15

Hi,
From art’s pov:

  • don’t call ART-cli.exe separately for every file, you can call it once to process all files (if they are really many you can hit the limit of the command line length, I don’t know if this is an issue on windows though)

  • do you need the full sized pics or are you resizing them? If the latter, you can consider the fast export pipeline (-f flag) which can speed up significantly the process depending on the final output size

HTH

16

dcraw -e *.raw

1 Like
17

I initially was going to call art-cli.exe once on the directory, not the files individually. The only reason I ‘went’ multiple invocations was to potentially insert a sleep (or equivalent) every now and then. Per Google Windows 11’s command line length is 8191 chars total. So, if I have (~1061 images * 12 chars per filename) + params, it’ll apparently overflow. I could divide them up into a handful of directories easily enough, though.

Oops, I didn’t think about resizing. For my purposes HD (1920x1080) is good. So fast export will speed up resizing, but will non-resized images be still faster? Or is there a benefit to fast export regardless? We’re talking about 24mp (6000x4000) images.

Or, as @afre indirectly touched upon is there any advantage to using art-cli.exe vs just doing it in the GUI?

Thanks.

18

I have no reason to believe it does, it’s just a regular run-of-the-mill laptop. But if I found out the hard way I was wrong, that ounce of prevention might have been worth a laptop of cure. :slight_smile:

Thanks.

19

I wouldn’t be to worried about the computer, the CPU will slow itself down when it gets too hot. Depending on the orientation of the fans in the laptop you may want to elevate it to get more air flow under the chassis.

20

I was referring to the command line (@agriggio confirmed it is possible in his comment). Perhaps, batch files do not cause command line overflow. @snibgo may know more about command line and batch processing.