Lots of raws to process -- How to minimize impact on computer?

I was thinking of this:

and this (from the “change plan settings” in the above screenshot
image
but I haven’t really looked into it. Will depend on hardware too.

Now that’s a road trip!

You can have a range of sizes in DT …not sure of the quality but there are some settings around that as well… in ART I would have to go digging…

There are other settings there that one could explore. I tend to resort to that only when resources are already low and I want to smooth the throttling and fans a bit more.


@lphilpot Sorry for the loss. Glad you returned home safely.

1 Like

Yeah, that’s really at the OS level although by default it would cap anything art-cli was doing. Ideally I’d like to constrain only art-cli, but I’ll have to look into it.

After working on Windows in IT for so long, it became second nature on every new (and often, not so new) PC to go into power settings and disable every sleep, limit and otherwise “green” feature we could find. Not that we wanted to waste power, but those settings always seemed to cause issues.

1 Like

Thank you.

Mm. Yes. I was kind of working on the assumption that you’d leave it running overnight, so you wouldn’t be doing anything else…

unless your laptop has cooling issues, you should be fine to run it.

if you’re really that worried about the laptop, you can probably spin up a VPS on a cloud provider for a few bucks, rysnc all the files over, process them, and rsync it all back without too much issue.

2 Likes

Hi,
From art’s pov:

  • don’t call ART-cli.exe separately for every file, you can call it once to process all files (if they are really many you can hit the limit of the command line length, I don’t know if this is an issue on windows though)

  • do you need the full sized pics or are you resizing them? If the latter, you can consider the fast export pipeline (-f flag) which can speed up significantly the process depending on the final output size

HTH

dcraw -e *.raw

1 Like

I initially was going to call art-cli.exe once on the directory, not the files individually. The only reason I ‘went’ multiple invocations was to potentially insert a sleep (or equivalent) every now and then. Per Google Windows 11’s command line length is 8191 chars total. So, if I have (~1061 images * 12 chars per filename) + params, it’ll apparently overflow. I could divide them up into a handful of directories easily enough, though.

Oops, I didn’t think about resizing. For my purposes HD (1920x1080) is good. So fast export will speed up resizing, but will non-resized images be still faster? Or is there a benefit to fast export regardless? We’re talking about 24mp (6000x4000) images.

Or, as @afre indirectly touched upon is there any advantage to using art-cli.exe vs just doing it in the GUI?

Thanks.

I have no reason to believe it does, it’s just a regular run-of-the-mill laptop. But if I found out the hard way I was wrong, that ounce of prevention might have been worth a laptop of cure. :slight_smile:

Thanks.

I wouldn’t be to worried about the computer, the CPU will slow itself down when it gets too hot. Depending on the orientation of the fans in the laptop you may want to elevate it to get more air flow under the chassis.

I was referring to the command line (@agriggio confirmed it is possible in his comment). Perhaps, batch files do not cause command line overflow. @snibgo may know more about command line and batch processing.

Batch operations from the command line are possible, just point art-cli.exe at a directory rather than at an individual file. The selection criteria for the batch op is then just whatever’s in the directory.

I do loads of batch processing, eg from frames of videos, mostly using ffmpeg and ImageMagick. I’ve never used art-cli.exe.

On my Windows 11 laptop, and previously with desktops, this leads to lots of fan-noise. This has never been a problem. Just ensure the vent holes aren’t blocked, eg by accumulated dust. If a job takes my computer all night to process, then it does, no problem.

The Windows start command can run a program or batch file at a low priority. If the computer isn’t doing other work at the same time, then this makes almost no difference to anything. Putting work at the back of the queue, when there is no queue, makes no difference.

Some software (eg ImageMagick) contains facilities to throttle CPU usage, and limit the number of threads. These can be set in policy.xml. These settings will reduce the CPU heat.

I can’t find a Windows setting that can, for example, limit a process to 25% of CPU usage.

If desired, pauses can be inserted between intensive commands in a batch job to allow the CPU to cool down. But if the fan still needs to run during the intensive commands, there isn’t much point. The temperature cycling may even be slightly detrimental to the hardware.

1 Like

Sorry about my previous, terse reply – I wanted to help but had little time. I have more now.

First and foremost, I’m sorry about your loss.

As for the embedded JPGs, besides dcraw, one can use e.g. exiftool (since it’s really about headers and metadata, not about the raw file itself). The man page of exiftool provides these examples:

   exiftool -b -ThumbnailImage image.jpg > thumbnail.jpg
        Save thumbnail image from "image.jpg" to a file called "thumbnail.jpg".

   exiftool -b -JpgFromRaw -w _JFR.JPG -ext NEF -r .
        Recursively extract JPG image from all Nikon NEF files in the current directory,
            adding "_JFR.JPG" for the name of the output JPG files.

   exiftool -a -b -W %d%f_%t%-c.%s -preview:all dir
        Extract all types of preview images (ThumbnailImage, PreviewImage,
            JpgFromRaw, etc.) from files in directory "dir", adding the tag name
            to the output preview image file names.

exiv2 also provides this functionality, using the command:

   ex | extract
          Extract metadata to *.exv, XMP sidecar (*.xmp) and thumbnail image files.  Modification commands can be applied on‐the‐fly.

The command also takes a ‘target’ argument:

   ‐e   tgt  ‐‐extract Extract target(s) for the ’extract’ action.

Where

   tgt       a | c | e | i | p | t | x | C | X | XX | ‐
             all, comment, exif, iptc, preview, thumb, xmp, ICC Profile, SideCar, RawXMP, stdin/out

An example in the man page:

   $ exiv2 -et img1.jpg img2.jpg
   Extracts the Exif thumbnails from the two files into img1-thumb.jpg and img2-thumb.jpg.

Actually, I added the -f switch as recommended by @agriggio, resized to HD and it didn’t take long to run. No problems!

The earth-shattering results of my images and playing around with Shotcut – https://youtu.be/_EPgIMQgKVo

Obviously I have lots to learn. That is was a time lapse of opportunity, but rain was setting in, dinner was waiting and there were other tasks so it was a one-shot opportunity.

Thanks.

2 Likes

Thank you.

@lphilpot sorry to hear about your loss.
For giving Windows the needed power you may want to switch on the ultimate mode, which is preinstalled, but needs to be activated.

1 Like

That’s great! I was hoping you’d share. Good job IMO.

1 Like