Let me start by pointing out that I’m a newbie with respect to both RT & raw images. I’ve read the RawPedia section titled “Processing Profiles” but still a bit confused about how RT intends to operate. On most computer software used for editing just about anything there is a concept of saving your work, at certain points during the work-flow, with the idea that you can subsequently experiment without risk of losing what’s been saved. In that, you can easily return to the point you were when the work was saved.
As I understand it, which is certainly not authoritative, the work that is done editing images is recorded in automatically generated profiles (AGP) which are also called sidecar files and that nothing is ever written to the input file selected for editing (e.g., file containing raw image produced by the camera). While this is good from the point of view of preserving the raw images there is a need to save an updated image someplace/somehow. From experimentation it appears whenever I choose to edit an input file one of these AGP files gets created. This happens even before I do anything which could be called editing although I have come to realize that some edits are invariably done even before I start to make any decisions about what changes I’d like to make. Finally, it appears as though updates to this file are the main result of the work done during the editing session. However, I haven’t figured out how to control either what is written or when it is written.
If RT fails/crashes what should I expect this file to contain?
When you achieve some result that you’d like to retain, even if you are still experimenting with other ideas, how do you save something that you might like to think of as a version of the image?
I have figured out how to save an image in a standard format (e.g., jpeg, tiff, png) but these aren’t what you’d intend to use as input for RT. In fact, from what I’ve seen so far it looks like RT can only use the jpeg file as input and using a lossy format to save something for future editing is certainly not consistent with the idea of processing raw images.
I’ve also noticed that I can create my own profiles but this doesn’t look like a mechanism that was intended for the purpose of saving per image instances but rather for producing something that might generally apply to a certain camera or particular conditions that might be associated with certain kinds of photo shoots.
The sidecar files in RT are those files with the extension PP3. Besides the RAW file you find a PP3 which is saved, as far as I understand, when you leave the editing of a raw file, either by selecting the next raw from the film strip or going back to the explorer.
When you export a developed image via the batch queue or via direct export you can click the checkbox “save parameters together with the image”. In that case a second PP3 is created which is near the jpeg or tiff or whatever format you use.
If you then continue editing and export again, a new name for the exported file is generated, using a running number. That means the previous export is not overwritten. Together with this new image a new pp3 is saved too.
If you want to apply the settings of any version of such an image you can use the load button to load any pp3 from your disk.
And yes, saving the pp3 on your own saves a general use profile.
OK! Maybe it is simpler than I thought but still a bit odd. My confusion came about from the combination of reading the documentation and trying to experiment with new, to me, software that uses an unconventional UI (i.e., no menu bar and especially no “File” menu).
It looks like what needs to be kept in mind is that these processing profiles (sidecar files) are what is being produced by RT. In that, they are the work product that results from editing. Saving the profile is analogous to what I had in mind for saving the image. The big difference is that when it comes to wanting to reproduce an image that includes the saved revisions it is necessary to mate the profile (.pp3) file with the correct raw file. Furthermore, when RT is closed or another image is selected for editing an updated profile is always saved with the indicated naming convention which reflects the overall editing status at that point in time. In that, unlike what I’d call conventional software, RT does NOT bother to ask if you want to save your work. Therefore, when you save an intermediate result and it is the only result you are interested in retaining you end up with a second profile and you are responsible for keeping track of what result corresponds to each.
Does that sound like a correct interpretation?
My experimentation has also lead to another curious observation. The so-called filmstrip that is displayed in the top center panel contains a kind of thumbnail image for each of a set of recognizable image files in a selected folder. These images have the appearance of the .jpg files, as opposed to raw files, that are produced by the camera. As soon as they are selected for editing the appearance changes to that of a raw file. In this post I learned that the camera raw files include some number of jpg images. My guess is that this is one of the ways such secondary files, that as best I can tell cannot be accessed by an end user, are used.
Does this sound correct or is there a different explanation that as a beginner I’d like to understand?
There is no “File” menu as a typical workflow with raw files is to assume the raw file as read-only.
The idea is that all work stems from the raw file down to other processing (in case you were wondering, it’s not an “unconvential UI” in this regard, Adobe Lightroom operates the same way).
You are not asked to “save” your work because it is constantly being saved for you, in the form of your .pp3 files (the list of operations being applied to your raw image to get to where you left off).
Otherwise you are on the right track!
The raw file has an embedded JPG preview in it, and that is what you are seeing in thumbnail views (no other “secondary” file - the thumbnail is a part of the raw file itself).
OK! I’m beginning to understand but for me a more conventional approach would be to recognize that the .pp3 file is the object that is being manipulated, created, and ultimately that I might like to save for later reuse. In that sense, it would be logical to make it the file that is opened (i.e., Menu>File>Open>some-name.pp3). In this sense, it would be the raw file, that never changes but is needed, which is thought of as the sidecar. If I understand correctly, these profiles can be used with any raw file and this may be what’s driving the design. In that, they simply represent a set of operations that could be applied to any raw file. While generally speaking they may not be specific to a raw file, the ones I have in mind are specific to a raw file and I would prefer that there is a way for me to clearly specify which raw file of the many present in the file system. Based on my current understanding that is something that needs to be done outside of RT which begs for the inevitable human error that to me is odd. Such a convention will be peculiar to my work habit/conventions and not something that any other untrained person could be expected to grasp. I suppose this means that raw files are not to be shared and that everything I want to share should be converted to a standard format. The big problem with that idea is that RT isn’t able to edit such a file (i.e., use it for input).
Now that I know better what is intended I’ll try to adapt to RT. Many thanks for helping to straighten me out.
Well, if it would be implemented that way: what would you open when the raw files is there but was never opened before. In that case there is no PP3 to load. And in that moment using “create” would make thinks much more complicated.
And if you set the flag “save pp3 with the image” (sorry, can’t look the right naming of that checkbox as I’m not on my pc) than that exact version of the pp3 which was active while creating the jgeg/tiff/… is saved directly near the created file. So if you save your image you can save the relevant pp3 in the same moment.
I am learning that there are some complexities associated with this application that may not fit perfectly with the conventional model but there is a pretty simple answer to the question. It is called Menu>File>Open>New that is the conventional, and I might say intuitive, way to create a new instance of something when nothing yet exists.
However, this does imply that the profile rather than the image is what the user is focused on and intending to produce. Looking at it from my perspective which comes from lots of experience doing many different kinds of things on computers other than image processing this is what seems more natural but I now want to add image processing to that perspective and very much appreciate the help I’m getting here.
Well I can agree on that statement, But you are running a program that develops raw files. So that menu path would create a new raw file. However, this is done in the camera. That is the reason for the raw processor that you can’t create a file. You always just open it.
However, if this is the original intention of the developers I don’t know, but for me it seems to be logical to not have a file menu.
It looks like we’ve arrived at the source of the disparity. When I started using RT I also thought I was developing an image. However, it took me a while to figure out, and some help was needed doing it, that in fact I was NOT actually producing an image but rather what RT refers to as a profile. Since the profile is what RT is producing it is the profile (i.e., .pp3) file that conventional UI mechanisms would have you open and that is what I meant when mentioning the conventional Menu>File>Open>New (.pp3 file). Of course based on what is kept in existing profile files this would probably preclude the idea of selecting what you wanted to work on from thumbnails. The File Browser and Filmstrip do look like significant parts of RT that I’m sure are features appreciated by lots of users that I’m trying to become one of.
In the way of comparison, GIMP uses a conventional menu bar that has a File selection. I can open quite a variety of image files or, unlike RT, I start with nothing and create a New image. Like RT, when the original image comes from some other source, like a camera, such files are never altered. However, instead of the profile concept GIMP builds an entirely new rendering of the image in its own unique (i.e., non-standard) format. It only saves (i.e., Menu>File>Save [or “Save AS”]) files in this GIMP specific format. The term “Export” is used when the time comes to produce a result in one of many standard formats.
A big difference between GIMP and RT is that GIMP rewrites all of the basic image data into the working file (i.e., what can be repeatedly Opened and Saved for further revision) whereas RT avoids doing this and as I understand it only writes data that can be used to reprocess the image data that resides in a separate file. RT is clearly saving a lot of storage space at what I’d call the expense of making users devise a sufficiently precise work flow to know with certainty what separate raw image file has to be mated with a specific profile to reproduce a previously achieved outcome. This is aided by the RT naming convention but that is pretty vulnerable to unintended mistakes that might occur with storage management procedures. To be sure some user specific work flow is also needed with GIMP. While each GIMP specific file that is produced is capable of reproducing a specific image without needing other input, as in the case of RT, you can have a very difficult time figuring out what specific file (e.g., from your camera) corresponds to the image you are working on unless you’ve kept track of that yourself. This has a lot to do with GIMP’s extremely poor handling of metadata which is supposed to get fixed in the next stable release (Version 2.10) if and when that ever happens.
Ironically, perhaps, the way Rt, darktable, and other raw processors work is very similar to making a print from film, rather than the way that gimp or Photoshop works.
Think about your raw file as your negative, and the tiff/jpeg output from RT as your print. You negative doesn’t change, you always have the negative. You can always take your negative an put it in the enlarger, do some contrast adjustments, dodge and burn, etc etc, and end up with a print. No matter how many different prints you produce, you always have the same negative to start from. RT enhances this process with it’s pp3 sidecar file, which is essentially instructions on how to produce the “print” again and again.
When editing Nikon raw images (*.nef) files here and designating version file names, if possible I tend to just suffix the resulting (draft or final) image name with a date field (dsc_1259-20170904.tif) If I’m saving more than one revision during the same day (for which is rare), I then name something similar to “dsc_1260-20170904-01.tif, dsc_1260-20170904-02.tif, …”.
Since I sometimes use other raw editors, I’ll then add either “-rt-” for RawTherapee or “-nxd-” for Nikon’s Capture NX-D prior to the date field. (Lower case with underscores for spaces is best if you’re typing via command line, albeit another popular method is using camel case without underscores for spaces. Shrugs.)
For *.pp3 setting files, I haven’t mucked around with renaming any of them for designating revisions and they remain named as they were by RawTherapee, as it’s more time consuming unless I’ve have an “Oh My God… Great Photo!” (Then I may have multiple TIFF/JPEG along with their associated multiple PP3 files saved to the desktop or some other prominent folder.) If and when I do create a TIFF or JPEG, a PP3 settings file is created alongside the file, so in essence the saved PP3 file is a revision snapshot, granted lacking much description! (Would be nice to note somewhere, “Draft edit, colors look good, etc”, similar to a dated/timed CVS/SVN/GIT log entry.)
Shrugs concerning using GIT for version snapshots; I can barely grasp and utilize CVS/SVN as it is! I can just suffix a file name (eg. *.txt, dot config file, …) with a date field with far less thinking, far less typing and is much much quicker. (eg. readme-20170904.txt, .vimrc.20170904, …) From here, I can run a diff far quicker, for which I rarely do. For dot config or /etc files, I just suffix the filename with a date for which is automatically scripted and performed by VIM. I do use GIT for my one maintained BASH scripts for /home/roger/bin though, albeit sometimes fall behind on commits, and have yet to really have a need to search the GIT log history or utilized GIT for backups. GIT is just something here that sucks more from my time from my timeless life!
I change the file name to match the content. Also do it in ~/.config profiles folder where you can this way have multiple different pp3 files for the same shot. Manual work, but straight forward. This way:
It sounds to me like Roger is saying that .tiff is a good format for saving intermediate images.
I must admit that when I penned my prior remarks I thought that the only kind of files other than raw that RT could edit were of type jpeg. I’m not sure where that came from but was based on reading various documentation about raw editing. Particularly RawPedia. Because of the lossy compression jpeg would be undesirable for this purpose. Jpeg is desirable once you know you have a finished product.
Based on some new reading and experiments Roger’s suggestion looks like a good approach. I’ve now learned that RT seems to be able to edit, at least some, .tiff files. To the extent that RT can edit at least the .tiff files that it creates this looks like it could be a good way to save an intermediate result that you’d not want to lose but also want to retain the ability to continue editing it. I’d also like to thank Roger for elaborating on some of the details he finds useful.
Being able to edit exif data is a good answer for what I had in mind. Knowing that it is kept in the profile is especially good. While I suspect everyone invents file naming conventions as a way to include more information about the corresponding files there are limits to what can be done which I think are solved by being able to maintain exif data.
Usually, I prefer to archive TIFF image files when I can, for genealogy work or other. Although, 16 bit TIFF files can be quite large. As a last resort, archiving JPEG images.
Upon purchasing my Nikon D5600, I usually prefer archiving the camera’s ~20-25MB RAW image files, for which apparently is much better than the much larger TIFF files. (I then export a snapshot after editing the image to JPEG.) My rule of thumb, use RAW when I’m taking photos of something worthwhile looking at within a raw image editor for several hours. Use JPEG for all other photos.
