Drop bzip support

RT has some weird code to support bz2 compressed raw files. That is a relict from old times where storage was expensive and compressed raw formats have been rare. For that reason I dropped support for bz2 compressed raw files with this commit. If there are objections please tell me. For the rare case that you have bz2 compressed raw files, just uncompress them.

Ingo

3 Likes

Storage isnā€™t expensive, but network bandwidth still is :slight_smile:

Of course network bandwidth is expensive. But who uses bz2 to compress raw files and expects that rt opens this bz2 compressed files like uncompressed files?

I have a lot of raws compressed to bzip2 (crw created with chdk with powershots).

I didnā€™t even know that raw therapee supported them!

Anyway i think nowadays the best way to compress a raw is to convert to dng (for me in a lossless way), even if there isnā€™t an open source converter (i use dng converter with wine), so i think there is no problem dropping support (in fact i plan too to convert bz2 compressed raw to dng, one day or anotherā€¦)

@heckflosse to substantiate this choice, I can add that from ages ago, I never made Windows builds with bzip option. Nobody ever complained or asked for.

As explained e.g. here, this might not be the best idea. @patdavid is way too polite in this post, it would at least deserve some words written in ALL CAPS, e.g. the DONā€™T! This is Just a heads-up, really not trying to start a flame war.

To say the truth I donā€™t understand much.

CRW Files from chdk were without exif, and without matrix information (in fact they are really raw data, even without a container)

First i convert them to dng with dng4ps-2 (whilst it still work, it tries also to gather exif datas from the corrisponding jpeg), then i convert them still to dng with dng converter (to gain preview and compress zeros, they are 10 bit/pixel but written as 16 bit/pixel, and even compress them a bit with what i think and iā€™m quite sure itā€™s a lossless algorithm).

Anyway i have bluray copies of bzipped2 originals.

Last but not least, i have tried to point rawtherapee 4.2.448 (i think the last i could compile with ubuntu 14.04) to that raws, and they are not even previewed or showed.

Taking a look at the project page, seems it even has been updated :wink: (i used to run the windows version via wine, with added color matrix for my Powershot A720) : GitHub - crass/dng4ps2: Converter for Canon PowerShot raw images created with CHDK to DNG

That is a corner case in which conversion to DNG is fine. After all, there is no meta data that could be lost or corrupted. :slight_smile:

As @houz said, this is valid in that special case. In all other cases the following applies.

In general, a conversion might lose data, either metadata or image data (when using closed source software for conversion, you donā€™t even know what data is translated correctly and what is messed up, and due to missing documentation of raw formats by the OEMs, for open source you would only know what data is translated, and nothing about the missing pieces). This is probably not an issue at the moment, since your raw converter does not make use of this metadata, but a later version may use this data to further improve the results. There might even be data in the raw where dng has no option for storing it. Or, there might even be data stored in the dng that the raw converter does not use, while it uses this data if the original raw is used. Furthermore, without checking you even do not know if the conversion was successful. So you would have to compare the raw data in the original and in the dng file before deleting the original raw. This may be particularly difficult when number formats change, e.g. 10 bit to 16 bit, as you mentioned.

One example: Some cameras store lens correction data in the raw files. If this data is lost, you might not be able to use automatic lens correction with some raw converters. While this might be a minor problem, it is just a handy example. A more drastic one: The raw to dng converter might introduce lossy compression and you are not aware, and for the first pictures you donā€™t notice, but later you may find out with a difficult image (e.g. a night shot with a lot of noise) and the original raws are already deleted. Noise reduction is way more flexible on raw data.

Please hold! This topic is about dropping bz2 support from rawtherapee. Are there any objections?
If no, we can close the topic as ā€˜Solvedā€™ and move the dng related anwers to a new topic.

2 Likes
  1. The BZ2 code is buggy,
  2. In fact it does not seem to work,
  3. Nobody Iā€™m aware of has ever mentioned using it, trying to use it or reported bugs regarding it,
  4. Nor has anyone offered to fix the bugs,
  5. Therefore it seems best to drop it.
4 Likes