A request for help: TIFF writing

CarVac · August 4, 2017, 4:34am

I recently discovered that TIFF writing in Filmulator has stopped working as expected, despite the TIFF writing code not having changed since before there was any GUI at all.

When I open a Filmulator-output TIFF in GIMP, it apparently has four pages to the TIFF, and only the first one has the image data, the other three don’t work.

When running tiffinfo on the file, it gives the following:

TIFF Directory at offset 0x8 (8)
  Image Width: 2920 Image Length: 4100
  Bits/Sample: 16
  Compression Scheme: None
  Photometric Interpretation: RGB color
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 3
  Planar Configuration: single image plane
  Make: Canon
  Model: Canon EOS 5D
  DateTime: 2017:08:01 16:12:15
  EXIFIFDOffset: 0xe4
TIFFFetchNormalTag: Warning, Incorrect count for "MakerNote"; tag ignored.
TIFFReadCustomDirectory: Warning, Wrong data type 3 for "PixelXDimension"; tag ignored.
TIFFReadCustomDirectory: Warning, Wrong data type 3 for "PixelYDimension"; tag ignored.
TIFFReadCustomDirectory: Warning, Unknown field with tag 40965 (0xa005) encountered.
TIFF Directory at offset 0xe4 (228)
  ExposureTime: 0.000625
  FNumber: 0.000000
  ExposureProgram: 3
  ISOSpeedRatings: 100
  ExifVersion: 0x30,0x32,0x32,0x31
  DateTimeOriginal: 2017:08:01 16:12:15
  DateTimeDigitized: 2017:08:01 16:12:15
  ComponentsConfiguration: 0x1,0x2,0x3,0x0
  ShutterSpeedValue: 10.625000
  ApertureValue: 2147483648.000000
  ExposureBiasValue: 0.000000
  MeteringMode: 0
  Flash: 16
  FocalLength: 0.000000
  UserComment: 0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0
  FlashpixVersion: 0x30,0x31,0x30,0x30
  ColorSpace: 1
  Tag 40965: 75992
  FocalPlaneXResolution: 3086.925781
  FocalPlaneYResolution: 3091.295166
  FocalPlaneResolutionUnit: 2
  CustomRendered: 0
  ExposureMode: 0
  WhiteBalance: 1
  SceneCaptureType: 0
TIFFReadDirectory: Warning, Unknown field with tag 513 (0x201) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 514 (0x202) encountered.
MissingRequired: TIFF directory is missing required "ImageLength" field.

Nowhere does it seem to show that there are any extra frames or anything…

Anyone have any idea why this might be? Filmulator’s TIFF writing code is in this file.

If we can’t figure this out we may try switching to another TIFF library, like maybe TinyTIFF…

Tobias · August 4, 2017, 7:33am

Are you sure the bug is in Filmulator and not in Gimp?

CarVac · August 4, 2017, 11:37am

You might be right.

The TIFF works fine in RawTherapee…

ggbutcher · August 4, 2017, 11:39pm

Before you indict libtiff, you might comment out the exiv2 code at the bottom of the routine, recompile, and generate a new test image. I don’t see anything out of order with your libtiff code.

CarVac · August 5, 2017, 12:03am

That worked. I suppose that our current strategy of copying all the exif over wholesale is what’s causing issues.

ggbutcher · August 5, 2017, 1:25am

I’ve gone back and forth about the whole exif enchilada, but I’ve found I really only need the essential exposure/focal length information carried forward. If I want to pick at the image exif-wise, I just open the raw; I put in a exiftool-based dialog box for that purpose.

Edit: Oh, TIFFs vex that approach; the libtiff-supported metadata is not camera-oriented.

Carmelo_DrRaw · August 5, 2017, 10:42am

I guess your problem is coming from the thumbnail(s) in the RAW file. I had to deal with the same problem in photoflow, and came up with a pretty simple solution with EXIV2. You can check the lines of code starting from here.

Hope this helps!

CarVac · August 5, 2017, 2:06pm

Interestingly, it didn’t help. I guess I’ll have to list all the tags and see which ones cause problems…

houz · August 6, 2017, 9:13am

You might want to look at darktable/exif.cc at master · darktable-org/darktable · GitHub and grep for dt_remove_exif_keys in there. We have a few places with lists of EXIF tags that may not be written to the exported images.

CarVac · December 17, 2020, 2:12pm

I, uh, finally got around to implementing this exif cleanup from darktable, but it seems like when I write exif data to a TIFF it still ends up having 4 subimages of some sort.

Unlike before, where the first image was the good one and the other three weren’t, now the first two both contain the full image, the third one is a corrupt thumbnail (below), and the fourth simply crashes GIMP.

At this point I’m not sure what to do besides disable TIFF exif writing again. The TIFFs crash Photoshop, which is obviously not desirable behavior.

ggbutcher · December 17, 2020, 2:38pm

Right now, I just use the rather dodge-ily-documented libtiff extensions to write a limited bit of exif:

github.com

butcherg/rawproc/blob/master/src/tiffimage.cpp#L230

    
      
          		buf = (unsigned char *) _TIFFmalloc(scanlinesize);
          		img = imagedata;
          
          
		for (unsigned row = 0; row < height; row++)
          		{
          			memcpy(buf, img, scanlinesize);
          			if (TIFFWriteScanline(tif, buf, row, 0) < 0) {
          				printf("TIFFWriteScanline got an error...\n");
          				TIFFError(NULL,NULL);
          				break;
          			}
          			img+=scanlinesize;
          		}
          
          
		TIFFWriteDirectory( tif );
          
          
		(void) TIFFClose(tif);
          		if (buf) _TIFFfree(buf);
          		return true;
          	}
          	else return true;

Are you using exiv2?

CarVac · December 17, 2020, 2:39pm

Yeah, I’m using exiv2. From what I gather it’s incompatible with libtiff.

I’ll try your method tonight probably.

kmilos · December 17, 2020, 2:51pm

There is also a line in dt saying you cannot write metadata via Exiv2 to an open TIFF file, you have to close it first and do it in a second pass. Or rather, you cannot write to a multi-page file, so you have to close after the first page, then add Exif, then append other pages.

CarVac · December 18, 2020, 2:31pm

I’m getting a few errors.

libtiff complains “cannot modify tag Compression while writing” even though I’m not modifying the tag.

The TIFFSetDirectory and TIFFSetField calls fail too, with libtiff complaining “unknown tag 34665” which is the TIFFTAG_EXIFIFD as in your code.

Any ideas?

ggbutcher · December 18, 2020, 2:42pm

Hmmm… I’m gonna guess right now that what you’re experiencing has to do with something you’re doing that I’m not doing before line 230…

I offer very few options for writing TIFFs, and compression is not one of them. Just a guess.

ggbutcher · December 18, 2020, 2:50pm

Back to exiv2, my understanding of its operation is that you build a data structure with the tags you want, then you apply that data structure to the image file with code that looks like this:

Exiv2::Image::AutoPtr image = Exiv2::ImageFactory::open(file);
image->setExifData(exifData);
image->writeMetadata();

from Exiv2 - Image metadata library and tools

This is creating new subimages in the file?

CarVac · December 20, 2020, 5:20am

Or something. I do not understand why it does this.

In any case, I got your method working to write the basic exposure info, thanks a lot!

ggbutcher · December 20, 2020, 4:49pm

After digging into the whole metadata thing a couple of years ago, I decided to rely on the raw file as the canonical source for all data related to the capture, and store only exposure information, if that, in my renditions. I used the various image libraries’ metadata writing functions, although jpeg and png required supplementary support to organize the tags in the IFDs.

The one bit of crucial metadata in my little world is the processing toolchain that describes how the rendition is produced from the source (raw) file, and I currently store that in EXIF:ImageDescription. One day I’ll explore using the XMP tags, but that’s a low priority right now.

I fiddled a bit with exiv2 in considering it for the upcoming rawproc 1.0. The main challenge was replacing or complementing all use of my image data C++ map structure, which also gets stuff like libraw-collected information and things I use internally. I’ll probably revisit when I resume consideration of how to embed the toolchain…

CarVac · January 30, 2021, 2:37pm

@ggbutcher

Seems like your TIFF writing code puts a floating point as the EXIF shutter speed tag, which should be rational…

I’m going to have to figure out how darktable manages to use exiv2 on TIFFs without breaking all sorts of things.

ggbutcher · January 30, 2021, 2:55pm

Ah, a to-do I’d forgotten about. I’m fairly certain liftoff has a way to write rationals, will try to look it up, seems we have internet at cabin-in-the-woods now…