Added write functionality for 2 and 4 bit tiff #1672

MathemanFlo · 2020-06-03T14:48:47Z

I added support for writing 2 and 4 bit tiffs in both modes: minisblack and miniswhite.
To do so I added the new option bitdepth to support 1, 2 and 4 bit tiff output.
I did not change the behavior of squash as it would break existing code but tried to be compatible with it in the sense that e.g. in case of 2 bit: values from 0 to 63 are treated as 0, 64 to 127 as 1 and so on. In fact one could merge the whole internal implementation of squash into bitdepth and further more bitdepth is flexible enough to allow for extensions to other bit depths and/or formats (e.g. png) in the future, too.
To implement this feature I added functions eightbit2twobit and eightbit2fourbit similiar to eightbit2onebit which could be merged into one function but I think this approach is more efficient as it avoids unnecessary branches and variables in the two and four bit case.

As this is my first contribution to libvips I welcome any feedback.

jcupitt · 2020-06-05T12:37:04Z

Thank you very much for this nice work @MathemanFlo!

I'm horribly busy this weekend, I'll try to look over it next Wednesday.

jcupitt · 2020-06-07T13:15:36Z

I had a quick look, it seems nice!

Regarding the API, how about deprecating squash and just using bitdepth?

You can tag operation arguments with VIPS_ARGUMENT_DEPRECATED (just add to flags) and the argument will no longer appear in docs or in things like intellisense, but will still work.

In the _build method, you can use vips_object_argument_isset() to see if the caller used the deprecated argument and use that to adjust the default of the non-deprecated one. Old code will continue to work, and new code will only see the new interface.

The no_subsample option to jpegsave does this, if you're curious.

MathemanFlo · 2020-06-08T06:56:01Z

Thank you for the positive feedback, your suggestions and hints. I definitely agree with you and I already started with removing squash from the internal code completely.
But I'm a bit unsatisfied with the behavior of the "new bitdepth":
If one specifies bitdepth then it only works for uchar and for float (if interpretation is lab and it has 3 bands, because this is the 2nd case of squash). In the uchar case it can convert to 1,2,4 bit and in the float case it squashes LAB down to LABQ. This does not sound logical to me and I would not expect this behavior from a library because the BITSPERSAMPLE tag depends also on the band format.
So if I want to save an image in 16 bits per sample and channel, I have to convert the image to short. The same holds for 32 bit where I have to convert to uint. If I have an image in uchar I can save in 1,2,4 bit per pixel using bitdepth, If I have and image in short or something else I have to convert it to uchar first and then use bitdepth.
I think the key problem is that there is now a second option to change the bit depth which isn't really compatible with ordinary way of using band formats.

Further more if you don't mind I would also like to add a load functionality for such files. Currently only paletted tiffs and 1 bit tiffs are supported. To support these kind of files no API changes are necessary. The code is similar to rtiff_parse_onebit in tiff2vips.c and shouldn't be a big deal.

jcupitt · 2020-06-08T10:00:26Z

Sure, a matching loader is a good idea.

pngsave needs a similar API for saving as 1/2/4 bits, though it includes palette generation and dithering, which I suppose you don't need. It would obviously be good to have the same API for both operations if possible.

MathemanFlo · 2020-06-08T10:35:08Z

Al right. Thank you. Then I will clarify the cases in the code and add several sanity check in order to warn users in case of a wrong usage. For the conversion from LAB to LABQ what should be the bit depth? This does not make too much sense in this context...

MathemanFlo · 2020-06-09T09:35:58Z

Now I added the things you suggested, the read functionality, improved the performance and added warnings in case the parameter was used incorrectly. The only remaining issue is the LAB issue described above.

jcupitt · 2020-06-16T10:54:36Z

libvips/foreign/tiff2vips.c

+                for( z = 0; z < 4; z++ ) {
+					/* The grey shade is the value four times concatenated */
+                        q[x] = (bits & 0xC0) | ((bits & 0xC0) >> 2) 
+							| ((bits & 0xC0) >> 4) | (bits >> 6);


You could write this as:

VipsPel fourbits = (bits & 0xC0) | ((bits & 0xC0) >> 2); q[x] = fourbits | (fourbits >> 4);

Though I've no idea if that gives a useful speedup.

Yes one could do that for sure and I also think it is easier to understand. I'll change it in the next iteration of my proposal. Any further ideas?

Oh I guess you could remove the & as well if you build from the low bits up:

VipsPel twobits = bits >> 6; VipsPel fourbits = twobits | (twobits << 2); VipsPel eightbits = fourbits | (fourbits << 4); q[x] = eightbits;

jcupitt · 2020-06-16T11:00:02Z

libvips/foreign/tiffsave.c

@@ -605,10 +622,23 @@ vips_foreign_save_tiff_buffer_init( VipsForeignSaveTiffBuffer *buffer )
 * MINISBLACK TIFFs where black is a 0 bit, but if you set @miniswhite, it
 * will use 0 for a white bit. Many pre-press applications only work with
 * images which use this sense. @miniswhite only affects one-bit images, it
- * does nothing for greyscale images. 
+ * does nothing for greyscale images. Consider @bitdepth for depths 1, 2 and 4.
 *
 * Set @squash to squash 3-band float CIELAB images down to 8-bit CIELAB.


This line needs to change too.

Yes you are right but I didn't know how to properly change squash for the CIE stuff as commented above.

jcupitt · 2020-06-16T11:03:51Z

libvips/foreign/vips2tiff.c

@@ -195,7 +195,8 @@
 * 	- add PAGENUMBER support
 * 23/5/20
 * 	- add support for subifd pyramid layers
- */
+ * 8/6/20
+ * 	- add bitdepth support for 2 and 4 bit greyscale images


Needs a */ to close the comment.

jcupitt · 2020-06-16T11:05:01Z

libvips/foreign/vips2tiff.c

@@ -1039,13 +1041,13 @@ ready_to_write( Wtiff *wtiff )

 	/* "squash" float LAB down to LABQ.
 	 */
-	if( wtiff->squash &&
+	if( wtiff->bitdepth == 1 &&


How about allowing 8 as well here? Setting bitdepth to 1 to save 32-bit float as 8-bit uint seems unintuitive.

Ok then l will do it as follows: Allow bitdepth = 8 for CIE LAB but not for integral images because there the proper way to change bitdepth from e.g. 16 to 8 is to change the type

Sounds good! You'll need to allow 1 as well, of course.

Why do I have to allow 1 as well? CIE LAB allows for squash to 8 bit. I don't get your point.

You set bitdepth = 1 if squash is on, so you'll need to allow that here for lab -> labq squash to still work (I think).

I see I did it differently, I set bitdepth to 8 in case squash is on.

But won't that break the old behaviour of squash writing 8-bit uchar as 1-bit? I'm probably confused.

No you are not. I'm the one who missed that and now I get your point. Thank you so much.

jcupitt · 2020-06-16T11:14:52Z

libvips/foreign/vips2tiff.c

-		!(wtiff->ready->Coding == VIPS_CODING_NONE && 
+	if( wtiff->bitdepth && (wtiff->bitdepth == 1 || wtiff->bitdepth == 2 
+		|| wtiff->bitdepth == 4 )
+        && !(wtiff->ready->Coding == VIPS_CODING_NONE &&


I'd just use wtiff->bitdepth here, you've checked the values in the test above.

jcupitt · 2020-06-16T11:21:21Z

libvips/foreign/vips2tiff.c

+	if( !wtiff->miniswhite ) { //avoid unnecessary branches
+		for( x = 0; x < n; x++ ) {
+			bits <<= 2;
+			bits |= p[x] >> 6;


You could do this as:

bits |= (p[x] >> 6) ^ mask;

where mask is wtiff->miniswhite ? 3 : 0 somewhere outside the loop. It would save the loop copy-paste.

jcupitt · 2020-06-16T11:21:59Z

libvips/foreign/vips2tiff.c

+	if( !wtiff->miniswhite ) { //avoid unnecessary branches
+		for( x = 0; x < n; x++ ) {
+				bits <<= 4;
+				bits |= p[x] >> 4;


Same here. I don't like the loop duplication :(

jcupitt · 2020-06-16T11:28:33Z

Sorry about the delay, I finally read the whole thing again carefully. You did a great job!

I should have started a review, I ended up with more comments than I expected. There are some small style issues too (line length, tab size, etc.) but they can be fixed later, of course.

We'll need some tests too. Would you like to add some, or shall I?

png, spng and ppm save have a squash param as well, they should also be swapped to bitdepth. I can do this if you like.

I agree about bitdepth being confusing for (for example) saving a uint32 image as a uint16 TIFF. Unfortunately I can't think of a name that expresses the meaning more clearly. fractional_bitdepth? It looks very ugly. Do you have any ideas on this?

MathemanFlo · 2020-06-16T14:38:50Z

Thank you very much for all your amazing comments and for your help. I think I addressed the things you mentioned and I also tried to fix the style issues.
I also added some test files which can be integrated in the test suit. 256palette is used to check whether the quantization to 4 or 16 is correct and the other two files are the result of this procedure and can be used for loading checks.
I also took a look into the code for saving png and ppm. Probably I'm confused but I couldn't find any code or documentation to squash pngs or do you want me to add this functionality to png? The ppm code has the squash parameter and if you want to I change everything to bitdepth but I do not really see the point in that (besides consistency) because only bitdepth 1 is supported and no other fractional bit depths are available.

I can't think of a name that expresses the meaning more clearly. fractional_bitdepth? It looks very ugly. Do you have any ideas on this?

To be honest I have no better name and maybe bitdepth is not the worst one because it would allow features like the conversion you described.

bitdepth_check_files.zip

jcupitt · 2020-06-16T15:18:09Z

You're right, squash was planned for pngsave, but hasn't gone in yet.

ppmsave can only save one bit images, but it should use the same API for consistency. We can fix that later.

Thanks for the test images, let's revise the test suite for this in another PR.

It all looks fine now, barring some tiny formatting. Let's merge so the fuzzers can start work.

Nice work!

jcupitt · 2020-06-17T13:42:13Z

I reworked the things to pack and unpack bits to make them a bit smaller and neater, it might be worth you testing git master again to make sure I didn't foul up.

I added some tests as well, and credited you in the changelog.

MathemanFlo · 2020-06-18T13:11:01Z

Thank you for your awesome work and your incredible support. Initially I had exactly aneightbit2nbit function, too but my initial implementation was too different for the three cases until I refactored everything but then I forgot about it. Anyway I read your code carefully and I found a bug in the left over bit handling.
Line 1314 should look like *q++ = (bits ^ mask) << (8 - ((x & pixel_mask) << (wtiff->bitdepth/2)));
To reproduce the problem use the following example
examples.zip
I propose 1bit.tif and so on as a replacement for the current tests because they have a width of 259 instead of 256.
If it helps you I can also add bitdepth to ppmsave...

we were not adjusting for pixel size, thanks MathemanFlo see #1672 (comment)

jcupitt · 2020-06-18T13:55:59Z

Ooop, you're right, I've comitted a fix, thanks!

(don't be too impressed, I'm not normally so responsive, we're just trying to clear the backlog ready for 8.10)

jcupitt · 2020-06-18T13:56:47Z

I added bitdepth to ppmsave already. I think 8.10 just needs testing now, and there's one final PR to land.

MathemanFlo force-pushed the master branch from f44be41 to b55285d Compare June 4, 2020 06:50

Florian Heinrich added 2 commits June 9, 2020 11:23

Added read and write functionality for 2 and 4 bit greyscale TIF images

035e0bc

Merge branch 'master' of https://github.com/libvips/libvips

17a9bf3

MathemanFlo force-pushed the master branch from b55285d to 17a9bf3 Compare June 9, 2020 09:26

jcupitt reviewed Jun 16, 2020

View reviewed changes

Florian Heinrich added 2 commits June 16, 2020 16:21

Fixed issues with bitdepth

00bd91a

Fixed typo in bitdepth code

0f0e8cb

jcupitt merged commit 63b8e16 into libvips:master Jun 16, 2020

jcupitt added a commit that referenced this pull request Jun 18, 2020

fix flush of left-over bits

b2feb48

we were not adjusting for pixel size, thanks MathemanFlo see #1672 (comment)

Uh oh!

Added write functionality for 2 and 4 bit tiff #1672

Added write functionality for 2 and 4 bit tiff #1672

Uh oh!

Conversation

MathemanFlo commented Jun 3, 2020

Uh oh!

jcupitt commented Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcupitt commented Jun 7, 2020

Uh oh!

MathemanFlo commented Jun 8, 2020

Uh oh!

jcupitt commented Jun 8, 2020

Uh oh!

MathemanFlo commented Jun 8, 2020

Uh oh!

MathemanFlo commented Jun 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcupitt commented Jun 16, 2020

Uh oh!

MathemanFlo commented Jun 16, 2020

Uh oh!

jcupitt commented Jun 16, 2020

Uh oh!

jcupitt commented Jun 17, 2020

Uh oh!

MathemanFlo commented Jun 18, 2020

Uh oh!

jcupitt commented Jun 18, 2020

Uh oh!

jcupitt commented Jun 18, 2020

Uh oh!

Uh oh!

jcupitt commented Jun 5, 2020 •

edited

Loading