Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@gdsotirov
Copy link

Set proper values for properties VERSION and SOVERSION of target snappy, so that CMake produces the link libsnappy.so.1 to the shared library libsnappy.so.1.1.5 that is built. This link was available in 1.1.4 (with autotools) and it is still necessary for the run time linker to work. Its absence leads to run time errors like "error while loading shared libraries: libsnappy.so.1: cannot open shared object file: No such file or directory".

Set proper values for properties `VERSION` and `SOVERSION` of target `snappy`, so that CMake produces the link `libsnappy.so.1` to the shared library `libsnappy.so.1.1.5` that is built. This link was available in 1.1.4 (with autotools) and it is still necessary for the run time linker to work. Its absence leads to run time errors like "error while loading shared libraries: libsnappy.so.1: cannot open shared object file: No such file or directory".
@ilovezfs
Copy link

@pwnall @alkis this same issue was reported to Homebrew: Homebrew/homebrew-core#15274

For 1.1.5 we switched from Autotools to CMake (because there was no release tarball and because the Autotools build is currently broken due to README not being found, as Makefile.am only produces README.tmp, not README). But apparently the switch to CMake has broken the hadoop build due to the missing libsnappy.1.dylib file.

@ilovezfs
Copy link

It seems this change will require all the reverse dependencies to be bumped again, unfortunately.

dyld: Library not loaded: /usr/local/opt/snappy/lib/libsnappy.1.1.5.dylib
  Referenced from: /usr/local/opt/rocksdb/lib/librocksdb_lite.5.6.dylib
  Reason: Incompatible library version: librocksdb_lite.5.6.dylib requires version 1.1.5 or later, but libsnappy.1.dylib provides version 1.0.0

I'm not entirely sure why, because that does not seem to be the case if I use an Autotools build.

@ilovezfs
Copy link

@gdsotirov is it possible Autotools is also not setting VERSION now?

@pwnall
Copy link
Member

pwnall commented Jul 11, 2017

I made a mess when I bumped the autotools SOVERSION, in 513df5f. I missed the memo about how libtool versions translate to soversions at https://github.com/google/snappy/blob/1.1.5/configure.ac#L6.

Given that the autotools SOVERSION for snappy 1.1.4 is 1.3.1, I propose re-releasing 1.1.5 as 1.4.0, with SOVERSION set to 1.4.0 for both CMake and autotools. Per the notice in the 1.1.5 release, 1.4.0 would be the last release with autotools support. Afterwards, the insanity can end.

(Quick) input on this proposal is welcome. I plan to make it happen fairly soon.

@pwnall
Copy link
Member

pwnall commented Jul 11, 2017

@ilovezfs If you prefer to revert to autotools until we figure out the CMake situation, I think you can get the tarball you'd need at https://github.com/google/snappy/archive/1.1.5.tar.gz

@gdsotirov
Copy link
Author

... Autotools build is currently broken due to README not being found, as Makefile.am only produces README.tmp, not README).

@ilovezfs Yes, for which I initially made PR #42. However, in 1.1.5 with Autotools I get libsnappy.so.0.4.1, which totally brakes the existing dependencies, because in 1.1.4 the library was libsnappy.so.1.3.1 and the link libsnappy.so.1 was present (I wrote about this to @pwnall by email).

It seems this change will require all the reverse dependencies to be bumped again, unfortunately.

@ilovezfs I'm really not sure I understand you on this.

@gdsotirov is it possible Autotools is also not setting VERSION now?

@ilovezfs With Autotools in 1.1.5 I'm getting libsnappy.so.0.4.1 and thus errors for missing libsnappy.so.1 link. I haven't investigated further how to change the SO version of libsnappy with Autotools as I decided to switch to CMake for building Snappy given the notice that 1.1.5 would be the last version to support Autotools.

Given that the autotools SOVERSION for snappy 1.1.4 is 1.3.1, I propose re-releasing 1.1.5 as 1.4.0, with SOVERSION set to 1.4.0 for both CMake and autotools.

@pwnall This would work as well as long as the libsnappy.so.1 link is present, because this is all the run-time linker cares about :-)

@gdsotirov
Copy link
Author

I have wrote to @pwnall by email, but with CMake in 1.1.5 there is also the problem that LIBDIR isn't configurable (e.g. I cannot set /usr/lib64 for example). For now I'm overcoming this problem by changing LIBDIR in package creation script after make install, but it would be good if this is fixed as well.

@ilovezfs
Copy link

@ilovezfs I'm really not sure I understand you on this.

If I adopt this PR, then the dependents (AKA reverse dependencies) break with this error:

dyld: Library not loaded: /usr/local/opt/snappy/lib/libsnappy.1.1.5.dylib
  Referenced from: /usr/local/opt/folly/lib/libfolly.57.dylib
  Reason: Incompatible library version: libfolly.57.dylib requires version 1.1.5 or later, but libsnappy.1.1.5.dylib provides version 1.0.0

@gdsotirov
Copy link
Author

@ilovezfs I understand you now, but I do not use Apple, so I'm not able to check and find remedy for this myself. I however guess the difference is coming from SOVERSION property being set to just ${PROJECT_VERSION_MAJOR}. However, if I set it the same as VERSION or leave it unset (in which case it should be same as VERSION as CMake's documentation states) the result is that symbolic link libsnappy.so.1 is not created and on Linux the run-time linker reports errors of not being able to find the library. I'm thus not sure what's the best solution here, but I checked another library that I'm building with CMake - libproxy - and it seems to be using the same approach of setting VERSION to full version and SOVERSION to just major version - see /libproxy/cmake/libproxy.cmk at line 21. I'm open for the best solution here that is independent of the platform, but I'm only able to verify and confirm on Linux (Slackware Linux to be precise).

@ilovezfs
Copy link

@gdsotirov to be clear, I don't mind if we need to rebuild the rev-deps, if that's the right thing to do. I'm just noting that this change has consequences beyond the creation of the symlink.

@gdsotirov
Copy link
Author

@ilovezfs And I got your point, but I'm really not sure how to get CMake to create the necessary for Linux's run-time linker libsnappy.so.1 symbolic link with SOVERSION (explicitly or implicitly) set to full project version (instead of just major version as I suggest). In SOVERSION is set to full project version CMake only generates libsnappy.so symlink, which is not enough and the absence of libsnappy.so.1 breaks (reverse) dependencies in Linux, so it's not just about the symlink.
I'm not that much familiar with CMake, but with Autotools in Snappy 1.1.4 (and earlier) this link was available, so I'm searching a solution for Snappy 1.1.5 with CMake and for me this the the patch I suggested in this PR. The other option is to create the link in the package generation script, but I believe that this is function of the build system (like for all other library packages I'm building).

@ilovezfs
Copy link

@gdsotirov yeah I think your solution is likely correct.

@pwnall
Copy link
Member

pwnall commented Jul 12, 2017

I looked into version numbers some more. On OSX, I used otool -L libsnappy.dylib and on Linux I used objdump -p

Linux looks easy -- the only number that matters (as far as I can tell) is the major SOVERSION. However, in OSX, the situation is a bit more complicated.

1.1.4 autotools: /usr/local/lib/libsnappy.1.dylib (compatibility version 5.0.0, current version 5.1.0)
1.1.5 autotools: /usr/local/lib/libsnappy.0.dylib (compatibility version 5.0.0, current version 5.1.0)
1.1.5 cmake: @ rpath/libsnappy.1.1.5.dylib (compatibility version 1.1.5, current version 0.0.0)

1.4.0 would look like this:

autotools (libtool version [5:0:4]): /usr/local/lib/libsnappy.1.dylib (compatibility version 6.0.0, current version 6.0.0)
cmake (with PR #45): @ rpath/libsnappy.1.dylib (compatibility version 1.0.0, current version 1.4.0)

After looking at the numbers (discussed below), I'm also considering releasing a 1.1.6, which would look like this:

1.1.6 autotools (libtool version [2:6:1]): /usr/local/lib/libsnappy.1.dylib (compatibility version 3.0.0, current version 3.6.0)
1.1.6 cmake (with PR #45): @ rpath/libsnappy.1.dylib (compatibility version 1.0.0, current version 1.1.6)

What I don't like about both alternatives: according to the dylib compatibility / current versions, upgrading from an autotools build to a cmake build looks like an incompatible downgrade. The only way to avoid this seems to be re-releasing 1.1.5 as 6.0.0, which seems excessive.

1.4.0's advantage over 1.1.6 is that the autotools version numbers aren't disrupted, so users can stay on the autotools train for one more version without any problem. However, the version number 1.1.6 better represents what's actually happening (bugfix release) than 1.4.0, and it seems like 1.4.0 isn't a good long-term solution anyways, as the next release (1.4.1 or 1.5.0) would be cmake-only, and would break people upgrading from autotools release 1.4.0.

@ilovezfs @gdsotirov Do you know if the compatibility / current version numbers reported by otool matter? If not, I'm more inclined to release 1.1.6. If they do matter, is there any workable long-term solution asides from 6.0.0?

@gdsotirov
Copy link
Author

Linux looks easy -- the only number that matters (as far as I can tell) is the major SOVERSION

@pwnall Yes, because SONAME (derived form SOVERSION) is all that the dynamic linker/loader cares about, so setting CMake's SOVERSION property to ${PROJECT_VERSION_MAJOR} seems the right thing, The major version number should normally be incremented only when ABI changes.

Do you know if the compatibility / current version numbers reported by otool matter?

@pwnall Unfortunately, I'm not familiar with OSX and otool (shall correct this in future), but compatibility/current version numbers look weird to me like this. I presume it the best when dynamic library version matches software/project version, because it simplifies a lot of things for distribution packages and developers like me.

Anyway, I more for 1.1.6 release, but you pointed out some problems with both 1.1.6 and 1.4.0, so I'm really not sure. How are the compatibility/current version numbers defined? And can this be changed, so there is no apparent downgrade after the switch to CMake?

@ilovezfs
Copy link

It seems most combinations you can dream up are possible:

iMac-TMP:~ joe$ ls -l /usr/lib/libcurl.*
lrwxr-xr-x  1 root  wheel      15 Dec 25  2015 /usr/lib/libcurl.3.dylib -> libcurl.4.dylib
-rwxr-xr-x  1 root  wheel  785312 Mar 12  2016 /usr/lib/libcurl.4.dylib
lrwxr-xr-x  1 root  wheel      15 Dec 25  2015 /usr/lib/libcurl.dylib -> libcurl.4.dylib
iMac-TMP:~ joe$ otool -L /usr/lib/libcurl.dylib | head -2
/usr/lib/libcurl.dylib:
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
iMac-TMP:~ joe$ ls -l /usr/lib/libpcre.*
-rwxr-xr-x  1 root  wheel  424928 Mar 12  2016 /usr/lib/libpcre.0.dylib
lrwxr-xr-x  1 root  wheel      15 Dec 25  2015 /usr/lib/libpcre.dylib -> libpcre.0.dylib
iMac-TMP:~ joe$ otool -L /usr/lib/libpcre.dylib | head -2
/usr/lib/libpcre.dylib:
	/usr/lib/libpcre.0.dylib (compatibility version 1.0.0, current version 1.1.0)
iMac-TMP:~ joe$ ls -l /usr/lib/libz.*
lrwxr-xr-x  1 root  wheel      12 Dec 25  2015 /usr/lib/libz.1.1.3.dylib -> libz.1.dylib
lrwxr-xr-x  1 root  wheel      12 Dec 25  2015 /usr/lib/libz.1.2.5.dylib -> libz.1.dylib
-rwxr-xr-x  1 root  wheel  177728 Mar 12  2016 /usr/lib/libz.1.dylib
lrwxr-xr-x  1 root  wheel      12 Dec 25  2015 /usr/lib/libz.dylib -> libz.1.dylib
iMac-TMP:~ joe$ otool -L /usr/lib/libz.dylib | head -2
/usr/lib/libz.dylib:
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
iMac-TMP:~ joe$ ls -l /usr/lib/libarchive.*
-rwxr-xr-x  1 root  wheel  457344 Apr 18 19:17 /usr/lib/libarchive.2.dylib
lrwxr-xr-x  1 root  wheel      18 Dec 25  2015 /usr/lib/libarchive.dylib -> libarchive.2.dylib
iMac-TMP:~ joe$ otool -L /usr/lib/libarchive.dylib | head -2
/usr/lib/libarchive.dylib:
	/usr/lib/libarchive.2.dylib (compatibility version 9.0.0, current version 9.2.0)
iMac-TMP:~ joe$ ls -l /usr/lib/libncurses.*
-rwxr-xr-x  1 root  wheel  529440 Mar 12  2016 /usr/lib/libncurses.5.4.dylib
lrwxr-xr-x  1 root  wheel      20 Dec 25  2015 /usr/lib/libncurses.5.dylib -> libncurses.5.4.dylib
lrwxr-xr-x  1 root  wheel      20 Dec 25  2015 /usr/lib/libncurses.dylib -> libncurses.5.4.dylib
iMac-TMP:~ joe$ otool -L /usr/lib/libncurses.dylib | head -2
/usr/lib/libncurses.dylib:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
iMac-TMP:~ joe$ ls -l /usr/lib/liblzma.*
-rwxr-xr-x  1 root  wheel  283312 Mar 12  2016 /usr/lib/liblzma.5.dylib
lrwxr-xr-x  1 root  wheel      15 Dec 25  2015 /usr/lib/liblzma.dylib -> liblzma.5.dylib
iMac-TMP:~ joe$ otool -L /usr/lib/liblzma.dylib | head -2
/usr/lib/liblzma.dylib:
	/usr/lib/liblzma.5.dylib (compatibility version 6.0.0, current version 6.3.0)
iMac-TMP:~ joe$

@pwnall
Copy link
Member

pwnall commented Jul 12, 2017

@ilovezfs Any chance you could verify that a project built against snappy 1.1.4 (i think the original bug has hadoop?) can pick up and use the snappy in your PR? Alternatively, branch rc in https://github.com/pwnall/snappy has the patch that I plan to apply to release 1.1.6. (this is still discussed internally)

@ilovezfs
Copy link

@pwnall testing your branch with rocksdb, before and after the change, it requires rocksdb to be rebuilt, but then it's fine.

Before:

iMac-TMP:homebrew-core joe$ otool -L /usr/local/Cellar/rocksdb/5.5.1/lib/librocksdb.dylib 
/usr/local/Cellar/rocksdb/5.5.1/lib/librocksdb.dylib:
	/usr/local/opt/rocksdb/lib/librocksdb.5.6.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/snappy/lib/libsnappy.1.1.5.dylib (compatibility version 1.1.5, current version 0.0.0)
	/usr/local/opt/gflags/lib/libgflags.2.2.dylib (compatibility version 2.2.0, current version 2.2.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
	/usr/local/opt/lz4/lib/liblz4.1.dylib (compatibility version 1.0.0, current version 1.7.5)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

After:

iMac-TMP:homebrew-core joe$ otool -L /usr/local/Cellar/rocksdb/5.5.1/lib/librocksdb.dylib 
/usr/local/Cellar/rocksdb/5.5.1/lib/librocksdb.dylib:
	/usr/local/opt/rocksdb/lib/librocksdb.5.6.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 1.0.0, current version 1.1.6)
	/usr/local/opt/gflags/lib/libgflags.2.2.dylib (compatibility version 2.2.0, current version 2.2.1)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
	/usr/local/opt/lz4/lib/liblz4.1.dylib (compatibility version 1.0.0, current version 1.7.5)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

pwnall added a commit to pwnall/snappy that referenced this pull request Jul 13, 2017
The migration from autotools to CMake in 1.1.5 wasn't as smooth as
intended. The SONAME / SOVERSION were broken in both build systems,
causing breakages in systems that upgraded from snappy 1.1.4 to 1.1.5,
as reported in Homebrew/homebrew-core#15274
and google#45.
@pwnall
Copy link
Member

pwnall commented Jul 13, 2017

@ilovezfs Thank you very much for the test results! Do you happen to know if a rebuild be required for an upgrade from 1.1.4 to 1.1.6?

@ilovezfs
Copy link

@pwnall yes, because with 1.1.4 the build was with Autotools, so now with 1.1.6, you get

iMac-TMP:homebrew-core joe$ brew test rocksdb
Testing rocksdb
==> Using the sandbox
==> /usr/bin/clang++ test.cpp -o db_test -v -std=c++11 -stdlib=libc++ -lstdc++ -lz -lbz2 -L/usr/local/Cellar/rocksdb/5.4.6/lib -lrocksdb_lite -L/usr/local/opt/snappy/lib -lsnappy -L/usr/local/opt/lz4/lib 
==> ./db_test
Last 15 lines from /Users/joe/Library/Logs/Homebrew/rocksdb/test.02.db_test:
2017-07-13 04:19:52 -0700

./db_test

dyld: Library not loaded: /usr/local/opt/snappy/lib/libsnappy.1.dylib
  Referenced from: /usr/local/opt/rocksdb/lib/librocksdb_lite.5.4.dylib
  Reason: Incompatible library version: librocksdb_lite.5.4.dylib requires version 5.0.0 or later, but libsnappy.1.dylib provides version 1.0.0

@ilovezfs
Copy link

If you mean an Autotools build of 1.1.6, that currently fails

iMac-TMP:homebrew-core joe$ brew instal -s snappy
==> Using the sandbox
==> Cloning https://github.com/pwnall/snappy.git
Updating /Users/joe/Library/Caches/Homebrew/snappy--git
==> Checking out revision 1b62ea916f21202b4e56e89bd60b7d4211adf8d3
==> ./autogen.sh
==> ./configure --prefix=/usr/local/Cellar/snappy/1.1.6-rc1
==> make install
Last 15 lines from /Users/joe/Library/Logs/Homebrew/snappy/03.make:
mkdir: /usr/local/Cellar/snappy/1.1.6-rc1/lib: File exists
 /bin/sh ./libtool   --mode=install /usr/bin/install -c   libsnappy.la '/usr/local/Cellar/snappy/1.1.6-rc1/lib'
 /usr/bin/install -c -m 644 snappy.pc '/usr/local/Cellar/snappy/1.1.6-rc1/lib/pkgconfig'
 /usr/bin/install -c -m 644 snappy.h snappy-sinksource.h snappy-stubs-public.h snappy-c.h '/usr/local/Cellar/snappy/1.1.6-rc1/include'
 /usr/bin/install -c -m 644 ChangeLog COPYING INSTALL NEWS ./README format_description.txt framing_format.txt '/usr/local/Cellar/snappy/1.1.6-rc1/share/doc/snappy'
install: ./README: No such file or directory
make[1]: *** [install-dist_docDATA] Error 71
make[1]: *** Waiting for unfinished jobs....
libtool: install: /usr/bin/install -c .libs/libsnappy.1.dylib /usr/local/Cellar/snappy/1.1.6-rc1/lib/libsnappy.1.dylib
libtool: install: (cd /usr/local/Cellar/snappy/1.1.6-rc1/lib && { ln -s -f libsnappy.1.dylib libsnappy.dylib || { rm -f libsnappy.dylib && ln -s libsnappy.1.dylib libsnappy.dylib; }; })
libtool: install: /usr/bin/install -c .libs/libsnappy.lai /usr/local/Cellar/snappy/1.1.6-rc1/lib/libsnappy.la
libtool: install: /usr/bin/install -c .libs/libsnappy.a /usr/local/Cellar/snappy/1.1.6-rc1/lib/libsnappy.a
libtool: install: chmod 644 /usr/local/Cellar/snappy/1.1.6-rc1/lib/libsnappy.a
libtool: install: ranlib /usr/local/Cellar/snappy/1.1.6-rc1/lib/libsnappy.a
make: *** [install-am] Error 2

If I intervene, and fix that, then the linkage ends up like this if we don't rebuild rocksdb:

iMac-TMP:homebrew-core joe$ brew test rocksdb
Testing rocksdb
==> Using the sandbox
==> /usr/bin/clang++ test.cpp -o db_test -v -std=c++11 -stdlib=libc++ -lstdc++ -lz -lbz2 -L/usr/local/Cellar/rocksdb/5.4.6/lib -lrocksdb_lite -L/usr/local/opt/snappy/lib -lsnappy -L/usr/local/opt/lz4/lib 
==> ./db_test
Last 15 lines from /Users/joe/Library/Logs/Homebrew/rocksdb/test.02.db_test:
2017-07-13 04:31:44 -0700

./db_test

dyld: Library not loaded: /usr/local/opt/snappy/lib/libsnappy.1.dylib
  Referenced from: /usr/local/opt/rocksdb/lib/librocksdb_lite.5.4.dylib
  Reason: Incompatible library version: librocksdb_lite.5.4.dylib requires version 5.0.0 or later, but libsnappy.1.dylib provides version 3.0.0

and

iMac-TMP:homebrew-core joe$ otool -L /usr/local/opt/snappy/lib/libsnappy.1.dylib
/usr/local/opt/snappy/lib/libsnappy.1.dylib:
	/usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 3.0.0, current version 3.6.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

@ilovezfs
Copy link

ilovezfs commented Jul 13, 2017

(Also, I noticed that unlike the Autotools build, which installed both the dylib and static library, the CMake build only installs the dylib. That doesn't affect me personally, but there may(?) be some users who still would like the static library available.)

@Optiligence
Copy link

Installing dynamic and static builds alongside each other with CMake is handled in #38.

@pwnall
Copy link
Member

pwnall commented Jul 13, 2017

I landed this PR in 548501c. @gdsotirov @ilovezfs Thank you very much for your help!

@ilovezfs
Copy link

@pwnall Thanks! Homebrew/homebrew-core#15583.

@gdsotirov
Copy link
Author

I landed this PR in 548501c. @gdsotirov @ilovezfs Thank you very much for your help!

@pwnall Thanks as well! I wish you could have considered also PR #44 for 1.1.6, because it's another problem for me with 1.1.5 :-)

@gdsotirov
Copy link
Author

No need for this to remain open.

@gdsotirov gdsotirov closed this Jul 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants