space efficient storage for a million EDG binaries #200
Description
the "million EDG binaries" (30MB zipped, 140MB raw) would compress well with git
transfer size would stay the same, but storage size would be much smaller = no need for amazon S3 server
migrate tarballs to git:
#!/bin/sh
if [ -d gitrepo ]; then
echo "error: folder exists: gitrepo. to run test again, run: rm -rf gitrepo"
exit 1
fi
mkdir gitrepo
git -C gitrepo init
# https://github.com/rose-compiler/rose/blob/weekly/src/frontend/CxxFrontend/EDG_VERSION
release_list="$(cat <<EOF
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.77.1
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.78.1
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.79.1
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.80.1
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.81.1
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.82.1
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.82.2
roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.82.3
EOF
)"
for release in $release_list
do
echo adding $release
[ -e $release.tar.gz ] || wget http://edg-binaries.rosecompiler.org/$release.tar.gz
[ -d $release ] || tar -xf $release.tar.gz
cp -r $release/* $release/.libs gitrepo/
# TODO use release date for commit + tag
git -C gitrepo add .
git -C gitrepo commit -m "$release"
git -C gitrepo tag "$release"
rm -rf $release
done
echo raw size
du -sh gitrepo/.git
echo
echo compressing ...
time git -C gitrepo gc
echo
echo compressed size
du -sh gitrepo/.git
echo
echo total size of tarballs
du -shc roseBinaryEDG-*.tar.gz | tail -n1raw size
247M gitrepo/.git
compressing ...
Enumerating objects: 35, done.
Counting objects: 100% (35/35), done.
Delta compression using up to 4 threads
Compressing objects: 100% (34/34), done.
Writing objects: 100% (35/35), done.
Total 35 (delta 15), reused 0 (delta 0), pack-reused 0
real 0m57.203s
user 0m52.688s
sys 0m3.180s
compressed size
34M gitrepo/.git
total size of tarballs
213M total
fetching a tarball would be as simple as
wget https://github.com/rose-compiler/edg-binaries/archive/roseBinaryEDG-5-0-x86_64-pc-linux-gnu-gnu-10-5.0.11.82.3.tar.gz
compression can be optimized by
compiling object code with the
-ffunction-sectionsand-fdata-sectionscompiler flags. This has the effect that if you 'insert' a function into a translation unit, the insertion does not cause all of the addresses to change across the whole object file.