ci: arm docker builds #4804

ethomson · 2018-09-11T18:22:42Z

Docker can leverage QEMU to run images on other platforms! Introduce an arm32 based build.

ethomson · 2018-09-12T10:25:45Z

This. Is. Blowing. My. Mind. 🤯

ethomson · 2018-09-12T10:36:01Z

2018-09-12T10:22:44.6331866Z Operating system version:
2018-09-12T10:22:44.6649758Z     Linux 2c96ecf832f7 4.15.0-1022-azure #22~16.04.1-Ubuntu SMP Thu Aug 16 10:31:05 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

ghuntley · 2018-09-12T11:19:22Z

shiftkey · 2018-09-12T12:51:08Z

neithernut · 2018-09-12T14:38:00Z

It says it could not bind to ::1. Maybe they (CI provider) do't provide IPv6 support on arm for some reason?

ethomson · 2018-09-12T16:38:33Z

It says it could not bind to ::1. Maybe they (CI provider) do't provide IPv6 support on arm for some reason?

Not sure what's going on. This is running in an emulator in docker, so there's a lot of strange things that could be going on. I pushed up some changes to the proxy so that it will try to bind only on 127.0.0.1 in case it is the IPv6 problem.

neithernut · 2018-09-12T17:12:29Z

Interestingly, something still tries to bind to ::1. And all Linux-based tests are now failing.

ethomson · 2018-09-12T17:15:28Z

/rebuild

libgit2-azure-pipelines · 2018-09-12T17:15:33Z

Okay, @ethomson, I started to rebuild this pull request.

neithernut · 2018-09-12T17:28:17Z

At least in the latest test the proxy server does bind to 127.0.0.1:

[2018-09-12 17:18:08] com.microsoft.tfs.tools.poxy.Poxy: Starting server on 127.0.0.1:8118

Still, something tries to bind to ::1, so it's probably another process. Maybe it's just a deceptive error message and it's actually the process trying to connect to the proxy which is printing the error? This would at least explain why the other tests fail all of a sudden.

ethomson · 2018-09-12T17:42:58Z

Still, something tries to bind to ::1, so it's probably another process.

Yes, I agree. It's a red herring in this, it appears. I can repro this locally - I thought I had all the tests passing last night. So either I misunderstood what I saw last night (likely) or I broke something between now and then (also possible).

ethomson · 2018-09-12T17:48:03Z

Aha. Running java directly yields a big fat clue:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000004000e18dd4, pid=17, tid=0x000000404d339200
#
# JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-aarch64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x3e3dd4]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /build/hs_err_pid17.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

This appears to happen on any jar. 😞

ethomson · 2018-09-12T18:23:44Z

Aha - this happens on bionic too with OpenJDK 11. However it errors with:

qemu: Unsupported syscall: 168

Right before the segfault. (I suspect that xenial does, too, and I just missed that message.) Right, then, let's try gcj. The poxyproxy is doing so precious little that it might just work.

ethomson · 2018-09-12T19:56:23Z

/rebuild

libgit2-azure-pipelines · 2018-09-12T19:56:27Z

Okay, @ethomson, I started to rebuild this pull request.

neithernut · 2018-09-12T21:06:20Z

.vsts-ci.yml

+      qemu: 'true'
+      imageName: 'libgit2/xenial-arm32:test'
+      environmentVariables:
+			  SKIP_PROXY_TESTS: true


Looks like the wrong kind of whitespace. CI complains about this line, too.

neithernut · 2018-09-12T21:06:28Z

.vsts-ci.yml

+      qemu: 'true'
+      imageName: 'libgit2/xenial-arm64:test'
+      environmentVariables:
+			  SKIP_PROXY_TESTS: true


ethomson · 2018-09-12T21:15:37Z

Thanks, @neithernut. In my haste I forgot to clean up tabs vs spaces.

…

On Wed, Sep 12, 2018 at 10:06 PM, Julian Ganz ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In .vsts-ci.yml <#4804 (comment)>: > @@ -56,6 +56,30 @@ jobs: CMAKE_OPTIONS=-DUSE_HTTPS=mbedTLS -DSHA1_BACKEND=mbedTLS LEAK_CHECK=valgrind +- job: linux_xenial_arm32 + displayName: 'Linux (Xenial; arm32)' + pool: + vmImage: 'Ubuntu 16.04' + steps: + - template: ci/vsts-docker.yml + parameters: + qemu: 'true' + imageName: 'libgit2/xenial-arm32:test' + environmentVariables: + SKIP_PROXY_TESTS: true Looks like the wrong kind of whitespace. CI complains about this line, too. *pointing out the obvious* ------------------------------ In .vsts-ci.yml <#4804 (comment)>: > + qemu: 'true' + imageName: 'libgit2/xenial-arm32:test' + environmentVariables: + SKIP_PROXY_TESTS: true + +- job: linux_xenial_arm64 + displayName: 'Linux (Xenial; arm64)' + pool: + vmImage: 'Ubuntu 16.04' + steps: + - template: ci/vsts-docker.yml + parameters: + qemu: 'true' + imageName: 'libgit2/xenial-arm64:test' + environmentVariables: + SKIP_PROXY_TESTS: true Same here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4804 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABE-Hj7ynYYj4HMjadaeRMcFindOl1cqks5uaXdsgaJpZM4Wj1aA> .

neithernut

Pointing out the obvious again.

neithernut · 2018-09-13T11:37:16Z

.vsts-ci.yml

@@ -56,6 +56,32 @@ jobs:
       CMAKE_OPTIONS=-DUSE_HTTPS=mbedTLS -DSHA1_BACKEND=mbedTLS
       LEAK_CHECK=valgrind

+- job: linux_xenial_x86


Maybe you should include "gcc" in the name

neithernut · 2018-09-13T11:37:34Z

.vsts-ci.yml

+       CC=gcc
+       LEAK_CHECK=valgrind
+
+- job: linux_xenial_x86


... and "clang" here.

.vsts-ci.yml

ethomson · 2018-09-14T10:49:42Z

/rebuild

libgit2-azure-pipelines · 2018-09-14T10:49:47Z

Okay, @ethomson, I started to rebuild this pull request.

pks-t

This is very cool, thanks a lot for working on this! My only worry right now is runtime, as the new jobs seem like they'd run by default. Doesn't this heavily impact the feedback loop?

pks-t · 2018-09-28T12:30:47Z

.vsts-ci.yml

+      qemu: 'true'
+      imageName: 'libgit2/xenial-arm64:test'
+      environmentVariables: |
+       SKIP_PROXY_TESTS=true


Is this all supposed to be part of our "main" CI infrastructure? If so, by how much does it increase our CI build times? If the delay is quite noticable, I'd vote to have this as a nightly job instead, as we have said in past discussions.

I 100% agree. I was just hoping to iterate on these by putting them in the PR validation configuration so that they would get built. (There are a handful of other ways to do this, I could have set up a new pipelines configuration, but I'm lazy.)

pks-t · 2018-09-28T12:32:39Z

.vsts-ci.yml

@@ -57,7 +57,7 @@ jobs:
       LEAK_CHECK=valgrind

 - job: linux_xenial_arm32
-  displayName: 'Linux (Xenial; arm32)'
+  displayName: 'Linux (Xenial; arm32; GCC)'


Shouldn't these jobs then also say whether they use OpenSSL or mbedTLS, if we aim for consistency?

pks-t · 2018-09-28T12:35:38Z

tests/buf/oom.c

@@ -40,6 +41,9 @@ void test_buf_oom__grow(void)
 	cl_assert(git_buf_oom(&buf));

 	git_buf_dispose(&buf);
+#else
+    cl_skip();
+#endif


You know, we should just use our custom allocators here. Create a custom test allocater that only has $n$ bytes available, plug it in and then run this test. This should result in a "real" OOM without us having to trick the host. I also think it would be quite helpful in other tests to verify exceptional behaviour.

Obviously, that doesn't have to be part of this PR.

Ooooooh, that's a good idea.

Addressing this in a separate pullrequest - forthcoming.

pks-t · 2018-09-28T12:38:08Z

libgit2_clar.supp

+	...
+	fun:gcry_pk_sign
+	obj:*libssh2.so*
+}


I really cannot imagine that libbsh2 or libgcrypt is to blame. I mean, I've tried multiple times to debug this issue and see whether we fail to use the APIs correctly, without any finds. But I just cannot imagine them being so lenient with regards to memory leaks.

As the number of each grow, separate the CI build scripts from the YAML definitions.

Use multiarch arm32 and arm64 docker images to run Xenial-based images for those platforms. We can support all the tests on ARM32 and 64 _except_ the proxy-based tests. Our proxy on ARM seems regrettably unstable, either due to some shoddy dependencies (with native code?) or the JREs themselves. Run these platforms as part of our nightly builds; do not run them during pull request or CI validation.

Bind the proxy specifically to 127.0.0.1 instead of all addresses. This is not strictly necessary for operations, but having a potentially open proxy on a network is not a good idea.

Use Bionic so that we have a modern libssh2 (for communicating with GitHub). We've ported fixes to our Trusty-based amd64 images, but maintaining patches for multiple platforms is heinous.

Newer dependencies means newer places to leak!

On a 32-bit Linux systems, the value large enough to make malloc guarantee a failure is also large enough that valgrind considers it "fishy". Skip this test on those systems entirely.

We don't need two separate docker images for OpenSSL and mbedTLS. They've been combined into a single image `trusty-amd64` that supports both.

ethomson · 2018-10-21T11:49:42Z

I've also created a repository that containers the Dockerfiles that we use to create these images.

pks-t · 2018-10-25T09:00:12Z

Cool, this is great! Is there any particular reason why you feel like the Dockerfiles shouldn't be part of libgit2 directly? We already have a "ci" directory, so I'd feel like it's the perfect place for them to live in.

ethomson · 2018-10-25T09:36:18Z

Cool, this is great! Is there any particular reason why you feel like the Dockerfiles shouldn't be part of libgit2 directly? We already have a "ci" directory, so I'd feel like it's the perfect place for them to live in.

🤔 No. I have no preference either way. I'd be happy to move them into ci if that's what makes the most sense.

pks-t · 2018-10-25T10:15:39Z

I think it makes sense to have them as part of libgit2. It's not like they make any sense as standalone files outside of the scope of libgit2, and they are part of our CI setup. Furthermore, I don't think they're much of a maintenance burden and don't add much to libgit2's size. So only pros, no cons, I think.

ethomson force-pushed the ethomson/qemu-build branch 3 times, most recently from 0c98559 to 2eafcde Compare September 12, 2018 10:21

ethomson changed the title ~~ci: arm32 docker build~~ ci: arm docker builds Sep 12, 2018

ethomson force-pushed the ethomson/qemu-build branch from ded95e1 to e1d1803 Compare September 12, 2018 18:34

ethomson force-pushed the ethomson/qemu-build branch from 1e3a832 to fbc1820 Compare September 12, 2018 20:24

neithernut reviewed Sep 12, 2018

View reviewed changes

ethomson force-pushed the ethomson/qemu-build branch from fbc1820 to 98c4cff Compare September 12, 2018 21:14

ethomson force-pushed the ethomson/qemu-build branch 5 times, most recently from 53b4399 to ca36b88 Compare September 13, 2018 11:24

neithernut reviewed Sep 13, 2018

View reviewed changes

ethomson force-pushed the ethomson/qemu-build branch 4 times, most recently from 9200495 to 7c96eb8 Compare September 13, 2018 18:57

ethomson force-pushed the ethomson/qemu-build branch 2 times, most recently from 7c96eb8 to b869a8a Compare September 18, 2018 02:15

pks-t reviewed Sep 28, 2018

View reviewed changes

ethomson force-pushed the ethomson/qemu-build branch from b869a8a to 78ef950 Compare October 21, 2018 08:42

ethomson added 2 commits October 21, 2018 09:44

ci: move configuration yaml to its own directory

4ec597d

As the number of each grow, separate the CI build scripts from the YAML definitions.

ci: reorganize naming for consistency

28f0558

ethomson force-pushed the ethomson/qemu-build branch from 78ef950 to 543a694 Compare October 21, 2018 08:44

ethomson added 7 commits October 21, 2018 10:37

ci: bind specifically to localhost for proxy

f195395

Bind the proxy specifically to 127.0.0.1 instead of all addresses. This is not strictly necessary for operations, but having a potentially open proxy on a network is not a good idea.

ci: introduce nightly x86 linux builds

b244ea7

ci: use bionic for non-amd64 builds

d82800e

Use Bionic so that we have a modern libssh2 (for communicating with GitHub). We've ported fixes to our Trusty-based amd64 images, but maintaining patches for multiple platforms is heinous.

valgrind: extend the suppressions file

9cc4ba5

Newer dependencies means newer places to leak!

tests: don't run buf::oom on 32-bit systems

415a8ae

On a 32-bit Linux systems, the value large enough to make malloc guarantee a failure is also large enough that valgrind considers it "fishy". Skip this test on those systems entirely.

ci: use trusty-amd64 for openssl and mbedtls

7c55716

We don't need two separate docker images for OpenSSL and mbedTLS. They've been combined into a single image `trusty-amd64` that supports both.

ethomson force-pushed the ethomson/qemu-build branch from 57ef0f3 to 7c55716 Compare October 21, 2018 09:38

ethomson merged commit 671b244 into master Oct 21, 2018

ethomson deleted the ethomson/qemu-build branch October 26, 2018 13:37

snyk-bot mentioned this pull request Feb 23, 2020

[Snyk] Upgrade nodegit from 0.4.1 to 0.26.4 saurabharch/Breezeblocks#1

Open

snyk-bot mentioned this pull request Apr 22, 2020

[Snyk] Upgrade nodegit from 0.24.3 to 0.26.5 aminatakonate000/Graviton-App#4

Open

snyk-bot mentioned this pull request May 5, 2020

[Snyk] Upgrade nodegit from 0.24.3 to 0.26.5 Barnstorm-Online/ngp-openapi-generator#1

Open

ci: arm docker builds #4804

ci: arm docker builds #4804

Uh oh!

Conversation

ethomson commented Sep 11, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

ghuntley commented Sep 12, 2018

Uh oh!

shiftkey commented Sep 12, 2018

Uh oh!

neithernut commented Sep 12, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

neithernut commented Sep 12, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

libgit2-azure-pipelines bot commented Sep 12, 2018

Uh oh!

neithernut commented Sep 12, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

ethomson commented Sep 12, 2018

Uh oh!

libgit2-azure-pipelines bot commented Sep 12, 2018

Uh oh!

neithernut Sep 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethomson commented Sep 12, 2018 via email

Uh oh!

neithernut left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ethomson commented Sep 14, 2018

Uh oh!

libgit2-azure-pipelines bot commented Sep 14, 2018

Uh oh!

pks-t left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethomson commented Oct 21, 2018

Uh oh!

pks-t commented Oct 25, 2018

Uh oh!

ethomson commented Oct 25, 2018

Uh oh!

pks-t commented Oct 25, 2018

neithernut Sep 12, 2018 •

edited

Loading