Support multi-arch Docker images #521

Juneezee · 2021-03-12T14:29:21Z

Currently the migrate/migrate Docker image only supports linux/amd64 OS/Arch.

This pull request adds support for multi CPU-architectures by using Docker Buildx to build the images.

@dhui In addition, I have also added the GitHub Actions configuration for building and pushing the multi-arch images automatically on new tags. You will need to add your Docker Hub's username and personal access token into the repository secrets (see Action Secrets documentation).

Reference: docker/setup-qemu-action for the list of supported platforms

Juneezee · 2021-03-12T15:05:16Z

.github/workflows/docker.yaml

+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}


@dhui In order for GitHub Actions to build and push the image automatically to Docker Hub, please add DOCKERHUB_USERNAME and DOCKERHUB_TOKEN into the repository secrets.

dhui

Forgive my ignorance, but I thought docker images would run on different CPU architectures. Is this just an optimization?

Would support for multiple architectures make the images larger? It looks like it won't make larger images and just allows multiple images to be associated with the same tag.

.github/workflows/docker.yaml

Juneezee · 2021-03-13T06:52:41Z

Forgive my ignorance, but I thought docker images would run on different CPU architectures. Is this just an optimization?

Would support for multiple architectures make the images larger? It looks like it won't make larger images and just allows multiple images to be associated with the same tag.

@dhui No, docker images would not run on different CPU architectures unless the image itself support multi-architecture. Take the golang image for example, if you do docker pull golang:latest on a linux/amd64 machine, Docker will only pull the linux/amd64 image. Therefore, it will not make the image larger.

Currently migrate/migrate only supports linux/amd64

dhui

Use CircleCI. Artifacts aren't produced by CircleCI yet... See: #252

Juneezee · 2021-03-14T19:01:23Z

Use CircleCI. Artifacts aren't produced by CircleCI yet... See: #252

There is a blog on CircleCI's website explaining how to build multi-arch Docker images on CircleCI. I find it too cumbersome that we have to install Buildx manually and run binfmt ourselves whenever we want to build multi-arch images on CircleCI.

On the other hand, Docker provides QEMU (docker/setup-qemu-action) and Buildx (docker/setup-buildx-action) integration into GitHub Actions. We don't need to reinvent the wheel and worry about the configuration at all.

May I know why we can't use GitHub Actions? It is free for public repositories

dhui · 2021-03-15T05:05:19Z

May I know why we can't use GitHub Actions?

CircleCI provides many more credits for open source projects compared to other providers, which allows us to run tests for PRs. The 2k minutes Github Actions provides would be exhausted after a measly 25 runs... (each build takes ~40m and we build for 2 Go versions, so that's ~80m per run)

Juneezee · 2021-03-15T05:23:34Z

The 2k minutes Github Actions provides would be exhausted after a measly 25 runs... (each build takes ~40m and we build for 2 Go versions, so that's ~80m per run)

The 2k minutes only count for private repositories. GitHub Actions is free for public repositories.

From https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions

dhui · 2021-03-15T06:16:27Z

Have you used over 2k Github Actions minutes on any of your public repos? The language isn't clear to me. e.g. free != unlimited

If you've confirmed that Github Actions usage is unlimited for public repos, would you be willing to migrate our build system to Github Actions? e.g. linter, tests (needs to spin up docker images), PR checks, code coverage, and artifact uploads (docker, package cloud, & github releases)

Juneezee · 2021-03-15T08:29:50Z

Have you used over 2k Github Actions minutes on any of your public repos? The language isn't clear to me. e.g. free != unlimited

Here is the GitHub Actions job logs that run in one of my public repository: docker-multiarch/docker-buildx. The job used 1 minutes and 9 seconds in total.

Looking into the organization Billing settings, as shown in the image below, the "1 minutes and 9 seconds" was not counted towards the 2k minutes quota.

I know the documentation is a little bit of confusing. But I can be sure that the GitHub Actions usage is free and unlimited for public repositories as I have seen many public repositories using it, to name a few

If you've confirmed that Github Actions usage is unlimited for public repos, would you be willing to migrate our build system to Github Actions? e.g. linter, tests (needs to spin up docker images), PR checks, code coverage, and artifact uploads (docker, package cloud, & github releases)

Sure, I can help with that

coveralls · 2021-03-18T07:09:09Z

Pull Request Test Coverage Report for Build 234

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 56.547%

Totals
Change from base Build 226:	0.0%
Covered Lines:	3308
Relevant Lines:	5850

💛 - Coveralls

Juneezee · 2021-03-18T11:00:49Z

@dhui I have migrated from Circle CI to GitHub Actions. This includes:

Uses golangci/golangci-lint-action for linting
Run tests on Go 1.15 and 1.16
Generate coverage using shogo82148/actions-goveralls
Build and push multi-architecture Docker images with docker/setup-qemu-action, docker/setup-buildx-action, and docker/build-push-action
Build binaries then upload them as artifacts for a new release

EDIT: The checks are failing because I have removed all CircleCI related files.

Juneezee · 2021-03-18T11:02:45Z

.github/workflows/ci.yaml

+      - name: Login to Docker Registry
+        uses: docker/login-action@v1
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}


@dhui You will need to add the username and personal access token of the Dockerhub account that will be used to push the image as secrets. See https://docs.github.com/en/actions/reference/encrypted-secrets

dhui · 2021-05-28T21:22:44Z

See also this PR, which uses GoReleaser with Docker's buildx.

Also, I think there is a 2K minute limit with GitHub Actions. I'm not sure if golangci-lint pays for more minutes, but their actions/tests take a lot less time to run than ours do. e.g. they're not running integration tests with a bunch of different DBs

Juneezee · 2021-05-28T21:28:55Z

Also, I think there is a 2K minute limit with GitHub Actions. I'm not sure if golangci-lint pays for more minutes, but their actions/tests take a lot less time to run than ours do. e.g. they're not running integration tests with a bunch of different DBs

@dhui The quota only counts for private repositories.

Juneezee · 2021-05-29T08:59:09Z

@dhui If PR #570 is merged first, then I'll update this PR to only perform the testing on GitHub Actions

Signed-off-by: Eng Zer Jun <[email protected]>

This commit also uses GitHub Actions to build and push multi-architecture Docker image to Dockerhub. Signed-off-by: Eng Zer Jun <[email protected]>

Juneezee · 2021-08-04T07:44:28Z

@dhui PR #570 has been merged, and this pull request is now obsolete. I have opened PR #605 to migrate from CircleCI to GitHub Actions.

Juneezee marked this pull request as draft March 12, 2021 14:47

Juneezee commented Mar 12, 2021

View reviewed changes

Juneezee marked this pull request as ready for review March 12, 2021 15:05

dhui requested changes Mar 13, 2021

View reviewed changes

.github/workflows/docker.yaml Outdated Show resolved Hide resolved

Juneezee requested a review from dhui March 13, 2021 07:20

dhui requested changes Mar 14, 2021

View reviewed changes

Juneezee marked this pull request as draft March 16, 2021 04:38

Juneezee marked this pull request as ready for review March 18, 2021 10:53

Juneezee requested a review from dhui March 18, 2021 11:00

Juneezee commented Mar 18, 2021

View reviewed changes

dhui mentioned this pull request Apr 20, 2021

Does new release is planned soon? #546

Closed

dhui mentioned this pull request May 6, 2021

Refactor pgx and postgres db drivers #549

Open

Juneezee added 2 commits May 29, 2021 16:59

build: support multi-architecture Docker image

7610d7e

Signed-off-by: Eng Zer Jun <[email protected]>

ci: migrate from CircleCI to GitHub Actions

a2da2ec

This commit also uses GitHub Actions to build and push multi-architecture Docker image to Dockerhub. Signed-off-by: Eng Zer Jun <[email protected]>

Juneezee closed this Aug 4, 2021

Juneezee deleted the multi-arch-docker-image branch August 4, 2021 07:44

		username: ${{ secrets.DOCKERHUB_USERNAME }}
		password: ${{ secrets.DOCKERHUB_TOKEN }}

Support multi-arch Docker images #521

Support multi-arch Docker images #521

Uh oh!

Conversation

Juneezee commented Mar 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Juneezee Mar 12, 2021

Choose a reason for hiding this comment

Uh oh!

dhui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Juneezee commented Mar 13, 2021

Uh oh!

dhui left a comment

Choose a reason for hiding this comment

Uh oh!

Juneezee commented Mar 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhui commented Mar 15, 2021

Uh oh!

Juneezee commented Mar 15, 2021

Uh oh!

dhui commented Mar 15, 2021

Uh oh!

Juneezee commented Mar 15, 2021

Uh oh!

coveralls commented Mar 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 234

💛 - Coveralls

Uh oh!

Juneezee commented Mar 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Juneezee Mar 18, 2021

Choose a reason for hiding this comment

Uh oh!

dhui commented May 28, 2021

Uh oh!

Juneezee commented May 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Juneezee commented May 29, 2021

Uh oh!

Juneezee commented Aug 4, 2021

Uh oh!

Uh oh!

Juneezee commented Mar 12, 2021 •

edited

Loading

Juneezee commented Mar 14, 2021 •

edited

Loading

coveralls commented Mar 18, 2021 •

edited

Loading

Juneezee commented Mar 18, 2021 •

edited

Loading

Juneezee commented May 28, 2021 •

edited

Loading