[GHA] Added build-images step into ci-test workflow #7821

wilsonccccc · 2021-03-05T20:25:12Z

split current docker publish workflow into 2 parts:

Added new step build-images to ci-test workflow for further usage by LBT merging. It has condition by github.event_name == 'push' && needs.prepare.outputs.test-rust == 'true' and only push pre_build_image
Removed auto branch task from docker-publish workflow, which ci-test workflow will be also triggered by auto branch

Motivation

(Write your motivation for proposed changes here.)

Have you read the Contributing Guidelines on pull requests?

(Write your answer here.)

Test Plan

tested new step will skip by pr pull
tested with /canary, current ci-test passed and new build-images step pushed pre_master_018999af image succeeded

Related PRs

(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/diem/diem/tree/master/developers.diem.com, and link to your PR here.)

If targeting a release branch, please fill the below out as well

Justification and breaking nature (who does it affect? validators, full nodes, tooling, operators, AOS, etc.)
Comprehensive test results that demonstrate the fix working and not breaking existing workflows.
Why we must have it for V1 launch.
What workarounds and alternative we have if we do not push the PR.

rexhoffman · 2021-03-05T20:28:27Z

.github/workflows/ci-test.yml

+    runs-on: ubuntu-latest-xl
+    continue-on-error: false
+    needs: prepare
+    if: ${{ github.event_name == 'push' && needs.prepare.outputs.test-rust == 'true' }}


nice this is a good way to prevent prs from trying to use secrets.

.github/workflows/ci-test.yml

rexhoffman · 2021-03-08T18:24:59Z

.github/workflows/ci-test.yml

+          key_password: ${{ secrets.DOCKERHUB_KEY_PASSWORD }}
+      - name: pre build docker images
+        run: |
+          if  [[ ! "$CHANGES_TARGET_BRANCH" =~ "^release-[0-9|.]+$" ]] && [[ ! "$CHANGES_TARGET_BRANCH" =~ "^gha-test-[0-9|.]+$" ]] ; then


you probably want to build if the target branch is master so you can use the images in cluster test.

oh yes, this step is copied from docker-publish workflow which has some other branches we need to filter. Actually we do not need this step here because ci-test and LBT both be triggered by auto and canary by push only. Thx for pointing out

You're right, as long as the step is not triggered on prs we're fine.

rexhoffman · 2021-03-08T18:27:32Z

.github/workflows/docker-publish.yml

    continue-on-error: false
    env:
      TAG: github-1
-      DOCKERHUB_ORG: libra


rexhoffman · 2021-03-08T18:35:04Z

.github/workflows/docker-publish.yml

-      - name: setup_aws_erc_login
+      - name: setup_aws_ecr_login


ahh a name fix. nice.

wilsonccccc · 2021-03-08T18:55:33Z

/canary

Closes: #7821

github-actions · 2021-03-08T19:29:16Z

Cluster Test Result

Test runner setup time spent 264 secs
Compatibility test results for land_e568d43c ==> land_13f48b07 (PR)
1. All instances running land_e568d43c, generating some traffic on network
2. First validator land_e568d43c ==> land_13f48b07, to validate storage
3. First batch validators (14) land_e568d43c ==> land_13f48b07, to test consensus
4. Second batch validators (15) land_e568d43c ==> land_13f48b07, to upgrade rest of the validators
5. All full nodes (30) land_e568d43c ==> land_13f48b07, to finish the network upgrade, time spent 517 secs
all up : 935 TPS, 4856 ms latency, 5600 ms p99 latency, no expired txns, time spent 250 secs
Logs: http://kibana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-03-08T19:09:11Z',to:'2021-03-08T19:29:14Z'))
Dashboard: http://grafana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/d/performance/performance?from=1615230551000&to=1615231754000
Validator 1 logs: http://kibana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-03-08T19:09:11Z',to:'2021-03-08T19:29:14Z'))&_a=(columns:!(log),query:(language:kuery,query:'kubernetes.pod_name:"val-1"'),sort:!(!('@timestamp',desc)))

Repro cmd:

./scripts/cti --tag land_e568d43c --cluster-test-tag land_13f48b07 -E RUST_LOG=debug -E BATCH_SIZE=15 -E UPDATE_TO_TAG=land_13f48b07 --report report.json --suite land_blocking_compat

🎉 Land-blocking cluster test passed! 👌

bors-libra · 2021-03-08T19:29:21Z

☀️ Canary successful

wilsonccccc · 2021-03-08T21:57:55Z

/land

Closes: diem#7821

github-actions · 2021-03-08T22:32:20Z

Cluster Test Result

Test runner setup time spent 248 secs
Compatibility test results for land_726fb419 ==> land_b1452ed4 (PR)
1. All instances running land_726fb419, generating some traffic on network
2. First validator land_726fb419 ==> land_b1452ed4, to validate storage
3. First batch validators (14) land_726fb419 ==> land_b1452ed4, to test consensus
4. Second batch validators (15) land_726fb419 ==> land_b1452ed4, to upgrade rest of the validators
5. All full nodes (30) land_726fb419 ==> land_b1452ed4, to finish the network upgrade, time spent 549 secs
all up : 936 TPS, 4848 ms latency, 5500 ms p99 latency, no expired txns, time spent 250 secs
Logs: http://kibana.ct-1-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-03-08T22:11:59Z',to:'2021-03-08T22:32:19Z'))
Dashboard: http://grafana.ct-1-k8s-testnet.aws.hlw3truzy4ls.com/d/performance/performance?from=1615241519000&to=1615242739000
Validator 1 logs: http://kibana.ct-1-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-03-08T22:11:59Z',to:'2021-03-08T22:32:19Z'))&_a=(columns:!(log),query:(language:kuery,query:'kubernetes.pod_name:"val-1"'),sort:!(!('@timestamp',desc)))

Repro cmd:

./scripts/cti --tag land_726fb419 --cluster-test-tag land_b1452ed4 -E RUST_LOG=debug -E BATCH_SIZE=15 -E UPDATE_TO_TAG=land_b1452ed4 --report report.json --suite land_blocking_compat

🎉 Land-blocking cluster test passed! 👌

[GHA] Added build-images step into ci-test workflow Closes: diem#7821 [GHA] Merge LBT workflow into ci-test Closes: diem#7856 [GHA]unify LBT switch job into LBT Closes: diem#7939

[GHA] Added build-images step into ci-test workflow Closes: #7821 [GHA] Merge LBT workflow into ci-test Closes: #7856 [GHA]unify LBT switch job into LBT Closes: #7939 Closes: #7997

Closes: diem#7821

diem-cla-bot bot added the cla-signed label Mar 5, 2021

wilsonccccc requested review from bmwill, rexhoffman and sausagee March 5, 2021 20:25

wilsonccccc changed the title ~~Added docker publish step into ci-test workflow~~ Added build-images step into ci-test workflow Mar 5, 2021

wilsonccccc changed the title ~~Added build-images step into ci-test workflow~~ [GHA]Added build-images step into ci-test workflow Mar 5, 2021

wilsonccccc changed the title ~~[GHA]Added build-images step into ci-test workflow~~ [GHA] Added build-images step into ci-test workflow Mar 5, 2021

wilsonccccc force-pushed the add_publish branch 2 times, most recently from 19de925 to 289e87a Compare March 5, 2021 20:31

rexhoffman reviewed Mar 5, 2021

View reviewed changes

sausagee reviewed Mar 6, 2021

View reviewed changes

.github/workflows/ci-test.yml Outdated Show resolved Hide resolved

wilsonccccc force-pushed the add_publish branch from 289e87a to 8c8217a Compare March 6, 2021 01:46

wilsonccccc requested review from rexhoffman and sausagee March 8, 2021 17:22

rexhoffman reviewed Mar 8, 2021

View reviewed changes

.github/workflows/ci-test.yml Outdated Show resolved Hide resolved

rexhoffman reviewed Mar 8, 2021

View reviewed changes

.github/workflows/ci-test.yml Outdated Show resolved Hide resolved

rexhoffman reviewed Mar 8, 2021

View reviewed changes

.github/workflows/docker-publish.yml

continue-on-error: false

env:

TAG: github-1

DOCKERHUB_ORG: libra

Copy link

Contributor

rexhoffman Mar 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good fix

rexhoffman reviewed Mar 8, 2021

View reviewed changes

wilsonccccc force-pushed the add_publish branch from 8c8217a to d24fcd4 Compare March 8, 2021 18:54

bors-libra pushed a commit that referenced this pull request Mar 8, 2021

[GHA] Added build-images step into ci-test workflow

13f48b0

Closes: #7821

wilsonccccc requested a review from rexhoffman March 8, 2021 19:29

rexhoffman approved these changes Mar 8, 2021

View reviewed changes

[GHA] Added build-images step into ci-test workflow

b1452ed

Closes: diem#7821

bors-libra force-pushed the add_publish branch from d24fcd4 to b1452ed Compare March 8, 2021 22:32

bors-libra closed this in b1452ed Mar 8, 2021

bors-libra merged commit b1452ed into diem:master Mar 8, 2021

wilsonccccc deleted the add_publish branch March 8, 2021 22:39

wilsonccccc mentioned this pull request Mar 19, 2021

[GHA]Merge LBT workflow into ci-test workflow #7997

Merged

bors-libra pushed a commit that referenced this pull request Mar 19, 2021

cherry-pick from main

c04b939

[GHA] Added build-images step into ci-test workflow Closes: #7821 [GHA] Merge LBT workflow into ci-test Closes: #7856 [GHA]unify LBT switch job into LBT Closes: #7939 Closes: #7997

bors-libra pushed a commit that referenced this pull request Mar 24, 2021

cherry-pick from main

e8baf8c

[GHA] Added build-images step into ci-test workflow Closes: #7821 [GHA] Merge LBT workflow into ci-test Closes: #7856 [GHA]unify LBT switch job into LBT Closes: #7939 Closes: #7997

bors-libra pushed a commit that referenced this pull request Mar 24, 2021

cherry-pick from main

99736d3

[GHA] Added build-images step into ci-test workflow Closes: #7821 [GHA] Merge LBT workflow into ci-test Closes: #7856 [GHA]unify LBT switch job into LBT Closes: #7939 Closes: #7997

bors-libra pushed a commit that referenced this pull request Mar 24, 2021

cherry-pick from main

e8c8f20

[GHA] Added build-images step into ci-test workflow Closes: #7821 [GHA] Merge LBT workflow into ci-test Closes: #7856 [GHA]unify LBT switch job into LBT Closes: #7939 Closes: #7997

zgfzgf pushed a commit to zgfzgf/diem that referenced this pull request Jun 8, 2021

[GHA] Added build-images step into ci-test workflow

2e604d8

Closes: diem#7821

[GHA] Added build-images step into ci-test workflow #7821

[GHA] Added build-images step into ci-test workflow #7821

Uh oh!

Conversation

wilsonccccc commented Mar 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related PRs

If targeting a release branch, please fill the below out as well

Uh oh!

rexhoffman Mar 5, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rexhoffman Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

wilsonccccc Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

rexhoffman Mar 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rexhoffman Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

rexhoffman Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

wilsonccccc commented Mar 8, 2021

Uh oh!

github-actions bot commented Mar 8, 2021

Uh oh!

bors-libra commented Mar 8, 2021

Uh oh!

wilsonccccc commented Mar 8, 2021

Uh oh!

github-actions bot commented Mar 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wilsonccccc commented Mar 5, 2021 •

edited

Loading

rexhoffman Mar 8, 2021 •

edited

Loading