Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Artifact Passing from Automate-Metrics to Build-Book #415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Mar 25, 2024
Merged

Artifact Passing from Automate-Metrics to Build-Book #415

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
c9e7baf
first pass at artifact passing
jukent Mar 20, 2024
530be5a
missed one
jukent Mar 20, 2024
5106b7f
pass 2
jukent Mar 20, 2024
315c76b
revert changes to nightly-build (to not deploy or get metrics), and s…
jukent Mar 20, 2024
bf828c4
rename build step
jukent Mar 21, 2024
2449155
reorder
jukent Mar 21, 2024
7804088
new placeholder code
jukent Mar 21, 2024
5bdc288
edit colorbar width, one more pass at publish-site metrics
jukent Mar 21, 2024
df27a32
does download of artifact happen inside build action?
jukent Mar 21, 2024
5a13bda
revert changes to automate metrics step
jukent Mar 21, 2024
4966b5f
fix zip job
jukent Mar 21, 2024
01927e7
oops indent
jukent Mar 21, 2024
b993a79
zip needs runs on, but then will it lose the metrics files?
jukent Mar 21, 2024
b135376
add zip step
jukent Mar 21, 2024
b10a4a5
rename
jukent Mar 21, 2024
b2b70b6
add automate metrics back to trigger-preview
jukent Mar 21, 2024
a163052
make strings
jukent Mar 21, 2024
bd38a4d
placeholder triggerbookbuild.yaml file
jukent Mar 21, 2024
fd7a2fc
fix inputs
jukent Mar 21, 2024
2f8e5da
rm curly brackets
jukent Mar 21, 2024
8f1e474
Merge pull request #129 from jukent/recovery-branch
jukent Mar 21, 2024
dbcbb0e
trigger_workflow input to book-build
jukent Mar 21, 2024
f9ba49d
comment out new input
jukent Mar 21, 2024
5728a8f
no ! # just #
jukent Mar 21, 2024
6ccd292
rm trigger-book-build
jukent Mar 21, 2024
e60716b
fix upload path
jukent Mar 22, 2024
7f3ec92
remove comment to use new build-book input
jukent Mar 23, 2024
85db33e
just try a full path for trigger_workflow
jukent Mar 25, 2024
a15092f
rm full path
jukent Mar 25, 2024
61ee7ad
use soon to be new workflow_conclusion as empty, rather than default …
jukent Mar 25, 2024
02ce165
Update .gitignore
jukent Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/get-metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
FOUNDATIONS_ID = '281776420'
COOKBOOKS_ID = '324070631'


# Access Secrets
PRIVATE_KEY_ID = os.environ.get('PRIVATE_KEY_ID')
# Ensure GH secrets doesn't intrudce extra '\' new line characters (related to '\' being an escape character)
Expand Down Expand Up @@ -56,6 +55,8 @@ def _format_rounding(value):
return f'{round(value / 1000, 1):.1f}K'


# The rest of this file alternates between functions for requesting information from Google Analytics
# And functions that use that request image to form either a .json or a .png file to be used in write-metrics-md.py
def _run_total_users_report(property_id):
"""
Function for requesting cumulative active users from a project since project start.
Expand Down Expand Up @@ -289,7 +290,7 @@ def plot_usersXcountry(FOUNDATIONS_ID):
)

# Add colorbar
cax = fig.add_axes([0.1, -0.015, 0.67, 0.03])
cax = fig.add_axes([0.05, -0.015, 0.7, 0.03]) # [x0, y0, width, height]
cbar = fig.colorbar(mappable=mappable, cax=cax, spacing='uniform', orientation='horizontal', extend='min')
cbar.set_label('Unique Users')

Expand Down
24 changes: 0 additions & 24 deletions .github/workflows/nightly-build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,7 @@ on:
- cron: '0 0 * * *' # Daily “At 00:00”

jobs:
automate-metrics:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Automate Metrics
env:
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
PRIVATE_KEY_ID: ${{ secrets.PRIVATE_KEY_ID }}
run: |
python -m venv analytics-api
source analytics-api/bin/activate
pip install google-analytics-data cartopy matplotlib

python .github/workflows/get-metrics.py
python .github/workflows/write-metrics-md.py

build:
needs: automate-metrics
if: ${{ github.repository_owner == 'ProjectPythia' }}
uses: ProjectPythia/cookbook-actions/.github/workflows/build-book.yaml@main
with:
Expand All @@ -37,10 +20,3 @@ jobs:
uses: ./.github/workflows/sphinx-link-checker.yaml
with:
path_to_source: 'portal'

deploy:
needs: build
uses: ProjectPythia/cookbook-actions/.github/workflows/deploy-book.yaml@main
with:
cname: projectpythia.org
publish_dir: 'portal/_build/html'
12 changes: 11 additions & 1 deletion .github/workflows/publish-site.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ on:
branches:
- main
workflow_dispatch:
schedule:
- cron: '0 0 * * 1' # Weekly on Monday

jobs:
automate-metrics:
Expand All @@ -23,6 +25,11 @@ jobs:

python .github/workflows/get-metrics.py
python .github/workflows/write-metrics-md.py
- name: Upload zip
uses: actions/upload-artifact@v4
with:
name: repo-zip
path: .

build:
needs: automate-metrics
Expand All @@ -32,7 +39,10 @@ jobs:
environment_name: pythia
path_to_notebooks: 'portal'
build_command: 'make -j4 html'

build_from_code_artifact: 'true'
code_artifact_name: 'repo-zip'
workflow: ''
workflow_conclusion: ''
deploy:
needs: build
uses: ProjectPythia/cookbook-actions/.github/workflows/deploy-book.yaml@main
Expand Down
18 changes: 1 addition & 17 deletions .github/workflows/trigger-preview.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,9 @@ on:
- completed

jobs:
automate-metrics:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Automate Metrics
env:
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
PRIVATE_KEY_ID: ${{ secrets.PRIVATE_KEY_ID }}
run: |
python -m venv analytics-api
source analytics-api/bin/activate
pip install google-analytics-data cartopy matplotlib

python .github/workflows/get-metrics.py
python .github/workflows/write-metrics-md.py

find-pull-request:
needs: automate-metrics
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am hesitant about this removal, i.e. I think keeping the "automate-metrics" job in the "trigger-preview.yaml" workflow would still be valuable to see how the Metrics page would render in the preview of any PR. This is because right after any PR is merged into main, there will be a publish-site workflow run, which will update the metrics on the website.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I was confused about each action a bit. I agree and will re-introduce this - but my attempts at introducing an artifact have broken the actions on my branch.

I think it might be better for me to focus on getting one action working (the publish-site), and then the knowledge I gain should transfer to setting up the trigger-preview to use automate-metrics again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I added this back for now, but I know it won't work yet and I'm not sure if it can work.

I think this because the trigger-preview is grabbing a book-build artifact.
artifact_name: book-zip-${{ needs.find-pull-request.outputs.number }} that is zipped from the trigger-site-build action (which is triggered by a PR and thus does not have access to the secrets necessary to run the metrics collection). And in turn the trigger-preview is triggered by the trigger-site-build.

There seems to be a possible work around with trigger-preview being allowed to access secrets, so we could rebuild the book in that step, but that seems computationally expensive. I have to think about this more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed again per our discussion

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if there is a benefit to moving the automate-metrics step to Cookbook-Actions even though we don't intend for it to be re-usable across repositories. It seems that if it existed as a re-usable workflow there are different permissions. For example, it could access secrets that live in the Cookbook-Actions repo and be triggered by a PR in the portal, zip up it's artifact and pass that back to the trigger-preview workflow. It would be a bit of an overall, and would be the next step after the artifact passing is functioning and well understood.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can revisit this in the future?

uses: ProjectPythia/cookbook-actions/.github/workflows/find-pull-request.yaml@main

deploy-preview:
needs: find-pull-request
if: github.event.workflow_run.conclusion == 'success'
Expand Down
4 changes: 2 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,5 +141,5 @@ resource-gallery-submission-input.json

# Analytics
.github/workflows/analytics-api/
.github/workflows/*.json
.github/workflows/*.png
portal/metrics/*.png
cisl-vast-pythia-*.json