Use Actions API to retrieve job run logs as a fallback mechanism #11172
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds a fallback mechanism for fetching job run logs via the GitHub Actions API when ZIP-based log retrieval fails, and refactors log handling into a pluggable interface with accompanying test updates.
- Introduce a
logFetcherinterface and two implementations (zipLogFetcherandapiLogFetcher), plusgetLogsinlogs.go - Refactor
displayRunLoginview.goto use the newlogscollection andcopyLogWithLinePrefix - Update tests to inline ZIP creation, remove the old fixture, and use
BinaryResponsein HTTP stubs
Reviewed Changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/httpmock/stub.go | Add BinaryResponse helper to return raw []byte bodies |
| pkg/cmd/run/view/view.go | Replace ZIPβonly logic with logs abstraction and API fallback |
| pkg/cmd/run/view/logs.go | New logFetcher implementations and getLogs logic |
| pkg/cmd/run/view/logs_test.go | Add unit tests for zipLogFetcher, apiLogFetcher, and getLogs |
| pkg/cmd/run/shared/shared.go | Remove Log *zip.File fields from Job and Step DTOs |
| pkg/cmd/run/view/view_test.go | Inline ZIP creation in tests and switch to BinaryResponse |
Comments suppressed due to low confidence (2)
pkg/cmd/run/view/logs.go:61
- [nitpick] Consider renaming the
logstype to something more descriptive (e.g.,logManagerorlogFetchers) to clarify its role.
type logs struct {
pkg/cmd/run/view/logs.go:222
- [nitpick] This truncateAsUTF16 comment is very lengthy; consider summarizing core behavior here and moving the extended explanation to a separate markdown or design doc.
If you're reading this comment by necessity, I'm sorry and if you're reading it for fun, you're welcome, you weirdo.
This comment was marked as spam.
This comment was marked as spam.
andyfeller
reviewed
Jun 27, 2025
Member
andyfeller
left a comment
There was a problem hiding this comment.
I appreciate the degree you've gone to ensure our users can reliably pull this information, @babakks! β€
I was curious about the work and had some questions / suggestions about specific parts, deferring to Will being the assigned PR reviewer.
pkg/cmd/run/view/logs.go
Outdated
Comment on lines
105
to
210
| // getLogs populates a logs struct with appropriate log fetchers based on the | ||
| // provided zip file and list of jobs. | ||
| // | ||
| // For any job/step that does not have a log in the zip file, it will create an | ||
| // API log fetcher that will fetch the logs from the GitHub API. | ||
| // | ||
| // The structure of zip file is expected to be as: | ||
| // | ||
| // zip/ | ||
| // βββ jobname1/ | ||
| // β βββ 1_stepname.txt | ||
| // β βββ 2_anotherstepname.txt | ||
| // β βββ 3_stepstepname.txt | ||
| // β βββ 4_laststepname.txt | ||
| // βββ jobname2/ | ||
| // | βββ 1_stepname.txt | ||
| // | βββ 2_somestepname.txt | ||
| // βββ 0_jobname1.txt | ||
| // βββ 1_jobname2.txt | ||
| // βββ -9999999999_jobname3.txt | ||
| // | ||
| // The function iterates through the list of jobs and tries to find the matching | ||
| // log file in the ZIP archive. If the log file is not found, an API log fetcher | ||
| // will be created instead. | ||
| // | ||
| // The top-level .txt files include the logs for an entire job run. Note that | ||
| // the prefixed number is either: | ||
| // - An ordinal and cannot be mapped to the corresponding job's ID. | ||
| // - A negative integer which is the ID of the job in the old Actions service. | ||
| // The service right now tries to get logs and use an ordinal in a loop. | ||
| // However, if it doesn't get the logs, it falls back to an old service | ||
| // where the ID can apparently be negative. |
Member
There was a problem hiding this comment.
question: how much of this should really be package level documentation?
The information is good and would be highly visible if brought up to the top as I think this is relevant context that helps understand much of the file.
Member
Author
There was a problem hiding this comment.
I'd say 80% of the code in this file is dealing with this stuff (i.e. ZIP archive content schema). So, moving the docs up to the top of the file makes sense. But, I'm not down for a package godoc because it's a just a part of the internals of this package. So, what do you think about moving it up (just below the package statement), and then refer to it in the getLogs (or getLogFetcherMap following up the above comment) function godoc?
| @@ -0,0 +1,294 @@ | |||
| package view | |||
Member
There was a problem hiding this comment.
question: when would we have logic like this live in pkg/cmd/run/shared or internal?
Member
Author
There was a problem hiding this comment.
I assume a shared package must have at least two different packages referencing it. So, it's not a shared yet.
Regarding internal, I don't think this is something to be hidden from external importers, if any. Also the types/funcs in this file are all private, so, there's no practical difference, right?
Member
williammartin
left a comment
There was a problem hiding this comment.
This PR as it is right now doesn't just fetch logs from the API when you provide --job, it will also fetch those individual job logs if you're trying to fetch the entire log for the run, and that could have many many jobs within a single run, for which each missing job log is going to result in a sequential API request, so significantly more load potentially and also slowed down gh experience because it's going to be doing n sequential HTTP requests.
The original acceptance criteria said this should work if --job is passed, but I also think it might be very confusing for our users if we don't do this when --jo is not passed because then they're going to say "well it gets the log here but it doesn't get it there". I'm not really sure how to resolve that.
Any thoughts? Maybe need to bring this up to the team.
Otherwise this is a really good PR. I like the test changes a lot.
Member
Author
|
@williammartin, I've implemented the heuristic approach in the last couple of commits. The changes is not as nice as it was before, mostly because we now have to track the API log fetchers we're assigning to make sure they're within the limits. |
babakks
commented
Jul 4, 2025
Comment on lines
321
to
330
| zlm := getZipLogMap(&runLogZip.Reader, jobs) | ||
| segments := populateLogSegments(jobs, zlm, opts.LogFailed) | ||
|
|
||
| return displayRunLog(opts.IO.Out, jobs, opts.LogFailed) | ||
| err = assignMissingLogs(segments, httpClient, repo, run.ID) | ||
| if err != nil { | ||
| if errors.Is(err, errTooManyAPILogFetchers) { | ||
| return fmt.Errorf("too many API requests needed to fetch logs; try narrowing down to a specific job with the `--job` option") | ||
| } | ||
| return err | ||
| } | ||
|
|
||
| return displayLogSegments(opts.IO.Out, segments) |
Member
Author
There was a problem hiding this comment.
@williammartin, this is actually the overall look of the new implementation:
- Make a map of the log files available in the ZIP archive (the
logMapwe created earlier in this PR, is just now about the ZIP archive content). - A slice of
logSegments is put together based on the log files available in the ZIP archive. alogSegmentcan be an individual step log, or and entire job log. - Any
logSegmentwith an unassignedfetcherfield, will be assigned with anapiLogFetcher. Here we also apply the restriction on the count (which is 25) and error out if the number exceeds. - Display the
logSegments.
Signed-off-by: Babak K. Shandiz <[email protected]>
Signed-off-by: Babak K. Shandiz <[email protected]>
Signed-off-by: Babak K. Shandiz <[email protected]>
Signed-off-by: Babak K. Shandiz <[email protected]>
Signed-off-by: Babak K. Shandiz <[email protected]>
9c6e8c7 to
25ecbed
Compare
Member
AcceptanceWith a workflow that has 26 jobs: And has all the logs: 1. Fallback to API callRemoving 1 job from the zip: Given I have a job run that is missing from the downloaded ZIP archive 2. Error when too many API calls are neededGiven I have a workflow run where more than 25 job log files are missing from the downloaded ZIP archive Get 25 JobsSee the error |
williammartin
approved these changes
Jul 4, 2025
Member
williammartin
left a comment
There was a problem hiding this comment.
Works really well, though at 25 jobs it does take a bit of time:
β test-repo git:(main) time ~/workspace/cli/bin/gh run view 16076027406 --log
0.32s user 0.35s system 5% cpu 11.517 total
Signed-off-by: Babak K. Shandiz <[email protected]>
Member
|
Also updated the help text: |
tmeijn
pushed a commit
to tmeijn/dotfiles
that referenced
this pull request
Jul 27, 2025
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.74.2` -> `v2.76.1` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.76.1`](https://github.com/cli/cli/releases/tag/v2.76.1): GitHub CLI 2.76.1 [Compare Source](cli/cli@v2.76.0...v2.76.1) #### `gh pr create` regression fix This release fixes a regression introduced in `v2.76.0` where organization teams were retrieved outside of intentional use cases. This caused problems for GitHub Enterprise Server users using the GitHub Actions automatic token that does not have access to organization teams. For more information, see cli/cli#11360 #### What's Changed ##### π Fixes - Fix: `gh pr create`, only fetch teams when reviewers contain a team by [@​BagToad](https://github.com/BagToad) in cli/cli#11361 ##### π Docs & Chores - add tenancy aware for san matcher by [@​ejahnGithub](https://github.com/ejahnGithub) in cli/cli#11261 - Run Lint and Tests on `push` to `trunk` branch by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11325 - update ownership of pkg/cmd/release/shared/ by [@​ejahnGithub](https://github.com/ejahnGithub) in cli/cli#11326 - Automate spam issue detection by [@​babakks](https://github.com/babakks) in cli/cli#11316 - Improve `api` `--preview` docs by [@​jsoref](https://github.com/jsoref) in cli/cli#11274 - Incorporate govulncheck into workflows by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11332 - chore(deps): bump advanced-security/filter-sarif from 1.0.0 to 1.0.1 by [@​dependabot](https://github.com/dependabot)\[bot] in cli/cli#11298 - chore(deps): bump github.com/sigstore/sigstore-go from 1.0.0 to 1.1.0 by [@​dependabot](https://github.com/dependabot)\[bot] in cli/cli#11307 **Full Changelog**: cli/cli@v2.76.0...v2.76.1 ### [`v2.76.0`](https://github.com/cli/cli/releases/tag/v2.76.0): GitHub CLI 2.76.0 [Compare Source](cli/cli@v2.75.1...v2.76.0) ####Copilot Coding Agent Support GitHub Copilot Pro+ and Copilot Enterprise subscribers can now assign issues to GitHub Copilot during issue creation using: - Command-line flag: `gh issue create --assignee @​copilot` - Launching web browser: `gh issue create --assignee @​copilot --web` - Or interactively selecting `Copilot (AI)` as assignee in `gh issue create` metadata For more details, refer to [the full changelog post for Copilot coding agent](https://github.blog/changelog/2025-05-19-github-copilot-coding-agent-in-public-preview/). #### What's Changed ##### β¨ Features - Assign Copilot during `gh issue create` by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11279 - Display immutable field in `release view` command by [@​bdehamer](https://github.com/bdehamer) in cli/cli#11251 ##### π Fixes - FIX: Do not fetch logs for skipped jobs by [@​babakks](https://github.com/babakks) in cli/cli#11312 - Transform `extension` and `filename` qualifiers into `path` qualifier for web code search by [@​samcoe](https://github.com/samcoe) in cli/cli#11211 ##### π Docs & Chores - FIX: Workflow does not contain permissions by [@​BagToad](https://github.com/BagToad) in cli/cli#11322 - Add automated feature request response workflow by [@​BagToad](https://github.com/BagToad) in cli/cli#11299 **Full Changelog**: cli/cli@v2.75.1...v2.76.0 ### [`v2.75.1`](https://github.com/cli/cli/releases/tag/v2.75.1): GitHub CLI 2.75.1 [Compare Source](cli/cli@v2.75.0...v2.75.1) #### What's Changed ##### π Fixes - Ensure hostnames are visible in CLI website by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11295 - Revert "Fix: `gh pr create` prioritize `--title` and `--body` over `--fill` when `--web` is present" by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11300 ##### π Docs & Chores - Ensure go directive is always .0 version in bump by [@​williammartin](https://github.com/williammartin) in cli/cli#11259 - Minor (1-word) documentation typo in generated `~/.config/gh/config.yml` by [@​kurahaupo](https://github.com/kurahaupo) in cli/cli#11246 - Automate closing of stale issues by [@​babakks](https://github.com/babakks) in cli/cli#11268 - Filter the `third-party/` folder out of CodeQL results by [@​BagToad](https://github.com/BagToad) in cli/cli#11278 - Exclude `third-party` source from golangci-lint by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11293 #####
Dependencies - Bump Go to 1.24.5 by [@​github-actions](https://github.com/github-actions)\[bot] in cli/cli#11255 - chore(deps): bump github.com/sigstore/protobuf-specs from 0.4.3 to 0.5.0 by [@​dependabot](https://github.com/dependabot)\[bot] in cli/cli#11263 - chore(deps): bump golang.org/x/term from 0.32.0 to 0.33.0 by [@​dependabot](https://github.com/dependabot)\[bot] in cli/cli#11266 - chore(deps): bump golang.org/x/sync from 0.15.0 to 0.16.0 by [@​dependabot](https://github.com/dependabot)\[bot] in cli/cli#11264 - chore(deps): bump golang.org/x/text from 0.26.0 to 0.27.0 by [@​dependabot](https://github.com/dependabot)\[bot] in cli/cli#11265 - chore(deps): bump golang.org/x/crypto from 0.39.0 to 0.40.0 by [@​dependabot](https://github.com/dependabot)\[bot] in cli/cli#11275 #### New Contributors - [@​kurahaupo](https://github.com/kurahaupo) made their first contribution in cli/cli#11246 - [@​github-actions](https://github.com/github-actions)\[bot] made their first contribution in cli/cli#11255 **Full Changelog**: cli/cli@v2.75.0...v2.75.1 ### [`v2.75.0`](https://github.com/cli/cli/releases/tag/v2.75.0): GitHub CLI 2.75.0 [Compare Source](cli/cli@v2.74.2...v2.75.0) #### What's Changed ##### β¨ Features - init release verify subcommands by [@​ejahnGithub](https://github.com/ejahnGithub) in cli/cli#11018 - Embed Windows resources (VERSIONINFO) during build by [@​babakks](https://github.com/babakks) in cli/cli#11048 - Support `--no-repos-selected` on `gh secret set` by [@​williammartin](https://github.com/williammartin) in cli/cli#11217 ##### π Fixes - Fix: `gh pr create` prioritize `--title` and `--body` over `--fill` when `--web` is present by [@​dankrzeminski32](https://github.com/dankrzeminski32) in cli/cli#10547 - fix: get token for active user instead of blank if possible by [@​anuraaga](https://github.com/anuraaga) in cli/cli#11038 - Use Actions API to retrieve job run logs as a fallback mechanism by [@​babakks](https://github.com/babakks) in cli/cli#11172 - Fix query object state mutation during pagination by [@​babakks](https://github.com/babakks) in cli/cli#11244 - Handle `HTTP 404` when deleting remote branch in `pr merge` by [@​babakks](https://github.com/babakks) in cli/cli#11234 ##### π Docs & Chores - chore: fix function name by [@​jinjingroad](https://github.com/jinjingroad) in cli/cli#11149 - chore: update Go version to 1.24 in devcontainer configuration and docs by [@​tMinamiii](https://github.com/tMinamiii) in cli/cli#11158 - Ensure lint workflow checks whether 3rd party license and code is up to date by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11047 - docs: install\_linux.md: add Solus linux install instructions by [@​chax](https://github.com/chax) in cli/cli#10823 - Fix missing newline in install\_linux.md by [@​BagToad](https://github.com/BagToad) in cli/cli#11160 - Ensure automation uses pinned go-licenses version by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11161 - Add `workflow_dispatch` support to MR Help Wanted check by [@​BagToad](https://github.com/BagToad) in cli/cli#11179 - Remove unused `GH_TOKEN` env variable from workflow by [@​BagToad](https://github.com/BagToad) in cli/cli#11190 - Add workflow to automate go version bumping by [@​williammartin](https://github.com/williammartin) in cli/cli#11189 - Fix inconsistent use of tabs and spaces by [@​Stefan-Heimersheim](https://github.com/Stefan-Heimersheim) in cli/cli#11194 - Decouple arg parsing from MR finder by [@​babakks](https://github.com/babakks) in cli/cli#11192 - docs: consistently use `apt` in installation instructions by [@​tklauser](https://github.com/tklauser) in cli/cli#11216 - Ensure bump go script has git user configured by [@​williammartin](https://github.com/williammartin) in cli/cli#11229 - Inject token into bump-go workflow by [@​williammartin](https://github.com/williammartin) in cli/cli#11233 - Reinstating Primer Style CLI content within `cli/cli` repository by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11060 - Add setup-go to bump-go workflow by [@​williammartin](https://github.com/williammartin) in cli/cli#11237 - Ensure GoReleaser does not break on Mac OS and Linux when skipping Windows `.rsyso` generation script by [@​andyfeller](https://github.com/andyfeller) in cli/cli#11257 #####
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #11169
This PR is actually a paired effort outcome, thanks to @williammartin! π
This PR adds a fallback mechanism to the process we use to retrieve job run logs. Our main approach is to download a ZIP archive of logs, but there are cases where
ghcannot find the logs in the archive (e.g. due to changes to the job-name-to-file-name translation, and/or special char sanitisation). The fallback implemented in this PR uses a REST API endpoint to fetch the log trail for a given job run.Other changes made in this PR:
The structure of the DTO types (i.e.
JobandStepstructs inpkg/cmd/run/shared/shared.go) has been modified to remove theLog *zip.Filefield, which, semantically is a state. Instead of those fields, we now use a new type,logs, that is basically a map, translating jobs and steps to their correspondinglogFetcherinstances.A
logFetcherinterface is introduced to abstract away the way we'd retreive the logs. There are two implementations of it now;zipLogFetcherandapiLogFetcher.All logic related to extracting logs are now moved into a new file,
logs.go, tests of which added tologs_test.go. Also the test cases inTest_attachRunLogare now moved into a new test function,TestGetLogs, that exercises thegetLogsfunction in various scenarios (especially, regarding zip/API fetcher behaviour).The ZIP archive fixture (under
pkg/cmd/run/view/fixtures/run-log.zip), is now removed in favour of inlining the ZIP archive creation. The reasons for this change are:Verifying A/C
The new code works well with the example cases in #10868.
I also made an artificial test case, by removing a job run log file from the downloaded/cached ZIP archive, and then tried
gh run view --logagain. The result was as expected and the logs were fetched from the API. To replicate you can try the following:(Workflow run used in this example: https://github.com/cli/cli/actions/runs/15880181640)