Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Simplify CI pipeline #1461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 17, 2023
Merged

Simplify CI pipeline #1461

merged 1 commit into from
Jan 17, 2023

Conversation

@wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Jan 14, 2023 •
edited
Loading

The combined changes help make the pipeline and Makefile more readable and cuts the CI validation pipeline from ~12 minutes to ~6 minutes:

  • Adds go build cache to CI cache (cuts the build time from 6-ish minutes to less than 1 minute)
  • Fixes the fingerprint for java cache to include the new native build test fixture (was being rebuilt on every run)
  • Combines the cache fingerprinting to a single step and makes other cache-fingerprinting steps consistent (a separate makefile close to where the cache lives)
  • Combines common actions into a single shared composite action (around setting up go, bootstrapping tools, handling go and tool cache)
  • Splits out benchmark testing to a separate workflow for a couple reasons. The first being that benchmark tests are informative at the PR level and are not a quality gate themselves (thus is not a validation, so doesn't belong in the "validations" workflow). Given that, running benchmark tests on main doesn't help, so this effectively removes this task from running on main (unless by workflow_dispatch)
  • Replace artifact upload/download with action cache for performance (see This process is way too slow actions/upload-artifact#199)

Screen Shot 2023-01-14 at 9 43 19 AM

@wagoodman wagoodman force-pushed the simplify-ci-pipeline branch from 0e2984b to 275aff3 Compare January 14, 2023 14:26
@github-actions
Copy link

github-actions bot commented Jan 14, 2023 •
edited
Loading

Benchmark Test Results

Benchmark results from the latest changes vs base branch 
goos: linux
goarch: amd64
pkg: github.com/anchore/syft/test/integration
cpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
                                                          │ ./.tmp/benchmark-75076ae.txt │
                                                          │            sec/op            │
ImagePackageCatalogers/alpmdb-cataloger-2                                   12.40m ± 20%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             1.397m ±  5%
ImagePackageCatalogers/python-package-cataloger-2                           3.453m ±  1%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   1.131m ±  1%
ImagePackageCatalogers/javascript-package-cataloger-2                       821.6µ ±  2%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   933.7µ ±  0%
ImagePackageCatalogers/rpm-db-cataloger-2                                   1.364m ±  1%
ImagePackageCatalogers/java-cataloger-2                                     14.98m ±  2%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     7.575µ ±  2%
ImagePackageCatalogers/apkdb-cataloger-2                                    940.7µ ±  1%
ImagePackageCatalogers/go-module-binary-cataloger-2                         15.62µ ±  1%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              1.452m ±  1%
ImagePackageCatalogers/portage-cataloger-2                                  763.2µ ±  1%
ImagePackageCatalogers/sbom-cataloger-2                                     4.735m ±  1%
ImagePackageCatalogers/binary-cataloger-2                                   6.215m ±  0%
geomean                                                                     1.094m

                                                          │ ./.tmp/benchmark-75076ae.txt │
                                                          │             B/op             │
ImagePackageCatalogers/alpmdb-cataloger-2                                   5.040Mi ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             204.4Ki ± 0%
ImagePackageCatalogers/python-package-cataloger-2                           953.8Ki ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   217.3Ki ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                       157.9Ki ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   199.8Ki ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                   299.1Ki ± 0%
ImagePackageCatalogers/java-cataloger-2                                     3.364Mi ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     1.375Ki ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                    181.6Ki ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                         1.809Ki ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              374.7Ki ± 0%
ImagePackageCatalogers/portage-cataloger-2                                  137.7Ki ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                     724.3Ki ± 0%
ImagePackageCatalogers/binary-cataloger-2                                   987.2Ki ± 0%
geomean                                                                     217.9Ki

                                                          │ ./.tmp/benchmark-75076ae.txt │
                                                          │          allocs/op           │
ImagePackageCatalogers/alpmdb-cataloger-2                                    85.84k ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                              4.349k ± 0%
ImagePackageCatalogers/python-package-cataloger-2                            16.81k ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                    5.606k ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                        3.430k ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                    4.574k ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                    8.303k ± 0%
ImagePackageCatalogers/java-cataloger-2                                      58.32k ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                       32.00 ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                     5.331k ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                           55.00 ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                               7.226k ± 0%
ImagePackageCatalogers/portage-cataloger-2                                   3.681k ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                      25.30k ± 0%
ImagePackageCatalogers/binary-cataloger-2                                    34.50k ± 0%
geomean                                                                      5.220k

@wagoodman wagoodman force-pushed the simplify-ci-pipeline branch from 275aff3 to 6c0741b Compare January 14, 2023 14:42
@wagoodman wagoodman requested a review from a team January 14, 2023 14:45
@wagoodman wagoodman self-assigned this Jan 14, 2023
@wagoodman wagoodman marked this pull request as ready for review January 14, 2023 14:45
@wagoodman wagoodman force-pushed the simplify-ci-pipeline branch from 6c0741b to 73e5abb Compare January 14, 2023 14:53
@wagoodman wagoodman enabled auto-merge (squash) January 17, 2023 14:48
@wagoodman wagoodman merged commit 05611c2 into main Jan 17, 2023
@wagoodman wagoodman deleted the simplify-ci-pipeline branch January 17, 2023 15:04
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
@wagoodman
bootstrap within composite action (anchore#1461)
ba804dc 
Signed-off-by: Alex Goodman <[email protected]>

Signed-off-by: Alex Goodman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@spiffcs spiffcs spiffcs approved these changes

Assignees

@wagoodman wagoodman

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wagoodman @spiffcs