Replace distro type #742

wagoodman · 2022-01-10T21:33:37Z

Removes the distro.Distro object in favor for a new linux.Release struct. The existing distro object takes part of the raw information that is found during analysis and attempts to identify the release relative to a hard-coded enumeration. The remaining raw information discovered is thrown away. There are problems with this:

Syft is meant to raise up as much raw information as possible and attempt not to synthesize data or conclude about interpretations of the data. The distro object is in conflict with this.
Throwing away the unused information from the underlying data source hinders downstream use of such data
Grype is really in need of this distro.Type enumeration, which leads to unnecessary updates of syft to improve grype matches (e.g. adding a new supported distro).

The new linux.Release struct raises up information found in /*/os-release, which is extremely common across all distributions.

This makes a breaking change to the JSON schema, thus, it is being bumped in this PR from 2 to 3. The existing .distro.idLike field is a string. The underlying field is a space-separated list of distribution names which the current linux release is most close to. This field is being transformed into a list to eliminate the need for downstream consumers to split the field on space before use.

Closes #342

Signed-off-by: Alex Goodman <[email protected]>

github-actions · 2022-01-10T21:37:02Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.36ms ± 1%    1.66ms ± 8%  +22.32%  (p=0.016 n=4+5)
ImagePackageCatalogers/python-package-cataloger-2            3.10ms ± 1%    3.86ms ± 8%  +24.59%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     961µs ± 1%    1158µs ± 3%  +20.50%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         845µs ± 0%     983µs ± 3%  +16.36%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     963µs ± 1%    1173µs ± 5%  +21.86%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      878µs ± 1%    1029µs ± 4%  +17.28%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      13.4ms ± 0%    15.0ms ± 2%  +11.94%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.29ms ± 1%    1.48ms ± 3%  +14.52%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          1.94µs ± 2%    2.14µs ±10%  +10.27%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               252kB ± 0%     253kB ± 0%   +0.49%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            1.06MB ± 0%    1.07MB ± 0%   +0.31%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     230kB ± 0%     230kB ± 0%   +0.33%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         207kB ± 0%     208kB ± 0%   +0.61%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     252kB ± 0%     254kB ± 0%   +0.54%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      234kB ± 0%     235kB ± 0%   +0.45%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.62MB ± 0%    3.63MB ± 0%     ~     (p=0.056 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.30MB ± 0%    1.30MB ± 0%   +0.12%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            560B ± 0%      560B ± 0%     ~     (all equal)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               6.31k ± 0%     6.33k ± 0%   +0.35%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2             21.3k ± 0%     21.4k ± 0%   +0.30%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     6.04k ± 0%     6.06k ± 0%   +0.36%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         5.32k ± 0%     5.34k ± 0%   +0.41%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     7.05k ± 0%     7.07k ± 0%   +0.31%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      6.78k ± 0%     6.80k ± 0%   +0.32%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                       69.5k ± 0%     69.7k ± 0%   +0.26%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      7.34k ± 0%     7.36k ± 0%   +0.30%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            13.0 ± 0%      13.0 ± 0%     ~     (all equal)

Signed-off-by: Alex Goodman <[email protected]>

spiffcs

LGTM - I'll wait like the notes said for the grype ~~and grype-db~~ branches so we can check it all together but no real notes on this one.

spiffcs · 2022-01-11T15:21:08Z

internal/anchore/import_package_sbom.go

 }

+// importSBOM mirrors all elements found on the syftjson model format object relative to the anchore engine import schema.
+type importSBOM struct {


I didn't see any notes about this in the PR description. Is this type/method needed for internal anchore purposes for when we're changing the schema with breaking changes 2 -> 3?

great point, let me update the PR description to reflect this change and why

internal/formats/syftjson/to_format_model.go

spiffcs · 2022-01-11T15:29:04Z

syft/pkg/dpkg_metadata.go

 	pURL := packageurl.NewPackageURL(
 		// TODO: replace with `packageurl.TypeDebian` upon merge of https://github.com/package-url/packageurl-go/pull/21
 		// TODO: or, since we're now using an Anchore fork of this module, we could do this sooner.
 		"deb",


With the new release struct do we have enough information to replace this field with information we've pulled from the os-information or do we want to keep this hardcoded for now?

an upcoming PR is about to change this, I think in the way that you're describing (derive the value directly from the release ID)

test/integration/distro_test.go

wagoodman · 2022-01-12T14:49:22Z

I'm going to hold off on merging this until the sunsetting of grype-db is completed (since this changes the types shared between grype and grype-db)

…hub.com/hectorj2f/syft into hectorj2f/add_dependencies_to_cyclonedx * 'hectorj2f/add_dependencies_to_cyclonedx' of https://github.com/hectorj2f/syft: (29 commits) Improve CycloneDX format output (#710) Add additional PHP metadata (#753) Update Syft formats for SyftJson (#752) Add support for "file" source type in syftjson unmarshaling (#750) remove contains file from spdx dependency generation support .sar for java ecosystem (#748) Start developer documentation (#746) Align SPDX export more with SPDX 2.2 specification (#743) Replace distro type (#742) update goreleaser with windows checksums (#740) bump stereoscope version to remove old containerd (#741) Add support for multiple output files in different formats (#732) Add support for searching for jars within archives (#734) 683 windows filepath (#735) Fix CPE encode/decode when it contains special chars (#714) support .par for java ecosystems (#727) Add arm64 support to install script (#729) Revert "bump goreleaser to v1.2 (#720)" (#731) Add a version flag (#722) Add lpkg as java package format (#694) ...

* remove strong distro type Signed-off-by: Alex Goodman <[email protected]> * bump json schema to v3 (breaking distro shape) Signed-off-by: Alex Goodman <[email protected]> * fix linting Signed-off-by: Alex Goodman <[email protected]> * allow for v2 decoding of distro idLikes field in v3 json decoder Signed-off-by: Alex Goodman <[email protected]> * fix casing in simple linux release name Signed-off-by: Alex Goodman <[email protected]> * use discovered name as pretty name in simple linux release Signed-off-by: Alex Goodman <[email protected]> Signed-off-by: fsl <[email protected]>

* remove strong distro type Signed-off-by: Alex Goodman <[email protected]> * bump json schema to v3 (breaking distro shape) Signed-off-by: Alex Goodman <[email protected]> * fix linting Signed-off-by: Alex Goodman <[email protected]> * allow for v2 decoding of distro idLikes field in v3 json decoder Signed-off-by: Alex Goodman <[email protected]> * fix casing in simple linux release name Signed-off-by: Alex Goodman <[email protected]> * use discovered name as pretty name in simple linux release Signed-off-by: Alex Goodman <[email protected]> Signed-off-by: Christopher Phillips <[email protected]>

* remove strong distro type Signed-off-by: Alex Goodman <[email protected]> * bump json schema to v3 (breaking distro shape) Signed-off-by: Alex Goodman <[email protected]> * fix linting Signed-off-by: Alex Goodman <[email protected]> * allow for v2 decoding of distro idLikes field in v3 json decoder Signed-off-by: Alex Goodman <[email protected]> * fix casing in simple linux release name Signed-off-by: Alex Goodman <[email protected]> * use discovered name as pretty name in simple linux release Signed-off-by: Alex Goodman <[email protected]>

wagoodman added 3 commits January 10, 2022 11:40

remove strong distro type

7317e0d

Signed-off-by: Alex Goodman <[email protected]>

bump json schema to v3 (breaking distro shape)

f8359b2

Signed-off-by: Alex Goodman <[email protected]>

fix linting

ba8fdf0

Signed-off-by: Alex Goodman <[email protected]>

wagoodman self-assigned this Jan 10, 2022

wagoodman changed the title ~~Remove distro type~~ Replace distro type Jan 10, 2022

spiffcs self-requested a review January 10, 2022 21:34

wagoodman added 2 commits January 10, 2022 17:09

allow for v2 decoding of distro idLikes field in v3 json decoder

7b2281c

Signed-off-by: Alex Goodman <[email protected]>

fix casing in simple linux release name

ef255ed

Signed-off-by: Alex Goodman <[email protected]>

This was referenced Jan 10, 2022

Add strong distro type from grype anchore/grype-db#41

Closed

Add strong distro type anchore/grype#585

Merged

use discovered name as pretty name in simple linux release

0ec102f

Signed-off-by: Alex Goodman <[email protected]>

wagoodman marked this pull request as ready for review January 10, 2022 23:06

wagoodman requested a review from a team January 10, 2022 23:06

spiffcs reviewed Jan 11, 2022

View reviewed changes

wagoodman added the breaking-change Change is not backwards compatible label Jan 11, 2022

spiffcs approved these changes Jan 11, 2022

View reviewed changes

wagoodman merged commit 706f291 into main Jan 12, 2022

wagoodman deleted the remove-strong-distro-type branch January 12, 2022 17:13

wagoodman mentioned this pull request Feb 24, 2022

Restore behavior of JSON distro block anchore/grype#643

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Replace distro type #742

Replace distro type #742

Uh oh!

wagoodman commented Jan 10, 2022 •

edited

Loading

Uh oh!

github-actions bot commented Jan 10, 2022 •

edited

Loading

Uh oh!

spiffcs left a comment •

edited by wagoodman

Loading

Uh oh!

spiffcs Jan 11, 2022

Uh oh!

wagoodman Jan 11, 2022

Uh oh!

Uh oh!

spiffcs Jan 11, 2022

Uh oh!

wagoodman Jan 11, 2022

Uh oh!

Uh oh!

wagoodman commented Jan 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Replace distro type #742

Replace distro type #742

Uh oh!

Conversation

wagoodman commented Jan 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Test Results

Uh oh!

spiffcs left a comment • edited by wagoodman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spiffcs Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

wagoodman Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spiffcs Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

wagoodman Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wagoodman commented Jan 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wagoodman commented Jan 10, 2022 •

edited

Loading

github-actions bot commented Jan 10, 2022 •

edited

Loading

spiffcs left a comment •

edited by wagoodman

Loading