Introduce a single SBOM document #606

wagoodman · 2021-10-29T16:40:37Z

This PR replaces poweruser.JSONDocumentConfig with sbom.SBOM. Specifically this datastructure is being promoted as a first-class object to be used internally for tasking, presentation, and processing concerns. This is proposed in #555 (comment) but closely affects #554 and #556.

The main proposal of the PR is the following datastructure:

syft/syft/sbom/sbom.go

Lines 10 to 23 in a26e9e4

    
           type SBOM struct { 
        
           	Artifacts Artifacts 
        
           	Source    source.Metadata 
        
           } 
        
           type Artifacts struct { 
        
           	PackageCatalog      *pkg.Catalog 
        
           	FileMetadata        map[source.Location]source.FileMetadata 
        
           	FileDigests         map[source.Location][]file.Digest 
        
           	FileClassifications map[source.Location][]file.Classification 
        
           	FileContents        map[source.Location]string 
        
           	Secrets             map[source.Location][]file.SearchResult 
        
           	Distro              *distro.Distro 
        
           }

All other changes attempt to promote usage of this data structure over passing parts of an SBOM around, for example... replacing:

func Encode(catalog *pkg.Catalog, metadata *source.Metadata, dist *distro.Distro, scope source.Scope, option format.Option) ([]byte, error) { ... }

with:

func Encode(s sbom.SBOM, option format.Option) ([]byte, error) { ... }

Notes:

Relationships has not been included but will later be added as a field under sbom.SBOM in other PRs implementing Prefer artifact relationships over package relationships #556
source.Scope was removed from any presenter output as described in Encapsulate all artifacts in syft JSON output #555 (comment)

github-actions · 2021-10-29T16:43:35Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

name                                                   old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2          1.37ms ± 3%    1.67ms ± 2%  +21.98%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        3.29ms ± 6%    3.88ms ± 6%  +18.15%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     821µs ± 2%     996µs ± 3%  +21.35%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 893µs ± 4%    1077µs ± 2%  +20.60%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  848µs ± 1%    1013µs ± 2%  +19.42%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  11.8ms ± 3%    14.2ms ± 1%  +20.49%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.22ms ± 7%    1.55ms ± 3%  +26.69%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2       654ns ± 4%     783ns ± 3%  +19.65%  (p=0.008 n=5+5)

name                                                   old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           248kB ± 0%     247kB ± 0%   -0.38%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        1.11MB ± 0%    1.11MB ± 0%   -0.33%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     199kB ± 0%     197kB ± 0%   -0.68%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 228kB ± 0%     228kB ± 0%   -0.39%  (p=0.016 n=5+4)
ImagePackageCatalogers/rpmdb-cataloger-2                  222kB ± 0%     221kB ± 0%   -0.48%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  3.24MB ± 0%    3.24MB ± 0%   -0.13%  (p=0.032 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.29MB ± 0%    1.29MB ± 0%   -0.08%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2        336B ± 0%      336B ± 0%     ~     (all equal)

name                                                   old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           6.82k ± 0%     6.79k ± 0%     ~     (p=0.079 n=4+5)
ImagePackageCatalogers/python-package-cataloger-2         26.3k ± 0%     26.3k ± 0%   -0.27%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     5.19k ± 0%     5.17k ± 0%   -0.46%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 6.67k ± 0%     6.65k ± 0%   -0.36%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  6.56k ± 0%     6.53k ± 0%   -0.37%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                   59.0k ± 0%     58.9k ± 0%   -0.20%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                  7.74k ± 0%     7.71k ± 0%   -0.30%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2        9.00 ± 0%      9.00 ± 0%     ~     (all equal)

spiffcs

First pass.

I like the new SBOM struct and how it just encapsulates current information. Is there a future where the underlying Artifacts struct is changed as we build into developing relationships between everything?

Also small comments on data loss across the new outputs of the golden snapshots. I wasn't sure if we wanted to start excluding the information, or if it got lost as a consequence of moving to the new encode architecture.

internal/anchore/import_package_sbom.go

internal/formats/spdx22json/test-fixtures/snapshot/TestSPDXJSONImagePresenter.golden

internal/formats/spdx22json/to_format_model.go

internal/formats/syftjson/test-fixtures/snapshot/TestImagePresenter.golden

wagoodman · 2021-11-02T17:41:31Z

@spiffcs

Is there a future where the underlying Artifacts struct is changed as we build into developing relationships between everything?

Indeed! I have some of that drafted out in #607, specifically:

syft/syft/sbom/sbom.go

Line 13 in c3a65f5

Relationships []artifact.Relationship

This is where the relationships object gets promoted to in the end (the sbom.SBOM struct).

Signed-off-by: Alex Goodman <[email protected]>

Signed-off-by: Christopher Angelo Phillips <[email protected]>

luhring

Very nice work! 👏

I had just one question from curiosity

cmd/packages.go

internal/formats/spdx22json/to_format_model.go

wagoodman · 2021-11-05T14:04:34Z

@spiffcs thanks a ton for solving what ended up being a caching issue I was having --I was starting to go a little crazy there 🙌

* [wip] single sbom doc Signed-off-by: Alex Goodman <[email protected]> * fix tests Signed-off-by: Alex Goodman <[email protected]> * fix more tests Signed-off-by: Alex Goodman <[email protected]> * fix linting Signed-off-by: Alex Goodman <[email protected]> * update cli tests Signed-off-by: Alex Goodman <[email protected]> * remove scope in import path Signed-off-by: Alex Goodman <[email protected]> * swap SPDX tag-value formatter to single sbom document Signed-off-by: Alex Goodman <[email protected]> * bust CLI cache Signed-off-by: Alex Goodman <[email protected]> * update fixture to byte diff Signed-off-by: Christopher Angelo Phillips <[email protected]> * byte for byte Signed-off-by: Christopher Angelo Phillips <[email protected]> * bust the cache Signed-off-by: Christopher Angelo Phillips <[email protected]> * who needs cache Signed-off-by: Christopher Angelo Phillips <[email protected]> * add jar for testing Signed-off-by: Christopher Angelo Phillips <[email protected]> * no more bit flips Signed-off-by: Christopher Angelo Phillips <[email protected]> * update apk with the delta for image and directory cases Signed-off-by: Christopher Angelo Phillips <[email protected]> * restore cache workflow Signed-off-by: Christopher Angelo Phillips <[email protected]> Co-authored-by: Christopher Angelo Phillips <[email protected]>

wagoodman requested a review from a team October 29, 2021 16:40

wagoodman self-assigned this Oct 29, 2021

wagoodman mentioned this pull request Oct 31, 2021

First class relationships #607

Merged

wagoodman marked this pull request as ready for review November 2, 2021 14:23

spiffcs mentioned this pull request Nov 2, 2021

Promote cataloging task pattern #554

Closed

spiffcs reviewed Nov 2, 2021

View reviewed changes

wagoodman force-pushed the single-sbom-document branch 2 times, most recently from bf6a2cd to 4757c25 Compare November 2, 2021 17:39

wagoodman added 7 commits November 2, 2021 14:12

[wip] single sbom doc

c6acdc8

Signed-off-by: Alex Goodman <[email protected]>

fix tests

a8b3280

Signed-off-by: Alex Goodman <[email protected]>

fix more tests

6d14c3a

Signed-off-by: Alex Goodman <[email protected]>

fix linting

39730ca

Signed-off-by: Alex Goodman <[email protected]>

update cli tests

af773d2

Signed-off-by: Alex Goodman <[email protected]>

remove scope in import path

a5df4b9

Signed-off-by: Alex Goodman <[email protected]>

swap SPDX tag-value formatter to single sbom document

95bd1fd

Signed-off-by: Alex Goodman <[email protected]>

wagoodman force-pushed the single-sbom-document branch from 4757c25 to 95bd1fd Compare November 2, 2021 18:24

bust CLI cache

292bdfd

Signed-off-by: Alex Goodman <[email protected]>

wagoodman force-pushed the single-sbom-document branch from 15a896f to 7c142e5 Compare November 2, 2021 21:41

spiffcs force-pushed the single-sbom-document branch 5 times, most recently from dabefda to 19b9013 Compare November 3, 2021 18:14

update fixture to byte diff

197c27b

Signed-off-by: Christopher Angelo Phillips <[email protected]>

spiffcs force-pushed the single-sbom-document branch from 19b9013 to 197c27b Compare November 3, 2021 18:41

spiffcs added 3 commits November 3, 2021 14:53

byte for byte

7e1d7e2

Signed-off-by: Christopher Angelo Phillips <[email protected]>

bust the cache

0bc33f5

Signed-off-by: Christopher Angelo Phillips <[email protected]>

who needs cache

fe6b1b6

Signed-off-by: Christopher Angelo Phillips <[email protected]>

spiffcs added 3 commits November 3, 2021 15:43

add jar for testing

fbfc02d

Signed-off-by: Christopher Angelo Phillips <[email protected]>

no more bit flips

cb31cee

Signed-off-by: Christopher Angelo Phillips <[email protected]>

update apk with the delta for image and directory cases

7dabab8

Signed-off-by: Christopher Angelo Phillips <[email protected]>

spiffcs requested a review from a team November 4, 2021 16:47

restore cache workflow

327cac9

Signed-off-by: Christopher Angelo Phillips <[email protected]>

luhring approved these changes Nov 4, 2021

View reviewed changes

cmd/packages.go Show resolved Hide resolved

internal/formats/spdx22json/to_format_model.go Show resolved Hide resolved

spiffcs approved these changes Nov 5, 2021

View reviewed changes

wagoodman merged commit bb0f35b into main Nov 5, 2021

wagoodman deleted the single-sbom-document branch November 5, 2021 14:05

wagoodman mentioned this pull request Nov 8, 2021

Use SBOM data structure in ImportConfig #621

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Introduce a single SBOM document #606

Introduce a single SBOM document #606

Uh oh!

wagoodman commented Oct 29, 2021

Uh oh!

github-actions bot commented Oct 29, 2021 •

edited

Loading

Uh oh!

spiffcs left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wagoodman commented Nov 2, 2021

Uh oh!

luhring left a comment

Uh oh!

Uh oh!

Uh oh!

wagoodman commented Nov 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	type SBOM struct {
	Artifacts Artifacts
	Source source.Metadata
	}

	type Artifacts struct {
	PackageCatalog *pkg.Catalog
	FileMetadata map[source.Location]source.FileMetadata
	FileDigests map[source.Location][]file.Digest
	FileClassifications map[source.Location][]file.Classification
	FileContents map[source.Location]string
	Secrets map[source.Location][]file.SearchResult
	Distro *distro.Distro
	}

Uh oh!

Introduce a single SBOM document #606

Introduce a single SBOM document #606

Uh oh!

Conversation

wagoodman commented Oct 29, 2021

Uh oh!

github-actions bot commented Oct 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Test Results

Uh oh!

spiffcs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wagoodman commented Nov 2, 2021

Uh oh!

luhring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wagoodman commented Nov 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Oct 29, 2021 •

edited

Loading