Improve CycloneDX format output #710

sambhav · 2021-12-21T12:18:10Z

This commit dramatically improves the information we add in the
cyclonedx output -

It adds -

CPEs
Authors
Publishers
External References (Website, Distribution, VCS)
Description
Fixes license (id is a required property and must follow SPDX license ids, name is optional)
syft metadata as arbitrary properties
distro information as an OS component

Signed-off-by: Sambhav Kothari [email protected]

Note: This organizes the cyclonedx helpers in a structure similar to the SPDX helpers to better organize formatting code.

Fixes #268
Partially Fixes #154
Partially Fixes #632
Depends on #714

sambhav · 2021-12-21T12:19:58Z

cc: @VinodAnandan, @coderpatros from CycloneDX team

VinodAnandan · 2021-12-21T14:38:27Z

Thank you @samj1912 !!
Cc + @nscuro, @stevespringett

sambhav · 2021-12-21T15:02:10Z

I am facing an issue with CPE binding strings that are generated by https://github.com/facebookincubator/nvdtools

It looks like it doesn't match the spec. See eg. at https://regexr.com/6c0et

EDIT - on further investigation, it looks like some of the CPEs we generate have versions of form {x}:{y}. Since the version itself contains a colon - it causes issues with cpe parsing and outputting. We should investigate this further.

EDIT 2 - On further investigation, it looks like the cause is packages like https://packages.ubuntu.com/search?keywords=zlib1g which contain a : in the version number which causes CPE generation to go haywire.

Edit 3 - TIL that the colon indicates an epoch for debian packages - https://askubuntu.com/questions/441879/why-do-some-packages-have-extra-numbers-before-a-colon-on-the-front-of-their-ver

Looks like we might want to handle CPE generation for packages with epochs separately.

sambhav · 2021-12-21T16:02:41Z

Created issue #712 - will fix it in a different PR

sambhav · 2021-12-22T16:31:49Z

Note on the commit that adds metadata props -

This commit adds arbitrary properties from the syft metadata to
CycloneDX. This builds on top of the previous work on enriching
the cyclonedx output. Creating a separate PR for this work as this
might be a point of contention whether we want to add these properties
or not or the way the properties are currently extracted using struct
tags.

This allows us to use struct tags to add cyclonedx property values. It works in the following way -

If it detects an explicit cyclonedx struct tag and it will use the value of that tag as the name for the property.

It stringifies the value of the struct field to get the prop value since cyclonedx props are strings only.

If the cyclonedx struct tag is not added or if the stringified value is empty, the prop is not added.

internal/formats/common/cyclonedxhelpers/hashes.go

spiffcs

Comments added thanks again @samj1912 for the great PR!

I didn't cover the cpe changes since I added comments in the dependent PR which we can merge before moving on this one.

internal/formats/common/cyclonedxhelpers/external_references.go

internal/formats/common/cyclonedxhelpers/hashes.go

internal/formats/common/cyclonedxhelpers/licenses.go

internal/formats/common/cyclonedxhelpers/properties.go

internal/formats/cyclonedx13json/test-fixtures/snapshot/TestCycloneDxDirectoryPresenter.golden

internal/formats/common/cyclonedxhelpers/properties.go

spiffcs · 2022-01-06T16:17:39Z

Now that the CPE changes have been merged it looks like this one just has some conflicts @samj1912

spiffcs · 2022-01-06T16:17:49Z

Thanks so much for the huge lift on these two massive PR

sambhav · 2022-01-06T21:20:34Z

@wagoodman @spiffcs - made several improvements to the properties output. We now store metadatatype and also pkg.Type. We also store the distro information in the top level sbom metadata properties. I believe this should result in all the information that grype may need to do vuln. matching with cyclonedx at this point.

sambhav · 2022-01-13T20:43:21Z

@wagoodman / @spiffcs if you could please take a look again that would be great :)

spiffcs · 2022-01-13T20:48:58Z

Yep! Sorry for the delay here @samj1912. I've been looking into signing sbom at the moment. Next time I get to review PRS this is at the top of the stack.

wagoodman

@samj1912 the PR is just about there! My only blocking comment is this one: https://github.com/anchore/syft/pull/710/files#r784914277 .

spiffcs

Viewed the rest of the file and LGTM. The only blocking comment here is what Alex had mentioned regarding including hashes from metadata files. I also agree with him that we should only be including hashes that we have explicitly read rather than pulling them from the metadata entries.

This commit dramatically improves the information we add in the cyclonedx output - It adds - * CPEs * Authors * Publishers * External References (Website, Distribution, VCS) * Description Signed-off-by: Sambhav Kothari <[email protected]>

This commit adds arbitrary properties from the syft metadata to CycloneDX. This builds on top of the previous work on enriching the cyclonedx output. Creating a separate PR for this work as this might be a point of contention whether we want to add these properties or not or the way the properties are currently extracted using struct tags. Signed-off-by: Sambhav Kothari <[email protected]>

Signed-off-by: Sambhav Kothari <[email protected]>

sambhav · 2022-01-14T22:49:18Z

@wagoodman / @spiffcs should be updated now :)

wagoodman

@samj1912 stellar contribution... thanks for your work on this!

sambhav · 2022-01-18T22:23:55Z

Thanks @wagoodman! Is this good to merge?

…hub.com/hectorj2f/syft into hectorj2f/add_dependencies_to_cyclonedx * 'hectorj2f/add_dependencies_to_cyclonedx' of https://github.com/hectorj2f/syft: (29 commits) Improve CycloneDX format output (#710) Add additional PHP metadata (#753) Update Syft formats for SyftJson (#752) Add support for "file" source type in syftjson unmarshaling (#750) remove contains file from spdx dependency generation support .sar for java ecosystem (#748) Start developer documentation (#746) Align SPDX export more with SPDX 2.2 specification (#743) Replace distro type (#742) update goreleaser with windows checksums (#740) bump stereoscope version to remove old containerd (#741) Add support for multiple output files in different formats (#732) Add support for searching for jars within archives (#734) 683 windows filepath (#735) Fix CPE encode/decode when it contains special chars (#714) support .par for java ecosystems (#727) Add arm64 support to install script (#729) Revert "bump goreleaser to v1.2 (#720)" (#731) Add a version flag (#722) Add lpkg as java package format (#694) ...

* Improve CycloneDX format output ## Additions to CycloneDX output * CPEs * Authors * Publishers * External References (Website, Distribution, VCS) * Description Signed-off-by: Sambhav Kothari <[email protected]> Signed-off-by: fsl <[email protected]>

* Improve CycloneDX format output * CPEs * Authors * Publishers * External References (Website, Distribution, VCS) * Description Signed-off-by: Sambhav Kothari <[email protected]>

* Improve CycloneDX format output ## Additions to CycloneDX output * CPEs * Authors * Publishers * External References (Website, Distribution, VCS) * Description Signed-off-by: Sambhav Kothari <[email protected]>

sambhav force-pushed the enrich-cdx branch from c8c105e to 33f46b7 Compare December 21, 2021 14:21

sambhav force-pushed the enrich-cdx branch 3 times, most recently from 3784b67 to b7d8142 Compare December 22, 2021 11:50

sambhav mentioned this pull request Dec 22, 2021

Add metadata properties to CycloneDX #717

Closed

sambhav force-pushed the enrich-cdx branch 6 times, most recently from 18aeca3 to ea4302e Compare December 22, 2021 21:08

This was referenced Dec 23, 2021

Support for NTIA minimum elements for an SBOM #632

Open

Extend cycloneDX presenters with syft-specific values #154

Closed

sambhav force-pushed the enrich-cdx branch 2 times, most recently from 761bf00 to 312b78c Compare December 23, 2021 16:16

wagoodman reviewed Dec 23, 2021

View reviewed changes

internal/formats/common/cyclonedxhelpers/hashes.go Outdated Show resolved Hide resolved

spiffcs reviewed Jan 3, 2022

View reviewed changes

wagoodman reviewed Jan 3, 2022

View reviewed changes

internal/formats/common/cyclonedxhelpers/properties.go Outdated Show resolved Hide resolved

sambhav force-pushed the enrich-cdx branch 4 times, most recently from 8bf0894 to e11f60e Compare January 6, 2022 21:19

sambhav force-pushed the enrich-cdx branch 4 times, most recently from 73c6311 to d5a7909 Compare January 7, 2022 15:25

wagoodman reviewed Jan 14, 2022

View reviewed changes

spiffcs reviewed Jan 14, 2022

View reviewed changes

sambhav added 6 commits January 14, 2022 16:43

Improve CycloneDX format output

d4ddc3b

This commit dramatically improves the information we add in the cyclonedx output - It adds - * CPEs * Authors * Publishers * External References (Website, Distribution, VCS) * Description Signed-off-by: Sambhav Kothari <[email protected]>

Add tests for cyclonedx encoders

7b78b10

Signed-off-by: Sambhav Kothari <[email protected]>

Add support for hashes

2da14be

Signed-off-by: Sambhav Kothari <[email protected]>

Add additional fields to properties

897ba79

Signed-off-by: Sambhav Kothari <[email protected]>

Remove package hashes and update distro info formatting

c4d16f5

Signed-off-by: Sambhav Kothari <[email protected]>

sambhav force-pushed the enrich-cdx branch from d5a7909 to c4d16f5 Compare January 14, 2022 22:48

wagoodman approved these changes Jan 18, 2022

View reviewed changes

spiffcs merged commit aebe843 into anchore:main Jan 19, 2022

sambhav deleted the enrich-cdx branch January 19, 2022 18:13

pbalogh-sa mentioned this pull request Jan 20, 2022

Add some additional property fileds to cycloneDX output. #685

Closed

sambhav mentioned this pull request Jan 20, 2022

Cyclonedx export with CPE #749

Closed

pbalogh-sa mentioned this pull request Jan 20, 2022

[CycloneDX] Add artifactID and groupID to the cycloneDX properties (support lower level struct as properties) #758

Merged

cjnosal mentioned this pull request Oct 6, 2022

Include Syft's cyclonedx component properties in Grype output anchore/grype#951

Closed

Uh oh!

Improve CycloneDX format output #710

Improve CycloneDX format output #710

Uh oh!

Conversation

sambhav commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sambhav commented Dec 21, 2021

Uh oh!

VinodAnandan commented Dec 21, 2021

Uh oh!

sambhav commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sambhav commented Dec 21, 2021

Uh oh!

sambhav commented Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

spiffcs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spiffcs commented Jan 6, 2022

Uh oh!

spiffcs commented Jan 6, 2022

Uh oh!

sambhav commented Jan 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sambhav commented Jan 13, 2022

Uh oh!

spiffcs commented Jan 13, 2022

Uh oh!

wagoodman left a comment

Choose a reason for hiding this comment

Uh oh!

spiffcs left a comment

Choose a reason for hiding this comment

Uh oh!

sambhav commented Jan 14, 2022

Uh oh!

wagoodman left a comment

Choose a reason for hiding this comment

Uh oh!

sambhav commented Jan 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sambhav commented Dec 21, 2021 •

edited

Loading

sambhav commented Dec 21, 2021 •

edited

Loading

sambhav commented Dec 22, 2021 •

edited

Loading

sambhav commented Jan 6, 2022 •

edited

Loading