Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add digest property to parent and nested java package metadata #941

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 21 commits into from
Apr 8, 2022
Merged

Add digest property to parent and nested java package metadata #941

merged 21 commits into from
Apr 8, 2022

Conversation

@spiffcs
Copy link
Contributor

@spiffcs spiffcs commented Apr 6, 2022 •
edited
Loading

Summary

Adding digests to the discovered java packages will give downstream SBOM consumers more options for querying artifact registries regarding specific details surrounding SBOM content.

A followup PR in grype will be made after this to allow it to optionally consume this information for matching against an upstream Maven repository for better vulnerability match fidelity

Addtions

  • Add digest property to JavaMetadata
  • Populate external formats (cyclonedx, cyclonedx-json, spdx, spdx-json, syft-json) with new digest values

Todo

  • Update CyconeDX and SPDX output with new digest properties and filesAnalyzed switch
  • Refactor and export digest cataloger functionality to be shared across code

Signed-off-by: Christopher Phillips [email protected]

@spiffcs
add digest to java_metadata
d957f9f 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs spiffcs force-pushed the 887-detect-java-namespaces-group-id-by-hash branch from a53f28c to d957f9f Compare April 6, 2022 20:05
@github-actions
Copy link

github-actions bot commented Apr 6, 2022 •
edited
Loading

Benchmark Test Results

Benchmark results from the latest changes vs base branch 
name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.04ms ± 4%    1.45ms ± 3%  +39.84%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            2.59ms ± 1%    3.78ms ± 0%  +45.96%  (p=0.016 n=5+4)
ImagePackageCatalogers/php-composer-installed-cataloger-2     877µs ±10%    1189µs ± 3%  +35.53%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         624µs ± 4%     793µs ± 2%  +27.06%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     758µs ± 2%     940µs ± 2%  +23.93%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      693µs ± 5%     832µs ± 3%  +20.05%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      12.3ms ± 4%    17.3ms ± 2%  +40.38%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.16ms ± 4%    1.45ms ± 3%  +25.00%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          2.02µs ± 1%    2.59µs ± 3%  +28.44%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               185kB ± 0%     184kB ± 0%   -0.21%  (p=0.016 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2             896kB ± 0%     894kB ± 0%   -0.19%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     196kB ± 0%     196kB ± 0%     ~     (p=0.690 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         140kB ± 0%     140kB ± 0%     ~     (p=0.690 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     175kB ± 0%     175kB ± 0%     ~     (p=0.548 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      163kB ± 0%     163kB ± 0%     ~     (p=0.841 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.19MB ± 0%    3.29MB ± 0%   +3.32%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.24MB ± 0%    1.24MB ± 0%     ~     (p=0.310 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            672B ± 0%      672B ± 0%     ~     (all equal)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               3.66k ± 0%     3.66k ± 0%     ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2             14.8k ± 0%     14.8k ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     4.94k ± 0%     4.94k ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         2.72k ± 0%     2.72k ± 0%     ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                     3.93k ± 0%     3.93k ± 0%     ~     (all equal)
ImagePackageCatalogers/rpmdb-cataloger-2                      4.01k ± 0%     4.01k ± 0%     ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                       52.2k ± 0%     52.2k ± 0%   +0.10%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      4.81k ± 0%     4.81k ± 0%     ~     (p=0.992 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            15.0 ± 0%      15.0 ± 0%     ~     (all equal)

spiffcs added 6 commits April 6, 2022 16:13
@spiffcs
remove design notes
eb7fb67 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
consume common digest struct across metadata
0018039 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
update parser test to redact digest
be0877b 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
set files analyzed to true for some java packages
83d3c37 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
update spdx22 with checksums from java metadata
217ef3a 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
add checksums to spdx22tagvalue
76fc08b 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs spiffcs force-pushed the 887-detect-java-namespaces-group-id-by-hash branch from 913c9ff to 76fc08b Compare April 7, 2022 02:50
@spiffcs spiffcs requested a review from a team April 7, 2022 02:51
@spiffcs spiffcs marked this pull request as ready for review April 7, 2022 02:51
spiffcs added 4 commits April 6, 2022 23:18
@spiffcs
add java package sha to cyclonedx format as exref
1f60b18 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
update static analysis
7ef0872 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
add decode path for encode/decode cycle
e737f7b 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
supress linter for now
db9968f 
Signed-off-by: Christopher Phillips <[email protected]>
spiffcs
spiffcs commented Apr 7, 2022
View reviewed changes
@spiffcs
update nolint until PR review
31a3a6a 
Signed-off-by: Christopher Phillips <[email protected]>
luhring
luhring reviewed Apr 7, 2022
View reviewed changes
kzantow
kzantow reviewed Apr 7, 2022
View reviewed changes
Copy link
Contributor

@kzantow kzantow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, this looks okay, but I'm a little concerned about the java/archive_parser_test.go that these hashes are different locally vs CI, what is the reason for this?

wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
spiffcs added 3 commits April 7, 2022 12:33
@spiffcs
refactor digest -> ArchiveDigests
de32030 
update metadata digests to be plural and rename
remove xerrors package usage
update error --> warning
change parse --> calculate

Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
refactor digest under syft/syft/file
78e2abd 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
update tests to redact to nil slice
2ffc85b 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs spiffcs force-pushed the 887-detect-java-namespaces-group-id-by-hash branch from c90327d to 2ffc85b Compare April 7, 2022 17:30
@spiffcs spiffcs linked an issue Apr 7, 2022 that may be closed by this pull request
Detect Java Namespaces/Group IDs by hash #887
Closed
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
wagoodman
wagoodman reviewed Apr 7, 2022
View reviewed changes
kzantow
kzantow reviewed Apr 7, 2022
View reviewed changes
@spiffcs
Copy link
Contributor Author

spiffcs commented Apr 7, 2022

Thanks for the feedback everyone! I'll take another stab and incorporate your comments. I committed some of the suggested changes and need to start looking at what's failing.

@spiffcs
using correct values for format encoding
08ae188 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs spiffcs force-pushed the 887-detect-java-namespaces-group-id-by-hash branch from 0efa3d9 to 08ae188 Compare April 7, 2022 21:36
spiffcs added 2 commits April 7, 2022 17:46
@spiffcs spiffcs force-pushed the 887-detect-java-namespaces-group-id-by-hash branch from 2bd5147 to 40ee003 Compare April 8, 2022 00:58
@spiffcs
Copy link
Contributor Author

spiffcs commented Apr 8, 2022

@wagoodman and @kzantow this has now been updated to address the last round of comments

kzantow
kzantow requested changes Apr 8, 2022
View reviewed changes
Copy link
Contributor

@kzantow kzantow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had another pass at this, I think I identified one nil pointer panic, where there's for _, hash := range *ref.Hashes {, I'm pretty sure ref could be nil if not found.

One other thing is the CycloneDX decoder is handling the hashes, it looks like but there isn't anything for SPDX decoding. You're probably not seeing any failing tests because they haven't been added to the encode-decode-encode cycle test. They should be but I suspect that would involve some work outside the scope of this PR. So know we either need a follow-on task to do this (which I would be okay with, personally, to add the spdx formats to encode-decode-encode, which would involve adding the decoding support for the digests) or add decoding support here.

spiffcs added 2 commits April 8, 2022 09:21
@spiffcs
PR Feedback
a9a23bc 
Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs
guard against any nil value returned by findExternalRef
6c7783f 
Signed-off-by: Christopher Phillips <[email protected]>
wagoodman
wagoodman reviewed Apr 8, 2022
View reviewed changes
syft/pkg/cataloger/java/archive_parser.go Outdated
}

// JavaArchiveHashes are all the current hash algorithms used to calculate archive digests
var JavaArchiveHashes = []crypto.Hash{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could unexport this

wagoodman
wagoodman approved these changes Apr 8, 2022
View reviewed changes
Copy link
Contributor

@wagoodman wagoodman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work 🙌

@spiffcs
Copy link
Contributor Author

spiffcs commented Apr 8, 2022

@kzantow @wagoodman I added decoding for spdx22 support can you take a look

@spiffcs spiffcs mentioned this pull request Apr 8, 2022
Merged
1 task
@spiffcs spiffcs changed the title Add digest property to parent and nested java package metadata Add digest property to parent and nested java package metadata Apr 8, 2022
@spiffcs spiffcs merged commit 782b2e3 into main Apr 8, 2022
@spiffcs spiffcs deleted the 887-detect-java-namespaces-group-id-by-hash branch April 8, 2022 19:12
spiffcs added a commit that referenced this pull request May 2, 2022
@spiffcs
Merge branch 'main' into 835-keyless-attestation-upgrade
cb25858 
* main: (31 commits)
  reduce noise of log output (#976)
  add version info and remove double config call (#977)
  Rename syft-id to package-id (#970)
  update to cyclonedx-go 0.5.2 (#971)
  refactor command package to remove globals and add dependency injection
  fix: #953 Derive language from pURL - https://github.com/anchore/syft… (#957)
  Fix typo in CPE-parsing error (#966)
  Preserve syft IDs on SBOM decode (#963)
  Update GitHub format package_url and correlator (#961)
  Ensure SPDXIDs are valid (#955)
  Auto-PR needs to run go mod tidy (#958)
  Add workflow for automatic PR for new stereoscope updates (#954)
  Minor readme update to correct format information (#948)
  Update spdx22json to only take uppercase checksum algorithm (#946)
  add additional vendors for springframework (#945)
  Add digest property to parent and nested java package metadata (#941)
  Update write permissions and log into ghcr.io for release (#942)
  Retry auth URL lookup without docker credentialhelper workaround (#939)
  Ensure that all cyclonedx components have bom-refs (#914)
  Additionally publish docker images to GHCR (#934)
  ...

Signed-off-by: Christopher Phillips <[email protected]>
rigzba21 pushed a commit to rigzba21/syft that referenced this pull request May 5, 2022
@spiffcs @rigzba21
Add digest property to parent and nested java package metadata (ancho…
07883c1 
…re#941)

Signed-off-by: rigzba21 <[email protected]>
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
@spiffcs
Add digest property to parent and nested java package metadata (ancho…
9decf1a 
…re#941)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wagoodman wagoodman wagoodman approved these changes

@kzantow kzantow kzantow approved these changes

+1 more reviewer

@luhring luhring luhring left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Detect Java Namespaces/Group IDs by hash

5 participants

@spiffcs @wagoodman @kzantow @luhring