Thanks to visit codestin.com
Credit goes to github.com

Skip to content

♻️ Remove async import #12042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: dev
Choose a base branch
from
Open

♻️ Remove async import #12042

wants to merge 7 commits into from

Conversation

manuel-sommer
Copy link
Contributor

@manuel-sommer manuel-sommer commented Mar 18, 2025 •
edited
Loading

No description provided.

@github-actions github-actions bot added settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR docs labels Mar 18, 2025
@dryrunsecurity DryRunSecurity
Copy link

dryrunsecurity bot commented Mar 18, 2025 •
edited
Loading

DryRun Security Summary

The pull request removes deprecated asynchronous finding import features across multiple files, reducing security risks and simplifying system configuration by eliminating experimental code paths and potential concurrency-related vulnerabilities.

Expand for full summary

PR Summary: Removal of deprecated asynchronous finding import feature across multiple files, including documentation updates, code cleanup in importers, and configuration settings modifications.

Security Findings:

  • Removal of experimental async processing methods reduces potential race conditions and concurrency-related security risks
  • Eliminates configuration settings for async finding imports, simplifying the system's configuration
  • Reduces attack surface by removing deprecated and experimental code paths
  • Removes potential entry points for unintended behavior in import processes
  • Cleans up unused imports that could potentially introduce security complexities

No direct security vulnerabilities were introduced by these changes.

Code Analysis

We ran 7 analyzers against 6 files and 1 analyzer had findings. 6 analyzers had no findings.

Analyzer Findings
Configured Codepaths Analyzer 7 findings

Overall Riskiness

🔴 Risk threshold exceeded.

We've notified @mtesauro, @grendel513.

View PR in the DryRun Dashboard.

@manuel-sommer manuel-sommer reopened this Mar 18, 2025
@valentijnscholten
Copy link
Member

Can I suggest to change the title to "Remove async import"?

@manuel-sommer manuel-sommer changed the title ♻️ Deprecate async import ♻️ Remove async import Mar 19, 2025
@manuel-sommer
Copy link
Contributor Author

Done

@Maffooch
Copy link
Contributor

@manuel-sommer thank you for doing this! It will definitely save us some time in the future. We are planning to remove this functionality in the June release to provide folks enough awareness and time. The earliest we could merge this would be shortly after the May release

@manuel-sommer
Copy link
Contributor Author

Sure, feel free to merge it later. :-)

@github-actions GitHub Actions
Copy link
Contributor

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@manuel-sommer
Copy link
Contributor Author

I will resolve the conflicts once this will be picked up again.

@Maffooch Maffooch added this to the 2.47.0 milestone Apr 7, 2025
@github-actions GitHub Actions
Copy link
Contributor

Conflicts have been resolved. A maintainer will review the pull request shortly.

@dryrunsecurity DryRunSecurity
Copy link

dryrunsecurity bot commented Apr 15, 2025 •
edited
Loading

DryRun Security

🔴 Risk threshold exceeded.

This pull request involves sensitive edits to multiple importer files in the dojo/importers directory, with potential risks including the removal of async processing methods, reduced performance scalability, and decreased flexibility in processing strategies for DefectDojo's import functionality.

⚠️ Configured Codepaths Edit in dojo/importers/base_importer.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/default_importer.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/default_reimporter.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/endpoint_manager.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/endpoint_manager.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/endpoint_manager.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/endpoint_manager.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/base_importer.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/default_importer.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/default_reimporter.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/base_importer.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/default_importer.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
⚠️ Configured Codepaths Edit in dojo/importers/default_reimporter.py
Vulnerability Configured Codepaths Edit
Description Sensitive edits detected for this file. Sensitive file paths and allowed authors can be configured in .dryrunsecurity.yaml.
💭 Unconfirmed Findings (4)
Vulnerability Potential Deprecated Feature Removal Risk
Description Removal of async processing methods in base_importer.py could break existing custom importers that rely on the experimental async feature, potentially causing unexpected behavior during import processes.
Vulnerability Potential Loss of Performance Scalability
Description Removal of async processing and chunking logic in endpoint_manager.py may reduce the system's ability to distribute endpoint processing across workers, potentially impacting performance for large-scale imports.
Vulnerability Reduced Flexibility in Processing Strategy
Description Elimination of configurable async processing settings in endpoint_manager.py limits the application's adaptability to different performance requirements and processing strategies.
Vulnerability Potential Inconsistent Behavior in Older Versions
Description Removal of configuration for the experimental feature in settings.dist.py could lead to unexpected results and compatibility issues in older DefectDojo versions.

We've notified @mtesauro.

All finding details can be found in the DryRun Security Dashboard.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 5, 2025

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 6, 2025

Conflicts have been resolved. A maintainer will review the pull request shortly.

@manuel-sommer
Copy link
Contributor Author

Hi @mtesauro and @Maffooch ,
I rebased. Ready to be reviewed.

mtesauro
mtesauro approved these changes May 8, 2025
View reviewed changes
Copy link
Contributor

@mtesauro mtesauro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

valentijnscholten
valentijnscholten reviewed May 8, 2025
View reviewed changes
docs/content/en/open_source/installation/running-in-production.md
@@ -87,22 +87,3 @@ You can execute the following command to see the configuration:

`docker compose exec celerybeat bash -c "celery -A dojo inspect stats"`
and see what is in effect.

### Asynchronous Import
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to leave a short section behind that informs users the feature has been removed in 2.47.0? There might be users searching for async import in the docs.

valentijnscholten
valentijnscholten reviewed May 8, 2025
View reviewed changes
dojo/settings/settings.dist.py
@@ -273,12 +273,6 @@
DD_RATE_LIMITER_ACCOUNT_LOCKOUT=(bool, False),
# when enabled SonarQube API parser will download the security hotspots
DD_SONARQUBE_API_PARSER_HOTSPOTS=(bool, True),
# when enabled, finding importing will occur asynchronously, default False
# This experimental feature has been deprecated as of DefectDojo 2.44.0 (March release). Please exercise caution if using this feature with an older version of DefectDojo, as results may be inconsistent.
DD_ASYNC_FINDING_IMPORT=(bool, False),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If True, would it make sense to create an announcement banner or other alert to notify the users / admins that they are using a feature that is no longer present?

valentijnscholten
valentijnscholten reviewed May 8, 2025
View reviewed changes
Copy link
Member

@valentijnscholten valentijnscholten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup, left two comments which are not really blocking the merge, just some thoughts.

@Maffooch Maffooch requested a review from hblankenship May 8, 2025 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@valentijnscholten valentijnscholten valentijnscholten left review comments

@mtesauro mtesauro mtesauro approved these changes

@Maffooch Maffooch Awaiting requested review from Maffooch Maffooch is a code owner

@hblankenship hblankenship Awaiting requested review from hblankenship

At least 4 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
docs settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR
Projects
None yet
Milestone
2.47.0
Development

Successfully merging this pull request may close these issues.
4 participants
@manuel-sommer @valentijnscholten @Maffooch @mtesauro