Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Reduce debug logging noise#3036

Merged
liquidsec merged 6 commits into
devfrom
logging-cleanup
May 18, 2026
Merged

Reduce debug logging noise#3036
liquidsec merged 6 commits into
devfrom
logging-cleanup

Conversation

@liquidsec
Copy link
Copy Markdown
Collaborator

@liquidsec liquidsec commented Apr 13, 2026

Cuts debug-log noise from a few high-volume sources. The biggest wins are in module-specific code; the core event pipeline is touched only where lines were redundant with adjacent ones.

HttpCompare (diff.py): drop per-comparison "status code different", "headers different", "body different", "ignored header removed", "switching to text parsing".

Serial submodule: drop 4 routine per-payload debugs (baseline match, probe result, "potential finding", "500 or body match"). Keep the useful ones (HttpCompareError, header-only diffs, unexpected status, transient baseline, error string detection).

Paramminer headers: log binary_search entry only on the top-level call (not every recursion — ~14k lines), and drop the per-word "Adding to wordlist".

Telerik: remove the stray self.debug(root_tool_path) line.

Event pipeline: only the lines whose info was duplicated by an adjacent log are removed — Got {event} from {module} (×2, duplicated by Handling {event} from {module}), {event} passed post-check (only the failure case is useful, which is already logged), Queueing {event} because {reason} (accept-side reason is rarely meaningful), Forwarding {event} (pairs trivially with Intercepting), and Not intercepting … precheck failed (mostly "watched_events didn't match" at huge volume). Finished handling, Not queueing, and Not intercepting … postcheck failed are kept.

@liquidsec
Copy link
Copy Markdown
Collaborator Author

improving issue raised in #2446

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 13, 2026

📊 Performance Benchmark Report

Comparing dev (baseline) vs logging-cleanup (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 4.23ms 4.37ms +3.3%
Bloom Filter Large Scale Dns Brute Force 17.85ms 17.81ms -0.2%
Large Closest Match Lookup 335.91ms 332.66ms -1.0%
Realistic Closest Match Workload 177.29ms 179.30ms +1.1%
Event Memory Medium Scan 1784 B/event 1784 B/event -0.0%
Event Memory Large Scan 1768 B/event 1768 B/event +0.0%
Event Validation Full Scan Startup Small Batch 375.05ms 377.06ms +0.5%
Event Validation Full Scan Startup Large Batch 532.38ms 533.91ms +0.3%
Make Event Autodetection Small 25.49ms 25.43ms -0.2%
Make Event Autodetection Large 255.10ms 255.68ms +0.2%
Make Event Explicit Types 10.76ms 10.75ms -0.1%
Excavate Single Thread Small 3.505s 3.441s -1.8%
Excavate Single Thread Large 8.892s 8.868s -0.3%
Excavate Parallel Tasks Small 3.730s 3.791s +1.6%
Excavate Parallel Tasks Large 6.154s 6.220s +1.1%
Is Ip Performance 3.19ms 3.18ms -0.3%
Make Ip Type Performance 11.76ms 11.39ms -3.1%
Mixed Ip Operations 4.58ms 4.47ms -2.3%
Memory Use Web Crawl 679.7 MB 658.5 MB -3.1%
Memory Use Subdomain Enum 33.3 MB 33.3 MB +0.0%
Memory Use Deep Chain 7.8 MB 7.8 MB +0.0%
Memory Use Parallel Chains 22.1 MB 20.3 MB -7.9%
Scan Throughput 100 3.733s 3.169s -15.1% 🟢🟢 🚀
Scan Throughput 1000 28.823s 21.966s -23.8% 🟢🟢🟢 🚀
Typical Queue Shuffle 64.99µs 61.46µs -5.4%
Priority Queue Shuffle 729.38µs 708.54µs -2.9%

🎯 Performance Summary

+ 2 improvements 🚀
  24 unchanged ✅

🔍 Significant Changes (>10%)

  • Scan Throughput 100: 15.1% 🚀 faster
  • Scan Throughput 1000: 23.8% 🚀 faster

🐍 Python Version 3.11.15

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (e892c8c) to head (c413772).
⚠️ Report is 26 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3036   +/-   ##
=====================================
- Coverage     90%     90%   -0%     
=====================================
  Files        444     444           
  Lines      38338   38319   -19     
=====================================
- Hits       34284   34262   -22     
- Misses      4054    4057    +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@liquidsec liquidsec changed the base branch from 3.0 to dev May 15, 2026 13:08
@liquidsec liquidsec requested a review from ausmaster May 15, 2026 13:22
# Conflicts:
#	bbot/core/helpers/diff.py
#	bbot/modules/base.py
@liquidsec liquidsec merged commit cc7e561 into dev May 18, 2026
19 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants