Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@ryanohoro
Copy link
Collaborator

Describe the change

Fixes #284

Refactors tasting to process files first without modification.

Adds conditional transformations that occur with an additional taste_yara() pass.

Example: If a file starts with whitespace, the data is run through lstrip() before tasting again, and adding to the previous taste results.

Preserves existing behavior that excludes transformed files from taste_mime(). It can be enabled with a minor performance hit, but there's no clear benefit to the magic library yet.

Adds a test that includes HTML with leading whitespace.

Updated config tests to match.

Performance difference is negligible:

Old:

backend_1      | 2023-02-17 22:02:06 - [DEBUG] root [strelka.distribute]: File: test.html
backend_1      | 2023-02-17 22:02:06 - [DEBUG] root [strelka.match_flavors]: Took: 0.0013206005s
backend_1      | 2023-02-17 22:02:45 - [DEBUG] root [strelka.distribute]: File: test_whitespace.html
backend_1      | 2023-02-17 22:02:45 - [DEBUG] root [strelka.match_flavors]: Took: 0.0011620522s

New:

backend_1      | 2023-02-17 22:12:47 - [DEBUG] root [strelka.distribute]: File: test.html
backend_1      | 2023-02-17 22:12:47 - [DEBUG] root [strelka.match_flavors]: Took: 0.0013809204s
backend_1      | 2023-02-17 22:13:24 - [DEBUG] root [strelka.distribute]: File: test_whitespace.html
backend_1      | 2023-02-17 22:13:24 - [DEBUG] root [strelka.match_flavors]: Took: 0.0012507439s\

Describe testing procedures

docker-compose -f build/docker-compose.yaml build --build-arg CONFIG_TESTS=true

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka
plugins: mock-3.10.0, unordered-0.5.2
collected 122 items

...

tests/test_scan_html.py ..

...
====================== 122 passed, 28 warnings in 45.26s =======================

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka
plugins: mock-3.10.0, unordered-0.5.2
collected 184 items

tests_configuration/test_distribute.py ..
tests_configuration/test_scanner_assignment.py .........................................................................................
tests_configuration/test_scanner_import.py .
tests_configuration/test_taste.py ............................................................................................

======================= 184 passed, 5 warnings in 13.38s =======================

Sample output
If this change modifies Strelka's output, then please include a sample of the output here.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of and tested my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings

@ryanohoro ryanohoro marked this pull request as ready for review February 19, 2023 04:09
"test.tlsh": {
"mime": ["application/x-mach-binary"],
"yara": ["macho_file", "credit_cards"],
"yara": unordered(["macho_file", "credit_cards"]),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call on the inclusion of unordered in this test.

@phutelmyer phutelmyer merged commit a8f4de8 into target:master Feb 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] taste_yara() Strips Leading Whitespace, Impacting Taste Yara Rules

2 participants