Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add ScanTlsh Scanner#296

Merged
phutelmyer merged 5 commits intomasterfrom
scan-tlsh
Jan 25, 2023
Merged

Add ScanTlsh Scanner#296
phutelmyer merged 5 commits intomasterfrom
scan-tlsh

Conversation

@phutelmyer
Copy link
Contributor

@phutelmyer phutelmyer commented Jan 24, 2023 •
edited
Loading

Describe the change
TLSH is a fuzzy matching algorithm that can be used for similarity comparisons. Similar files will have similar hash values which allows for the detection of similar objects by comparing their hash values TLSH has been adopted by a range of bodies and malware repositories.

This PR adds in a TLSH scanner, which compute the TLSH per file and compares against a local list of TLSH hashes to identify potential matches. The output of this scanner is a score based on a maximum threshold as well as a list of known TLSH hashes to compare against. If the comparison score between the source file and one of the list hashes is below a threshold (default: 30), a match is stored in the scanner's output.

The following chart provides an understanding of potential matches using score:

TLSH Comparison Score Score FP rate Detect rate
< 300 79.30% 98.8%
< 250 69.06% 98.8%
< 200 50.10% 98.8%
< 150 24.33% 98.1%
< 100 6.43% 94.5%
< 90 4.49% 92.3%
< 80 2.93% 89.0%
< 70 1.84% 83.6%
< 60 1.09% 76.0%
< 50 0.52% 65.3%
< 40 0.07% 49.6%
< 30 0.00181% 32.2%
< 20 0.00181% 17.3%
< 10 0.00181% 6.4%

With a default set to below 30, the expected false positive rate is low with an acceptable detection hit rate.

To use this scanner, you must generate a YAML file consisting of labels and corresponding TLSH values to compare against. The file at ./configs/python/backend/tlsh/rules.yaml is the default location for this match file, but can be overridden in the backend.yaml options for ScanTlsh.

Describe testing procedures
The scanner test: ./src/python/strelka/tests/test_scan_tlsh.py was created to validate this scanner.

Given an install of the requirements.txt file at ./src/python/requirements.txt, running src/python/strelka/tests/test_scan_tlsh.py should result in test success:

(virtualenv) test$ pytest src/python/strelka/tests/test_scan_tlsh.py 
=========================================================================================================================================================== test session starts ===========================================================================================================================================================
platform darwin -- Python 3.10.9, pytest-7.2.0, pluggy-1.0.0
plugins: unordered-0.5.2, mock-3.10.0
collected 1 item                                                                                                                                                                                                                                                                                                                          

src/python/strelka/tests/test_scan_tlsh.py .                                                                                                                                                                                                                                                                                        [100%]

============================================================================================================================================================ 1 passed in 0.69s ============================================================================================================================================================

Sample output

{
...
    "hash": {
      "elapsed": 0.000189,
      "md5": "83c2df5aad9adf3e761315baea9b5b68",
      "sha1": "5030560d3a8f7e363d802cb9b1e1c82a65d60de7",
      "sha256": "7ac19ffb133c73599774fcd0d056313c497f87d091ac9a08aa73d083aa67e2e7",
      "ssdeep": "96:9Y3uQPebqA96q+WVc1FPFB4A0BZzgk9JeQJHqpVRIFwppGh344:9Y+QPBOiGA0BdhEZIyP4",
      "tlsh": "T1538165343EAA430247F4217ED7D5C9BFE288F41027CAA299D8A5FD56945E918D323294"
    },
    "header": {
      "backslash": "Lorem ipsum dolor sit amet, consectetur adipiscing",
      "elapsed": 0.000062,
      "header": "Lorem ipsum dolor sit amet, consectetur adipiscing"
    },
    "tlsh": {
      "elapsed": 0.000104,
      "match": {
        "family": "FamilyA",
        "score": 6
      }
    },
...

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of and tested my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings

ryanohoro
ryanohoro reviewed Jan 25, 2023
View reviewed changes
Copy link
Collaborator

@ryanohoro ryanohoro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to work as advertised

ryanohoro
ryanohoro reviewed Jan 25, 2023
View reviewed changes
@phutelmyer phutelmyer merged commit 709cc4e into master Jan 25, 2023
@phutelmyer phutelmyer deleted the scan-tlsh branch January 30, 2023 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ryanohoro ryanohoro ryanohoro left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@phutelmyer @ryanohoro