Add HAR file writing support (#439) #440

aidansteele · 2023-02-12T01:38:00Z

Hi,

Here is the draft PR that I mentioned in #439. It is incomplete and only a draft right now, but I am sharing it early to get your feedback.

Which HAR package to use?

I looked at https://github.com/vvakame/go-harlog and some other packages (I found other packages by going through the results of this GitHub search). There were some issues with the other projects that already existed, so I created my own by forking github.com/vvakame/go-harlog. Those issues were:

Their design required that you use one http.RoundTripper per HAR archive. Dalfox creates a http.RoundTripper in a few different places and I didn't want to make big changes to Dalfox.
Their design would store the HAR archive in memory and serialize to JSON at the end. Dalfox can make a lot of requests and I didn't want Dalfox to use lots of memory, so I wanted something that would write the file incrementally during the scan.
Their design would only allow standard HAR files to be written, without modification. I wanted to be able to add a _messageId property to the HAR entries to correlate with Dalfox PoCs like ZAP does.

So that's why I created a new HAR package. The package does the same thing Dalfox does in JSON mode: it writes the [ first, then each JSON object, then commas, then the closing ]. It also allows multiple http.RoundTripper objects (in this case, *har.RoundTripper) to write to a single shared har.Writer.

Question: should it be a standalone package that Dalfox imports in its go.mod or would you prefer it to be contained entirely in Dalfox, e.g. github.com/hahwul/dalfox/pkg/har? I think I prefer it to be a Dalfox package, but I don't know how much code you want in the Dalfox repo. Let me know what you prefer.

Integration into Dalfox

What do you think of the changes to Dalfox so far? Is the design ok? Are there any changes you would like me to make? Right now the code works correctly and generates a HAR archive. I will add another commit that adds a _messageId property to the HAR entries and the Dalfox PoC JSON soon. Maybe in a couple of days.

codecov-commenter · 2023-02-12T13:11:33Z

Codecov Report

Merging #440 (c3a2e50) into main (f78a653) will not change coverage.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@           Coverage Diff           @@
##             main     #440   +/-   ##
=======================================
  Coverage   82.88%   82.88%           
=======================================
  Files          11       11           
  Lines         590      590           
=======================================
  Hits          489      489           
  Misses         79       79           
  Partials       22       22

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

hahwul · 2023-02-12T14:32:46Z

Hello again @aidansteele
First, Thank you so much for cool idea and PR :D

The answer to the question is I don't care. (I like both)
Of course, I prefer to be included package in Dalfox as you say. The reason is the advantage of fast maintenance and low-dependency. But I don't think it necessarily needs to be changed.

And yes I like the changes so far. I'm looking forward to the next commit as well :D

Dalfox has a separate Options Interface for importing and using Dalfox from other Go packages. and I think we need to write additional code for this part. I'll take care of it. (this interface was my stupid code. haha...😭)

aidansteele · 2023-02-13T03:43:29Z

@hahwul thank you for your feedback. I agree that it is easier to maintain if the code is in this repo, so I moved the package here. I've now implemented the code to add the "message IDs" to HAR entries and Dalfox output. Here is some demo of the changes:

Output (pretty-printed) from running url --no-spinner --no-color --output-all --follow-redirects --silence --format json http://testphp.vulnweb.com/listproducts.php?cat=2 --har-file-path dump.har:

Dalfox output (click to expand)

[
  {
    "type": "G",
    "inject_type": "BUILTIN",
    "poc_type": "plain",
    "method": "GET",
    "data": "http://testphp.vulnweb.com/listproducts.php?cat=%2F%2F%2Fwww.google.com%2F%252f%252e%252e",
    "param": "",
    "payload": "toOpenRedirecting",
    "evidence": "",
    "cwe": "",
    "severity": "Low",
    "message_id": 13
  },
  {
    "type": "R",
    "inject_type": "inHTML-URL",
    "poc_type": "plain",
    "method": "GET",
    "data": "http://testphp.vulnweb.com/listproducts.php?cat=2%22%3E%3Csvg%2FOnLoad%3D%22%60%24%7Bprompt%60%60%7D%60%22%3E",
    "param": "cat",
    "payload": "\"><svg/OnLoad=\"`${prompt``}`\">",
    "evidence": "48 line:  syntax to use near '\"><svg/OnLoad=\"`${prompt``}`\">' at line 1",
    "cwe": "CWE-79",
    "severity": "Medium",
    "message_id": 334
  },
  {
    "type": "R",
    "inject_type": "inHTML-URL",
    "poc_type": "plain",
    "method": "GET",
    "data": "http://testphp.vulnweb.com/listproducts.php?cat=2%27%3E%3Cimg%2Fsrc%2Fonerror%3D.1%7Calert%60%60%3E",
    "param": "cat",
    "payload": "'><img/src/onerror=.1|alert``>",
    "evidence": "48 line:  syntax to use near ''><img/src/onerror=.1|alert``>' at line 1",
    "cwe": "CWE-79",
    "severity": "Medium",
    "message_id": 342
  },
  {
    "type": "R",
    "inject_type": "inHTML-URL",
    "poc_type": "plain",
    "method": "GET",
    "data": "http://testphp.vulnweb.com/listproducts.php?cat=2%22%3E%3Cscript+y%3D%22%3E%3C%22%3E%2F%2A%3Cscript%2A+%2A%2Fprompt%28%29%3C%2Fscript",
    "param": "cat",
    "payload": "\"><script y=\"><\">/*<script* */prompt()</script",
    "evidence": "48 line:  syntax to use near '\"><script y=\"><\">/*<script* */prompt()</script' at line 1",
    "cwe": "CWE-79",
    "severity": "Medium",
    "message_id": 358
  },
  {
    "type": "V",
    "inject_type": "inHTML-URL",
    "poc_type": "plain",
    "method": "GET",
    "data": "http://testphp.vulnweb.com/listproducts.php?cat=2%27%22%3E%3Csvg%2Fclass%3Ddalfox+onload%3D%26%2397%26%23108%26%23101%26%23114%26%2300116%26%2340%26%2341%26%23x2f%26%23x2f",
    "param": "cat",
    "payload": "'\"><svg/class=dalfox onload=&#97&#108&#101&#114&#00116&#40&#41&#x2f&#x2f",
    "evidence": "48 line:  syntax to use near ''\"><svg/class=dalfox onload=&#97&#108&#101&#114&#00116&#40&#",
    "cwe": "CWE-79",
    "severity": "High",
    "message_id": 322
  },
  {}
]

And here's validating the entry appears in the HAR file by running cat dump.har| jq '.log.entries | map(select(._messageId == 322))':

jq output (click to expand)

[
  {
    "startedDateTime": "2023-02-13T14:32:31+11:00",
    "time": 413.50575,
    "request": {
      "method": "GET",
      "url": "http://testphp.vulnweb.com/listproducts.php?cat=2%27%22%3E%3Csvg%2Fclass%3Ddalfox+onload%3D%26%2397%26%23108%26%23101%26%23114%26%2300116%26%2340%26%2341%26%23x2f%26%23x2f",
      "httpVersion": "HTTP/1.1",
      "cookies": [],
      "headers": [
        {
          "name": "Accept",
          "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
        },
        {
          "name": "User-Agent",
          "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:75.0) Gecko/20100101 Firefox/75.0"
        }
      ],
      "queryString": [
        {
          "name": "cat",
          "value": "2'\"><svg/class=dalfox onload=&#97&#108&#101&#114&#00116&#40&#41&#x2f&#x2f"
        }
      ],
      "headersSize": -1,
      "bodySize": -1
    },
    "response": {
      "status": 200,
      "statusText": "200 OK",
      "httpVersion": "HTTP/1.1",
      "cookies": [],
      "headers": [
        {
          "name": "Server",
          "value": "nginx/1.19.0"
        },
        {
          "name": "Date",
          "value": "Mon, 13 Feb 2023 03:32:31 GMT"
        },
        {
          "name": "Content-Type",
          "value": "text/html; charset=UTF-8"
        },
        {
          "name": "X-Powered-By",
          "value": "PHP/5.6.40-38+ubuntu20.04.1+deb.sury.org+1"
        }
      ],
      "content": {
        "size": -1,
        "mimeType": "text/html; charset=UTF-8",
        "text": "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\"\n\"http://www.w3.org/TR/html4/loose.dtd\">\n<html><!-- InstanceBegin template=\"/Templates/main_dynamic_template.dwt.php\" codeOutsideHTMLIsLocked=\"false\" -->\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-2\">\n\n<!-- InstanceBeginEditable name=\"document_title_rgn\" -->\n<title>pictures</title>\n<!-- InstanceEndEditable -->\n<link rel=\"stylesheet\" href=\"style.css\" type=\"text/css\">\n<!-- InstanceBeginEditable name=\"headers_rgn\" -->\n<!-- InstanceEndEditable -->\n<script language=\"JavaScript\" type=\"text/JavaScript\">\n<!--\nfunction MM_reloadPage(init) {  //reloads the window if Nav4 resized\n  if (init==true) with (navigator) {if ((appName==\"Netscape\")&&(parseInt(appVersion)==4)) {\n    document.MM_pgW=innerWidth; document.MM_pgH=innerHeight; onresize=MM_reloadPage; }}\n  else if (innerWidth!=document.MM_pgW || innerHeight!=document.MM_pgH) location.reload();\n}\nMM_reloadPage(true);\n//-->\n</script>\n\n</head>\n<body> \n<div id=\"mainLayer\" style=\"position:absolute; width:700px; z-index:1\">\n<div id=\"masthead\"> \n  <h1 id=\"siteName\"><a href=\"https://www.acunetix.com/\"><img src=\"images/logo.gif\" width=\"306\" height=\"38\" border=\"0\" alt=\"Acunetix website security\"></a></h1>   \n  <h6 id=\"siteInfo\">TEST and Demonstration site for <a href=\"https://www.acunetix.com/vulnerability-scanner/\">Acunetix Web Vulnerability Scanner</a></h6>\n  <div id=\"globalNav\"> \n      \t<table border=\"0\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\"><tr>\n\t<td align=\"left\">\n\t\t<a href=\"index.php\">home</a> | <a href=\"categories.php\">categories</a> | <a href=\"artists.php\">artists\n\t\t</a> | <a href=\"disclaimer.php\">disclaimer</a> | <a href=\"cart.php\">your cart</a> | \n\t\t<a href=\"guestbook.php\">guestbook</a> | \n\t\t<a href=\"AJAX/index.php\">AJAX Demo</a>\n\t</td>\n\t<td align=\"right\">\n\t\t</td>\n\t</tr></table>\n  </div> \n</div> \n<!-- end masthead --> \n\n<!-- begin content -->\n<!-- InstanceBeginEditable name=\"content_rgn\" -->\n<div id=\"content\">\n\tError: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''\"><svg/class=dalfox onload=&#97&#108&#101&#114&#00116&#40&#41&#x2f&#x2f' at line 1\nWarning: mysql_fetch_array() expects parameter 1 to be resource, boolean given in /hj/var/www/listproducts.php on line 74\n</div>\n<!-- InstanceEndEditable -->\n<!--end content -->\n\n<div id=\"navBar\"> \n  <div id=\"search\"> \n    <form action=\"search.php?test=query\" method=\"post\"> \n      <label>search art</label> \n      <input name=\"searchFor\" type=\"text\" size=\"10\"> \n      <input name=\"goButton\" type=\"submit\" value=\"go\"> \n    </form> \n  </div> \n  <div id=\"sectionLinks\"> \n    <ul> \n      <li><a href=\"categories.php\">Browse categories</a></li> \n      <li><a href=\"artists.php\">Browse artists</a></li> \n      <li><a href=\"cart.php\">Your cart</a></li> \n      <li><a href=\"login.php\">Signup</a></li>\n\t  <li><a href=\"userinfo.php\">Your profile</a></li>\n\t  <li><a href=\"guestbook.php\">Our guestbook</a></li>\n\t\t<li><a href=\"AJAX/index.php\">AJAX Demo</a></li>\n\t  </li> \n    </ul> \n  </div> \n  <div class=\"relatedLinks\"> \n    <h3>Links</h3> \n    <ul> \n      <li><a href=\"http://www.acunetix.com\">Security art</a></li> \n\t  <li><a href=\"https://www.acunetix.com/vulnerability-scanner/php-security-scanner/\">PHP scanner</a></li>\n\t  <li><a href=\"https://www.acunetix.com/blog/articles/prevent-sql-injection-vulnerabilities-in-php-applications/\">PHP vuln help</a></li>\n\t  <li><a href=\"http://www.eclectasy.com/Fractal-Explorer/index.html\">Fractal Explorer</a></li> \n    </ul> \n  </div> \n  <div id=\"advert\"> \n    <p>\n      <object classid=\"clsid:D27CDB6E-AE6D-11cf-96B8-444553540000\" codebase=\"http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,29,0\" width=\"107\" height=\"66\">\n        <param name=\"movie\" value=\"Flash/add.swf\">\n        <param name=quality value=high>\n        <embed src=\"Flash/add.swf\" quality=high pluginspage=\"http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash\" type=\"application/x-shockwave-flash\" width=\"107\" height=\"66\"></embed>\n      </object>\n    </p>\n  </div> \n</div> \n\n<!--end navbar --> \n<div id=\"siteInfo\">  <a href=\"http://www.acunetix.com\">About Us</a> | <a href=\"privacy.php\">Privacy Policy</a> | <a href=\"mailto:[email protected]\">Contact Us</a> | &copy;2019\n  Acunetix Ltd \n</div> \n<br> \n<div style=\"background-color:lightgray;width:100%;text-align:center;font-size:12px;padding:1px\">\n<p style=\"padding-left:5%;padding-right:5%\"><b>Warning</b>: This is not a real shop. This is an example PHP application, which is intentionally vulnerable to web attacks. It is intended to help you test Acunetix. It also helps you understand how developer errors and bad configuration may let someone break into your website. You can use it to test other tools and your manual hacking skills as well. Tip: Look for potential SQL Injections, Cross-site Scripting (XSS), and Cross-site Request Forgery (CSRF), and more.</p>\n</div>\n</div>\n</body>\n<!-- InstanceEnd --></html>\n"
      },
      "redirectURL": "",
      "headersSize": -1,
      "bodySize": -1
    },
    "cache": null,
    "timings": {
      "blocked": 0.001791,
      "dns": 0.775084,
      "connect": 190.115667,
      "send": 0.016708,
      "wait": 222.568834,
      "receive": 0.80275,
      "ssl": -1
    },
    "_messageId": 322
  }
]

And here's an example of the HAR file being loaded in to Chrome dev tools:

aidansteele · 2023-02-13T03:50:47Z

I tried to fix all the Codacy issues but I don't know what to do with these ones. I have already added a package-level comment to round_tripper.go, so I don't think these other files need comments.

aidansteele · 2023-02-13T03:51:59Z

I also just force-pushed a squashed commit, so there won't be lots of fix-commits if this is merged 😂

hahwul

I'm sorry I'm late.
I also tested it locally with the this PR and both Unit test and execution work well.

Codacy issues are occurring as a whole whether the pattern has been added, so I made an exception. (I think the issue will probably be removed from the next scan)

Lastly, thank you so much for your wonderful contribution! If there are no more changes, I would like to merge. What do you think?

I will write the code for the interface issue I mentioned before with additional PR.

aidansteele · 2023-02-19T23:15:13Z

@hahwul thank you for reviewing it! I don't think there are any more changes that I need to make. So it can be merged now. Of course if you find a bug after merging then let me know and I will be happy to try fix it ASAP. But I have been using my fork for the last week and it seems to work :)

Also if you want me to help look at your PR for the interface thing then I would be happy to!

Thanks 💯

add HAR file writing support (hahwul#439)

b3dc29b

aidansteele force-pushed the main branch from 1a47eb2 to b3dc29b Compare February 13, 2023 03:51

aidansteele marked this pull request as ready for review February 13, 2023 03:52

hahwul approved these changes Feb 18, 2023

View reviewed changes

hahwul added this to the v2.9.0 milestone Feb 18, 2023

hahwul merged commit 09468c4 into hahwul:main Feb 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add HAR file writing support (#439) #440

Add HAR file writing support (#439) #440

Uh oh!

aidansteele commented Feb 12, 2023

Uh oh!

codecov-commenter commented Feb 12, 2023

Uh oh!

hahwul commented Feb 12, 2023

Uh oh!

aidansteele commented Feb 13, 2023 •

edited

Loading

Uh oh!

aidansteele commented Feb 13, 2023

Uh oh!

aidansteele commented Feb 13, 2023

Uh oh!

hahwul left a comment

Uh oh!

aidansteele commented Feb 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Add HAR file writing support (#439) #440

Add HAR file writing support (#439) #440

Uh oh!

Conversation

aidansteele commented Feb 12, 2023

Which HAR package to use?

Integration into Dalfox

Uh oh!

codecov-commenter commented Feb 12, 2023

Codecov Report

Uh oh!

hahwul commented Feb 12, 2023

Uh oh!

aidansteele commented Feb 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aidansteele commented Feb 13, 2023

Uh oh!

aidansteele commented Feb 13, 2023

Uh oh!

hahwul left a comment

Choose a reason for hiding this comment

Uh oh!

aidansteele commented Feb 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aidansteele commented Feb 13, 2023 •

edited

Loading