Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@FalafelGood
Copy link

@FalafelGood FalafelGood commented Feb 24, 2025

Description

(Related to #7539)
I've written a plugin for mitmproxy that enforces a whitelist with a valkey (redis) database.

On configure, this plugin:

  • Checks if the valkey server at the specified address is live.
  • Reads a list of allowed domain names from "whitelist.txt"
  • Uses a sha256 checksum to see if any changes were made to "whitelist.txt" since the last read
  • Pipes the elements into a valkey set called "whitelist"

On request, this plugin:

  • extracts the domain name from the http request
  • checks if the domain exists in the valkey database
  • If no, returns a custom 403 page.

Checklist

  • I have updated tests where applicable.
  • I have added an entry to the CHANGELOG.

Note: Because this is "just a plugin", I'm not sure if it's applicable for me to update tests and modify the changelog. Happy to do this if need be :-)

Alternatives to Valkey

Probably, you don't want to add a valkey dependency for mitmproxy just to support my random plugin. That's very understandable! A more knowledgeable friend of mine suggested I could work around this issue by writing a protocol for an abstract database object. I'm new to the concept of protocols, but I think I get the gist!

Let me know if this is the preferred approach, or if there's some other way I can improve this.
Open to feedback! Thanks for your time

@mhils
Copy link
Member

mhils commented Feb 25, 2025

Thank you for the PR! 🍰

  1. I think this fits best into /examples/contrib. We don't have to worry about dependencies and/or testing there. 😃
  2. s/whitelist/allowlist/ please, we want to stay consistent.
  3. Please still add a CHANGELOG entry :)

@SamuelMarks
Copy link

@mhils We were also thinking about having an abstractclass or Protocol for the whitelist. Perhaps with two implementations:

  1. Read from filesystem, store and query-from current-thread memory (builtin data-structure; maybe a frozenset?)
  2. Read from filesystem, store and query-from Valkey (the open-source redis fork)

Would you accept a PR for the abstractclass|Protocol & in-memory dependency-free implementation?

Then we can put the Valkey one in its own pip installable python package in a separate repository; and if your API changes we'll get a nice error in our codebase and don't need to have a separate fork to constantly keep in sync.

In our humble opinion: comprehensive whitelist support will greatly benefit the userbase—and encourage new users of—mitmproxy =)

@mhils
Copy link
Member

mhils commented Mar 7, 2025

I think this is really cool work and quite useful. But it's relatively specialized and I don't want to take over any maintenance burden. So putting it into examples/contrib is best for the time being.

@SamuelMarks
Copy link

@mhils How about a concurrently more extreme and more general solution, expanding your mitmproxy.addons.script to be able to refer to installed modules, that way we can maintain valkey & other whitelist support in an external Python package and use the script addon to load it in; something like:

diff --git a/mitmproxy/addons/script.py b/mitmproxy/addons/script.py
index bc5bc0e..c32aa54 100644
--- a/mitmproxy/addons/script.py
+++ b/mitmproxy/addons/script.py
@@ -1,6 +1,7 @@
 import asyncio
 import importlib.machinery
 import importlib.util
+import importlib
 import logging
 import os
 import sys
@@ -21,6 +22,11 @@ logger = logging.getLogger(__name__)
 
 
 def load_script(path: str) -> types.ModuleType | None:
+    if not os.path.isfile(path):
+        mod = importlib.import_module(path)
+        sys.modules[path] = mod
+        return mod
+
     fullname = "__mitmproxy_script__.{}".format(
         os.path.splitext(os.path.basename(path))[0]
     )

@FalafelGood
Copy link
Author

Hey @mhils, how does this look? I did everything you recommended. Seems like some checks are failing though... I don't know how to fix that, but if you point me in the right direction I'll give it my best shot!

Any thoughts on what @SamuelMarks suggested about supporting external plugins?

Thanks again for your time!

@mhils
Copy link
Member

mhils commented Apr 22, 2025

Thanks! Please remove the old file from mitmproxy/ and CI should turn green :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants