One thing to note here is that with many blocklists, the matching algorithm has quite a performance penalty due to the subnet coverage calculation. This doesn't necessarily have to be a problem, as it's properly locked and is only for pre-processing, but it's slower than before.

Just to ease review a bit, the data path is as follows:

• Input:

[blocklist:1]
description=default
rcode=NOERROR
address=0.0.0.0
source_nets=
cache_ttl=72000
excludes.default=.*localhost$
excludes.patterns=
includes.patterns=
includes.wildcards=

[blocklist:2]
description=<second>
rcode=NOERROR
address=0.0.0.0
source_nets=192.168.2.0/24,192.168.1.0/24
cache_ttl=72000
blocklists.<shortcode>=<URL> <-- only exists if one or more types set
blocklists.custom<loop.index>=<URL> <-- only exists if one or more lists set
excludes.default=.*localhost$
excludes.patterns=
includes.patterns=
includes.wildcards=

• Output is a single JSON file with the following structure:

{
	"data": {
		"<domain>": [
			{"idx": <index into the first logical policy it should check},
			{"idx": ...}
			...
		],
	},
	"config": {
		"<idx>": {
			"passlist": <compiled passlist>,
			"source_nets": [
				"192.168.1.0/24",
				...
			]
			...
		}
	}
}

• There may be multiple policies associated with a single domain, as source nets can differ, as well as the allowlist (passlist here) regex.
• Config output is used directly in dnsbl_module.py and is returned when queried in test mode.
• To keep the resulting output small, the indices are simple integers - we assume other blocklist handlers may be injected, therefore the index must be managed by the BlocklistParser class.
  • We first merge all configurations, separated by index
  • The handlers return their blocklists, which is an array of dictionaries, all of which may contain domains that overlap in the final blocklist.
  • _merge_items_with_config merges the blocklists together and decides in what order the policies are attached to the domains -> more specific subnets have priority, and in that set the bigger passlists also have priority.