-
Notifications
You must be signed in to change notification settings - Fork 311
Labels
bugIndicates an unexpected problem or unintended behaviorIndicates an unexpected problem or unintended behaviorcomponent: bots
Milestone
Description
Nullbytes in JSON-Data (in the field extra) can cause troubles:
psycopg2.errors.UntranslatableCharacter: unsupported Unicode escape sequence
DETAIL: \\u0000 cannot be converted to text.
CONTEXT: JSON data, line 1: ....0\"}, \"malware\": \"Conficker\", \"payload\": {\"text\":...
PL/pgSQL function directives_from_extra(bigint,json) line 5 during statement block local variable initialization
SQL statement \"SELECT directives_from_extra(NEW.id, NEW.extra)\"
PL/pgSQL function events_insert_directives_for_row() line 3 at PERFORM"
A problematic extra-value can look like:
'{"payload": {"text": "M41\u0012)3U>\bxӾ6\u0000\u0013M6M6M4M4]4y]4ӭ4"}}'
Source: Microsoft CTIP
While the json type in postgres allows (encoded) Nullbytes in principle, some systems have a problem with that. I have not found out yet, what causes some Postgres-Systems to throw an error here, while others work.
Tested with PostgreSQL 14.4. Works fine on e.g. openSUSE Tumbleweed, but not on Ubuntu 20.04 with Postgres-APT-Repo. But the latter is a target plattform of IntelMQ, so IntelMQ should be compatible with it.
The only solution I see is to escape the Nullbytes in the SQL Output for Postgres (SQLite does not have an issue with these characters).
Working on it.
Metadata
Metadata
Assignees
Labels
bugIndicates an unexpected problem or unintended behaviorIndicates an unexpected problem or unintended behaviorcomponent: bots