Thanks to visit codestin.com
Credit goes to github.com

Skip to content

PostgreSQL Output: Issue with Nullbytes in JSON-Data #2203

@wagner-intevation

Description

@wagner-intevation

Nullbytes in JSON-Data (in the field extra) can cause troubles:

psycopg2.errors.UntranslatableCharacter: unsupported Unicode escape sequence
DETAIL:  \\u0000 cannot be converted to text.
CONTEXT:  JSON data, line 1: ....0\"}, \"malware\": \"Conficker\", \"payload\": {\"text\":...
PL/pgSQL function directives_from_extra(bigint,json) line 5 during statement block local variable initialization
SQL statement \"SELECT directives_from_extra(NEW.id, NEW.extra)\"
PL/pgSQL function events_insert_directives_for_row() line 3 at PERFORM"

A problematic extra-value can look like:

'{"payload": {"text": "M41\u0012)3U>\bxӾ6\u0000\u0013M6M6M4M4]4y]4ӭ4"}}'

Source: Microsoft CTIP

While the json type in postgres allows (encoded) Nullbytes in principle, some systems have a problem with that. I have not found out yet, what causes some Postgres-Systems to throw an error here, while others work.
Tested with PostgreSQL 14.4. Works fine on e.g. openSUSE Tumbleweed, but not on Ubuntu 20.04 with Postgres-APT-Repo. But the latter is a target plattform of IntelMQ, so IntelMQ should be compatible with it.

The only solution I see is to escape the Nullbytes in the SQL Output for Postgres (SQLite does not have an issue with these characters).
Working on it.

Metadata

Metadata

Labels

bugIndicates an unexpected problem or unintended behaviorcomponent: bots

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions