KB is a new feature not in docs yet, it is just being developed.

KB is supposed to log interesting L7 information into an internal tree followint:

domain > hostname > protocol > protocol specific details

Tree can be dumped into JSON file by command, or by the scheduler. The tree is limited in number of tree nodes at a time, fresh nodes are kept, old are removed.

In current devel version (pre 0.9.31) it works for first few streams of HTTP/2 session. Engine is active only for certain amount of bytes in the beginning of TCP/UDP connection. Node database is limited to 1000 entries (will be configurable).

Currently, it looks as follows:

smithproxy(sx1)# execute kb print 
Knowledgebase dump:
{
    "com.bing.": {
        "www.bing.com": {
            "/hp/api/v1/imagegallery?format=json&setmkt=en-cz&today=1": {
                ":status": {
                    ".": 200,
                    "counter": 1
                },
                "set-cookie": {
                    "@1652926530": "_EDGE_S=SID=XXXXXXXXXXXXXXXXX&mkt=en-cz; domain=.bing.com; path=/; HttpOnly"
                }
            },
            "cookie": {
                "@1652926530": "_SS=SID=XXXXXXXXXXXXXXXXX"
            }
        }
    },
    "com.bitwarden.": {
        "identity.bitwarden.com": {
            "/connect/token":

The plan is:

Improve engine to work in some limited fashion also later in the TCP/UDP stream. This works already for DoH traffic, but on larger volumes it can become quite CPU intensive, which I want to avoid.
I have to also improve HTTP/1 engine substantially to match current HTTP/2 engine.
Having above issues sorted out, I can dump KB export periodically.

JSON is not great for appending to existing files, one must read it, parse it, add new data into the three and write again. If you look for more continuous data to, say, tail -f it, probably CSV format would be better.

Uh oh!

Lite logging #28

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions