Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Update: Add support for x_forwarded_for headers in apaches access logs #3251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Aug 29, 2022
Merged

Update: Add support for x_forwarded_for headers in apaches access logs #3251

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
c953563
Update: Add support for x_forwarded_for headers in apaches access logs
redcinelli May 2, 2022
0058ab5
Update: Pr link in the changelogs
redcinelli May 2, 2022
28ed558
Update: Painless check property exist before referencing it
redcinelli May 2, 2022
dbbe9a6
Update: simplify grok expression
redcinelli May 16, 2022
986bd02
Update: x-forwarded-for match only ip
redcinelli May 17, 2022
5904f15
Update: migrate ip adress string analysis to CIDR
redcinelli May 17, 2022
16579c1
Fix: support for vhost at the begining of the log
redcinelli May 17, 2022
96f1e32
Update: refactor try/catch, ctx can't be null, remove redundancy
redcinelli May 17, 2022
ac90936
Update: Updated format for apache logs
redcinelli May 18, 2022
3e5c9f8
Update: Fill client.ip instead of source.address
redcinelli May 23, 2022
73ecf83
Update: Add some documentation to explain supported format
redcinelli Jun 3, 2022
15d0614
Update packages/apache/_dev/build/docs/README.md
redcinelli Jun 7, 2022
1047981
Update: Build integration to generate Md doc
redcinelli Jun 7, 2022
499515f
Update: manifest.yml
redcinelli Jun 7, 2022
ded37f9
Merge branch 'main' into apache/support_x_forwarded_for_headers
redcinelli Jul 29, 2022
2c91c78
Fix: issue during merge
redcinelli Jul 29, 2022
3c9d133
Merge branch 'main' into apache/support_x_forwarded_for_headers
redcinelli Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update: Painless check property exist before referencing it
  • Loading branch information
@redcinelli
redcinelli committed Jul 18, 2022
commit 28ed55832c089f5ded7c6770fb162be5cec6068c
18 changes: 9 additions & 9 deletions packages/apache/data_stream/access/_dev/test/pipeline/test-access-basic.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508515600Z",
"ingested": "2022-05-02T13:43:24.918687300Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:29 +0200] \"GET /favicon.ico HTTP/1.1\" 404 209",
"outcome": "failure"
Expand Down Expand Up @@ -66,7 +66,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508560800Z",
"ingested": "2022-05-02T13:43:24.918722900Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:13 +0000] \"GET /hello HTTP/1.1\" 404 499 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down Expand Up @@ -130,7 +130,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508589600Z",
"ingested": "2022-05-02T13:43:24.918742700Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:48 +0200] \"-\" 408 -",
"outcome": "failure"
Expand Down Expand Up @@ -169,7 +169,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508609200Z",
"ingested": "2022-05-02T13:43:24.918760800Z",
"kind": "event",
"original": "172.17.0.1 - - [29/May/2017:19:02:48 +0000] \"GET /stringpatch HTTP/1.1\" 404 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" \"-\"",
"outcome": "failure"
Expand Down Expand Up @@ -229,7 +229,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508627800Z",
"ingested": "2022-05-02T13:43:24.918778100Z",
"kind": "event",
"original": "monitoring-server - - [29/May/2017:19:02:48 +0000] \"GET /status HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" \"-\"",
"outcome": "success"
Expand Down Expand Up @@ -290,7 +290,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508646100Z",
"ingested": "2022-05-02T13:43:24.918795700Z",
"kind": "event",
"original": "127.0.0.1 - - [02/Feb/2019:05:38:45 +0100] \"-\" 408 152 \"-\" \"-\"",
"outcome": "failure"
Expand Down Expand Up @@ -338,7 +338,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508664600Z",
"ingested": "2022-05-02T13:43:24.918811600Z",
"kind": "event",
"original": "monitoring-server - - [29/May/2017:19:02:48 +0000] \"GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" \"-\"",
"outcome": "success"
Expand Down Expand Up @@ -402,7 +402,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508683200Z",
"ingested": "2022-05-02T13:43:24.918823600Z",
"kind": "event",
"original": "10.0.0.2, 10.0.0.1, 89.160.20.112 - - [29/May/2017:19:02:48 +0000] \"GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" \"-\"",
"outcome": "success"
Expand Down Expand Up @@ -487,7 +487,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.508701700Z",
"ingested": "2022-05-02T13:43:24.918841300Z",
"kind": "event",
"original": "2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6, 10.225.192.17, 10.2.2.121 - - [29/May/2017:19:02:48 +0000] \"GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" \"-\"",
"outcome": "success"
Expand Down
12 changes: 6 additions & 6 deletions packages/apache/data_stream/access/_dev/test/pipeline/test-access-darwin.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.979121500Z",
"ingested": "2022-05-02T13:43:25.319844800Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:28 +0200] \"GET / HTTP/1.1\" 200 45",
"outcome": "success"
Expand Down Expand Up @@ -65,7 +65,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.979154800Z",
"ingested": "2022-05-02T13:43:25.319904Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:29 +0200] \"GET /favicon.ico HTTP/1.1\" 404 209",
"outcome": "failure"
Expand Down Expand Up @@ -116,7 +116,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.979172400Z",
"ingested": "2022-05-02T13:43:25.319917600Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:48 +0200] \"-\" 408 -",
"outcome": "failure"
Expand Down Expand Up @@ -155,7 +155,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.979182800Z",
"ingested": "2022-05-02T13:43:25.319926800Z",
"kind": "event",
"original": "89.160.20.156 - - [26/Dec/2016:18:23:35 +0200] \"GET / HTTP/1.1\" 200 45",
"outcome": "success"
Expand Down Expand Up @@ -223,7 +223,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.979189400Z",
"ingested": "2022-05-02T13:43:25.319936200Z",
"kind": "event",
"original": "89.160.20.156 - - [26/Dec/2016:18:23:41 +0200] \"GET /notfound HTTP/1.1\" 404 206",
"outcome": "failure"
Expand Down Expand Up @@ -291,7 +291,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:23.979200Z",
"ingested": "2022-05-02T13:43:25.319970600Z",
"kind": "event",
"original": "89.160.20.156 - - [26/Dec/2016:18:23:45 +0200] \"GET /hmm HTTP/1.1\" 404 201",
"outcome": "failure"
Expand Down
4 changes: 2 additions & 2 deletions ...es/apache/data_stream/access/_dev/test/pipeline/test-access-ssl-request.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.051311500Z",
"ingested": "2022-05-02T13:43:25.387258300Z",
"kind": "event",
"original": "[10/Aug/2018:09:45:56 +0200] 172.30.0.119 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 \"GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax\u0026amp;opts=%7B%22func%22%3A%22get_admin_tasks_html%22%2C%22args%22%3A%22%22%7D\u0026amp;nsp=b5c7d5d4b6f7d0cf0c92f9cbdf737f6a5c838218425e6ae21 HTTP/1.1\" 1375"
},
Expand Down Expand Up @@ -69,7 +69,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.051380200Z",
"ingested": "2022-05-02T13:43:25.387286100Z",
"kind": "event",
"original": "[16/Oct/2019:11:53:47 +0200] 89.160.20.156 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 \"GET /appl/ajaxhelper.php?cmd=getxicoreajax\u0026opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D\u0026nsp=c2700eab9797eda8a9f65a3ab17a6adbceccd60a6cca7708650a5923950d HTTP/1.1\" -"
},
Expand Down
18 changes: 9 additions & 9 deletions packages/apache/data_stream/access/_dev/test/pipeline/test-access-ubuntu.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.115992800Z",
"ingested": "2022-05-02T13:43:25.443545400Z",
"kind": "event",
"original": "127.0.0.1 - - [26/Dec/2016:16:18:09 +0000] \"GET / HTTP/1.1\" 200 491 \"-\" \"Wget/1.13.4 (linux-gnu)\"",
"outcome": "success"
Expand Down Expand Up @@ -77,7 +77,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116024800Z",
"ingested": "2022-05-02T13:43:25.443576200Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:00 +0000] \"GET / HTTP/1.1\" 200 484 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36\"",
"outcome": "success"
Expand Down Expand Up @@ -141,7 +141,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116039200Z",
"ingested": "2022-05-02T13:43:25.443593400Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:00 +0000] \"GET /favicon.ico HTTP/1.1\" 404 504 \"http://192.168.33.72/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36\"",
"outcome": "failure"
Expand Down Expand Up @@ -206,7 +206,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116056100Z",
"ingested": "2022-05-02T13:43:25.443607Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:08 +0000] \"GET / HTTP/1.1\" 200 484 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "success"
Expand Down Expand Up @@ -270,7 +270,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116073600Z",
"ingested": "2022-05-02T13:43:25.443618200Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:08 +0000] \"GET /favicon.ico HTTP/1.1\" 404 504 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down Expand Up @@ -335,7 +335,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116098300Z",
"ingested": "2022-05-02T13:43:25.443632Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:08 +0000] \"GET /favicon.ico HTTP/1.1\" 404 504 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down Expand Up @@ -400,7 +400,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116111100Z",
"ingested": "2022-05-02T13:43:25.443644600Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:10 +0000] \"GET /test HTTP/1.1\" 404 498 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down Expand Up @@ -464,7 +464,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116126200Z",
"ingested": "2022-05-02T13:43:25.443657700Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:13 +0000] \"GET /hello HTTP/1.1\" 404 499 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down Expand Up @@ -528,7 +528,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.116138600Z",
"ingested": "2022-05-02T13:43:25.443668600Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:17 +0000] \"GET /crap HTTP/1.1\" 404 499 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down
2 changes: 1 addition & 1 deletion packages/apache/data_stream/access/_dev/test/pipeline/test-access-vhost.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-05-02T12:08:24.211525800Z",
"ingested": "2022-05-02T13:43:25.540229800Z",
"kind": "event",
"original": "vhost1.domaine.fr 192.168.33.2 - - [26/Dec/2016:16:22:14 +0000] \"GET /hello HTTP/1.1\" 404 499 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down
4 changes: 4 additions & 0 deletions packages/apache/data_stream/access/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ processors:
}
}
try {
if (ctx?.source == null){
Map map = new HashMap();
ctx.put("source", map);
}
ctx.source.address = null;
if (ctx.apache.access.remote_ip_list == null) {
return;
Expand Down
8 changes: 4 additions & 4 deletions packages/apache/data_stream/error/_dev/test/pipeline/test-error-basic.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
},
"event": {
"category": "web",
"ingested": "2022-05-02T12:08:24.491755700Z",
"ingested": "2022-05-02T13:43:25.779939400Z",
"kind": "event",
"original": "[Mon Dec 26 16:22:08 2016] [error] [client 192.168.33.1] File does not exist: /var/www/favicon.ico",
"timezone": "GMT+2",
Expand Down Expand Up @@ -43,7 +43,7 @@
},
"event": {
"category": "web",
"ingested": "2022-05-02T12:08:24.491785800Z",
"ingested": "2022-05-02T13:43:25.779969Z",
"kind": "event",
"original": "[Mon Dec 26 16:15:55.103786 2016] [core:notice] [pid 11379] AH00094: Command line: '/usr/local/Cellar/httpd24/2.4.23_2/bin/httpd'",
"timezone": "GMT+2",
Expand Down Expand Up @@ -72,7 +72,7 @@
},
"event": {
"category": "web",
"ingested": "2022-05-02T12:08:24.491803700Z",
"ingested": "2022-05-02T13:43:25.779984900Z",
"kind": "event",
"original": "[Fri Sep 09 10:42:29.902022 2011] [core:error] [pid 35708:tid 4328636416] [client 89.160.20.156] File does not exist: /usr/local/apache2/htdocs/favicon.ico",
"timezone": "GMT+2",
Expand Down Expand Up @@ -129,7 +129,7 @@
},
"event": {
"category": "web",
"ingested": "2022-05-02T12:08:24.491819800Z",
"ingested": "2022-05-02T13:43:25.779999100Z",
"kind": "event",
"original": "[Thu Jun 27 06:58:09.169510 2019] [include:warn] [pid 15934] [client 89.160.20.156:12345] AH01374: mod_include: Options +Includes (or IncludesNoExec) wasn't set, INCLUDES filter removed: /test.html",
"timezone": "GMT+2",
Expand Down
4 changes: 2 additions & 2 deletions packages/apache/data_stream/error/_dev/test/pipeline/test-error-darwin.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
},
"event": {
"category": "web",
"ingested": "2022-05-02T12:08:24.559334800Z",
"ingested": "2022-05-02T13:43:25.841922500Z",
"kind": "event",
"original": "[Mon Dec 26 16:15:55.103522 2016] [mpm_prefork:notice] [pid 11379] AH00163: Apache/2.4.23 (Unix) configured -- resuming normal operations",
"timezone": "GMT+2",
Expand Down Expand Up @@ -41,7 +41,7 @@
},
"event": {
"category": "web",
"ingested": "2022-05-02T12:08:24.559363400Z",
"ingested": "2022-05-02T13:43:25.841952800Z",
"kind": "event",
"original": "[Mon Dec 26 16:15:55.103786 2016] [core:notice] [pid 11379] AH00094: Command line: '/usr/local/Cellar/httpd24/2.4.23_2/bin/httpd'",
"timezone": "GMT+2",
Expand Down
2 changes: 1 addition & 1 deletion packages/apache/data_stream/error/_dev/test/pipeline/test-error-trace.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
},
"event": {
"category": "web",
"ingested": "2022-05-02T12:08:24.613195200Z",
"ingested": "2022-05-02T13:43:25.892814400Z",
"kind": "event",
"original": "[Wed Oct 20 19:20:59.121211 2021] [rewrite:trace3] [pid 121591:tid 140413273032448] mod_rewrite.c(470): [client 10.121.192.8:38350] 10.121.192.8 - - [dev.elastic.co/sid#55a374e851c8][rid#7fb438083ac0/initial] applying pattern '^/import/?(.*)$' to uri '/'",
"timezone": "GMT+2",
Expand Down
Loading