chore: introduce API changes to ease support of zerolog. #3

jcchavezs · 2023-02-28T22:36:25Z

This PR aims to ease the introduction of zerolog in one hand to still use coraza types but let the API interact with zerolog api aswell.

Ping @anuraaga

* tests: adds more coverage for CTL action. * tests: improves naming.

…waf#675)

anuraaga

I think it will be good if the default still uses the standard library and we can provide a plugin to implement with zerolog, or zap or whatever. Does this PR achieve that?

actions/setvar.go

actions/ctl.go

ns-sundar · 2023-03-01T00:02:02Z

actions/expirevar.go

 	// Not supported
 	// TODO(jcchavezs): Shall we log a message?
-	// tx.WAF.Logger.Error("Expirevar was used but it's not supported", zap.Int("rule", r.Id))
+	// tx.WAF.Logger.Error().Msg("Expirevar was used but it's not supported", zap.Int("rule", r.Id))


What is the reason to retain a commented log?

Good question, let's remove it in another PR.

internal/seclang/directives.go

internal/corazawaf/logger.go

manojgop · 2023-03-01T04:23:28Z

@jcchavezs Thanks for raising this PR. Some checks are failing due to incorrect conversion between integer types

manojgop · 2023-03-01T04:30:19Z

internal/seclang/rule_parser.go

 				continue actionLoop
 			}
-			cval.WriteString(string(c))
+			cval.WriteRune(c)


Looks good. But this change is unrelated to zerolog and was initially added in this commit -> 187daf6

manojgop · 2023-03-01T05:21:43Z

I think it will be good if the default still uses the standard library and we can provide a plugin to implement with zerolog, or zap or whatever. Does this PR achieve that?

This is a good suggestion. However, zerolog uses a different API for log message formatting compared to log package in standard library. We need to avoid invoking fmt.Sprintf(message, args..) variadic function if Debug is disabled to avoid heap allocations of arguments by go compiler escape analysis. zerolog APIs are designed to avoid heap allocations and reflection. It could be tricky to use a generic API and to format the log message that could satisfy both zerolog and standard log library APIs and at the same time avoid heap allocations if Debug logs is disabled.

anuraaga · 2023-03-01T05:24:50Z

It could be tricky to use a generic API and to format the log message that could satisfy both zerolog and standard log library APIs and at the same time avoid heap allocations if Debug logs is disabled.

The zerolog version can be the optimized version, while it is good to work out of the box with the standard log library that can have extra heap allocations as a tradeoff.

Notably, being able to implement with zap does seem important - for example Caddy uses zap for logging, and while it's not the end of the world to use a different library with Coraza I think it'd generally be better to be able to plug zap logger in for that connector. So I'd like to make sure the logging implementation is decoupled from the logger plugin interface if at all possible.

manojgop · 2023-03-01T05:39:13Z

Just FYI - A related change in coraza-proxy-wasm to use zerolog is manojgop/coraza-proxy-wasm@e2cd02f. This would need modifications as well

jcchavezs · 2023-03-01T06:15:46Z

I think one idea would be to mimic the zerolog approach and have a similar API which can work with standard logger e.g. doing `message key1=value1 key2=value2` for std logger which is plain. While sticking with zerolog does not seem too crazy, I think @anuraaga made a point with caddy (using zap) and actually proxy-wasm has an own logger API which can fit in this API aswell.

…

On Wed, 1 Mar 2023, 06:39 manojgop, ***@***.***> wrote: Just FYI - A related change in coraza-proxy-wasm to use zerolog is ***@***.*** <manojgop/coraza-proxy-wasm@e2cd02f>. This would need modifications as well — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAVGGGEKN2NX3H3Y7FLWZ3OIZANCNFSM6AAAAAAVLJZQWA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

…awaf#681) Use string builder in parseActions() to optmize memory usage. Use string builder in parseActions() function to efficiently concatenate strings and optimize memory usage. go bench profile (testing/coreruleset) shows less allocations in seclang.parseActions function. Signed-off-by: Manoj Gopalakrishnan <[email protected]>

jcchavezs · 2023-03-01T10:49:30Z

loggers/debug.go

+func (e NopEvent) Stringer(key string, val fmt.Stringer) Event { return e }
+
 // DebugLogger is used to log SecDebugLog messages
 type DebugLogger interface {


How about this logger interface? @anuraaga @manojgop @ns-sundar

loggers/default.go

jcchavezs · 2023-03-01T11:06:36Z

I gave some turnarounds to the API. Please do review and let me know if this feels right so I can backfill tests.

manojgop · 2023-03-01T11:40:49Z

I gave some turnarounds to the API. Please do review and let me know if this feels right so I can backfill tests.

Could you please check that heap allocation is not there in Debug Log statements (if Debug logs are disabled) if we use zerolog instead of default log. Could you elaborate how can we replace default log package with zerolog (at compile time as a plugin) ?

manojgop · 2023-03-01T13:15:12Z

I gave some turnarounds to the API. Please do review and let me know if this feels right so I can backfill tests.

Could you please check that heap allocation is not there in Debug Log statements (if Debug logs are disabled) if we use zerolog instead of default log. Could you elaborate how can we replace default log package with zerolog (at compile time as a plugin) ?

Got it. I see that APIs are designed similar to zerolog APIs and with that Debug() log statements will avoid heap allocations if Debug log is disabled. Overall it looks good to me :-)

jcchavezs · 2023-03-01T21:20:54Z

I think the API is ready @anuraaga @ns-sundar @manojgop. If we merge this now then @manojgop can open a PR against coraza repo itself I guess.

doEvaluate() function heap allocation has reduced

Log statements are not allocating memory

Thanks @manojgop for the screenshots.

jcchavezs · 2023-03-01T21:24:27Z

debuglogger/logger.go

+// Logger is used to log SecDebugLog messages
+// This interface is highly inspired in github.com/rs/zerolog logger and the aim
+// is to avoid allocations while logging.
+type Logger interface {


Logger implementation would follow this interface, being zap or proxywasm cc @anuraaga

anuraaga

Seems reasonable to me

anuraaga · 2023-03-02T01:12:15Z

actions/ctl.go

 		limit, err := strconv.ParseInt(a.value, 10, 64)
 		if err != nil {
-			tx.WAF.Logger.Error("[ctl:RequestBodyLimit] Incorrect integer CTL value %q", a.value)
+			tx.WAF.Logger.Error().


This is missing Msg so won't be outputed - it seems that zerolog has Send() for msg-less logs.

Note that this is the trickiness of the zerolog API that achieves minimum allocations with the chance of not actually logging by mistake - it's definitely much harder to use than something like zap. But I guess we can deal with it. If there's not a linter for it yet I guess we could write it

Ah nice, there's an official one

https://github.com/rs/zerolog/blob/master/cmd/lint/README.md

Lint sounds good. BTW, one of the things I have been doing lately is to cover logs with unit tests but linter seems easy. Let me open an issue for that.

anuraaga · 2023-03-02T01:19:03Z

debuglogger/default.go

+		return
+	}
+
+	s := strings.Builder{}


Think e.logger.Printf would be better than using this builder to avoid the intermediate string.

e.logger.Printf("[%s] %s%s\n", e.level.String(), msg, e.fields)

anuraaga · 2023-03-02T01:20:05Z

debuglogger/default.go

+}
+
+func (l defaultLogger) Trace() Event {
+	if l.level < LogLevelTrace || l.Logger == nil {


Don't think Logger can be nil, especially after making it a private field as it probably should be

It was in NOP case but I just changed it based on this convenience https://cs.opensource.google/go/go/+/refs/tags/go1.20.1:src/log/log.go;l=72

ns-sundar · 2023-03-02T02:51:20Z

It could be tricky to use a generic API and to format the log message that could satisfy both zerolog and standard log library APIs and at the same time avoid heap allocations if Debug logs is disabled.

Do we necessarily need two logging frameworks? There may be ways to switch to Zerolog that may work with the current log statements, so that we don't need a forklift upgrade.

The zerolog version can be the optimized version, while it is good to work out of the box with the standard log library that can have extra heap allocations as a tradeoff.

The standard log library does not provide structured logging, unless you have something else in mind. So, having both will result in inconsistent logging.

As a data point, Crowdstrike switched to zerolog in full and saw good benefits.

Notably, being able to implement with zap does seem important - for example Caddy uses zap for logging, and while it's not the end of the world to use a different library with Coraza I think it'd generally be better to be able to plug zap logger in for that connector. So I'd like to make sure the logging implementation is decoupled from the logger plugin interface if at all possible.

This is a valid point. But, first, zerolog is widely used. Secondly, zap itself says that zerolog is faster. That said, if you are advocating some kind of neutral interface that will work with multiple loggers, for future-proofing, I am ok -- just wondering what effort that will entail.

anuraaga · 2023-03-02T03:06:42Z

just wondering what effort that will entail.

I think @jcchavezs has done most of the effort for it here, we already had a neutral interface for loggers and just have tweaked it now to enable a zero allocation backend.

As a core library rather than an app, we try to make it generic so that it is those integrating with it that can make their decisions. If apps can pick zerolog, zap, or even std library based on their existing dependencies or lack of dependencies, it makes Coraza a more useful tool in the toolbox than if we force a decision on it.

internal/seclang/directives.go

 		return err
 	}
-	return options.WAF.SetDebugLogLevel(lvl)
+	return options.WAF.SetDebugLogLevel(debuglogger.LogLevel(lvl))


zerolog is optimized for logging and does not allocate memory unlike fmr.Sprintf() for debug logs Signed-off-by: Manoj Gopalakrishnan <[email protected]>

anuraaga and others added 3 commits February 24, 2023 20:32

Remove MatchData.IsNil (corazawaf#677)

249700b

tests: adds more coverage for CTL action. (corazawaf#679)

f050500

* tests: adds more coverage for CTL action. * tests: improves naming.

Avoid intermediate string when formatting variables for debug (coraza…

c68f247

…waf#675)

anuraaga reviewed Mar 1, 2023

View reviewed changes

actions/setvar.go Show resolved Hide resolved

ns-sundar reviewed Mar 1, 2023

View reviewed changes

github-advanced-security bot found potential problems Mar 1, 2023

View reviewed changes

internal/seclang/directives.go Fixed Show fixed Hide fixed

manojgop reviewed Mar 1, 2023

View reviewed changes

internal/corazawaf/logger.go Outdated Show resolved Hide resolved

manojgop reviewed Mar 1, 2023

View reviewed changes

jcchavezs mentioned this pull request Mar 1, 2023

feat: generate different loggers per level to avoid allocations. corazawaf/coraza#673

Closed

4 tasks

jcchavezs commented Mar 1, 2023

View reviewed changes

loggers/default.go Outdated Show resolved Hide resolved

jcchavezs commented Mar 1, 2023

View reviewed changes

anuraaga reviewed Mar 2, 2023

View reviewed changes

github-advanced-security bot found potential problems Mar 2, 2023

View reviewed changes

jcchavezs mentioned this pull request Mar 2, 2023

Rework Logger API corazawaf/coraza#682

Merged

manojgop and others added 3 commits March 2, 2023 10:23

Use zerolog for logging.

7c64e4e

zerolog is optimized for logging and does not allocate memory unlike fmr.Sprintf() for debug logs Signed-off-by: Manoj Gopalakrishnan <[email protected]>

chore: introduce API changes to ease support of zerolog.

d1d5881

chore: uses interface of logging.

01a776c

jcchavezs added 7 commits March 2, 2023 10:24

chore: uniformize logger messages.

1bdd3c8

feat: polish debug logger API.

fe00e8a

chore: small nit in nop event.

23b9b20

chore: uniformize log fields.

817f028

docs: explains the timestamp skip.

5f09a60

chore: improves nop loggic and also print allocations.

94a96a6

chore: fixes missing log messages.

0f8ae9a

jcchavezs force-pushed the use_zerolog_frictionless branch from 1a585bd to 0f8ae9a Compare March 2, 2023 09:26

jcchavezs and others added 16 commits March 2, 2023 10:54

chore: generic improvements.

904955b

tests: improves coverage.

0ccdbe2

chore: readds log.

de4bf72

tests: improves coverage.

3b21685

feat: adds logger per transaction to allow 'ctlDebugLogLevel'.

1be5728

chore: nit printf.

1426314

tests: covers 'ctlDebugLogLevel'.

6be37d3

Enable xml processor on TinyGo (corazawaf#685)

801a1d6

feat: adds Logger.With context fields as per @ns-sundar suggestion.

04af417

chore: removes Event.Errs method to narrow down the API surface.

9fbb612

Merge branch 'v3/dev' into use_zerolog_frictionless

e4d12c2

docs: adds more comments on the IsEnabled method.

0162be8

chore: uses contextful logs for rule.

4aab51b

feat: adds PrinterFactory abstraction to ease the plug of new logs.

3ad819e

chore: refine printer.

62a9224

chore: rollsback creating a new log per rule evaluation.

2fb63aa

chore: introduce API changes to ease support of zerolog. #3

Are you sure you want to change the base?

chore: introduce API changes to ease support of zerolog. #3

Uh oh!

Conversation

jcchavezs commented Feb 28, 2023

Uh oh!

anuraaga left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

manojgop commented Mar 1, 2023

Uh oh!

manojgop Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manojgop commented Mar 1, 2023

Uh oh!

anuraaga commented Mar 1, 2023

Uh oh!

manojgop commented Mar 1, 2023

Uh oh!

jcchavezs commented Mar 1, 2023 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jcchavezs commented Mar 1, 2023

Uh oh!

manojgop commented Mar 1, 2023

Uh oh!

manojgop commented Mar 1, 2023

Uh oh!

jcchavezs commented Mar 1, 2023

Uh oh!

jcchavezs Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuraaga left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ns-sundar commented Mar 2, 2023

Uh oh!

anuraaga commented Mar 2, 2023

Uh oh!

Check failure

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

manojgop Mar 1, 2023 •

edited

Loading

jcchavezs Mar 1, 2023 •

edited

Loading