structured config #1: introduce mapstructure decoder for yaml parsing #2628

tuunit · 2024-05-04T14:42:35Z

Description

This PR introduces mapstructure for decoding and encoding the yaml files. Mapstructure is a defacto standard library used by many libraries like spf13/viper for more dynamic data loading between different interfaces.

Just using the golang json / yaml encoding / decoding would lead to overwriting the default set before loading the config file. Trying to load the default configs afterwards would be rather hard or impossible.

Motivation and Context

Cases to consider:

When directly loading into a struct you cannot identify if the user explicitly set a boolean to false in the config file or if it was set to the boolean default of false. This is problematic as we have boolean like Cookie.Secure that are supposed to be set to true by default.
For better readability we might want to squash elements in the yaml like HeaderValues either being a SecretSource or ClaimSource. Which isn't supported by the default json / yaml parser

# non-squashed
headers:
- name: X-Forwarded-User
  values:
  - claimSource:
      claim: user

# squashed
headers:
- name: X-Forwarded-User
  values:
  - claim: user

Parsing of time primitives like Duration or Time through strings. If you want to be able to configure Duration types with values like 2h. You need to implement a custom wrapper type and Marshal and Unmarshal method. Mapstructure allows for decoding hooks through the usage of relection.

How Has This Been Tested?

Test cases have been extend / adapted.
Starting the application with toml config file
Using the conversion function from toml to yaml
Starting the application with yaml config file

Checklist:

My change requires a change to the documentation or CHANGELOG.
I have updated the documentation/CHANGELOG accordingly.
I have created a feature (non-master) branch for my PR.
I have written tests for my code changes.

pkg/apis/options/header.go

github-actions · 2024-07-12T00:07:14Z

This pull request has been inactive for 60 days. If the pull request is still relevant please comment to re-activate the pull request. If no action is taken within 7 days, the pull request will be marked closed.

main.go

pkg/apis/options/duration.go

JoelSpeed · 2024-07-14T19:32:37Z

pkg/apis/options/header.go


 	// Allow users to load the value from a session claim
-	*ClaimSource `json:",omitempty"`
+	*ClaimSource `json:"claimSource,omitempty"`


We probably want to call this something different, such as, fromValue with the previous field being fromSecret

Adding json tags to these is odd though since they are inlined, do you recall why this was needed?

I know on another PR we are talking about moving the prefix, is this the PR where in fact we should make this a discriminated union?

They are inlined but because they are pointers mapstructures cannot resolve them properly. I wasn't able to get it to work otherwise, leading to this structure change:

You have to explicitly mention if you want to use a claimSource or secretSource for each HeaderValue you set.

pkg/apis/options/load.go

JoelSpeed · 2024-07-14T19:37:09Z

pkg/apis/options/providers.go

 	IssuerURL string `json:"issuerURL,omitempty"`
 	// InsecureAllowUnverifiedEmail prevents failures if an email address in an id_token is not verified
 	// default set to 'false'
-	InsecureAllowUnverifiedEmail bool `json:"insecureAllowUnverifiedEmail,omitempty"`


Why no longer omitempty?

I decided to not omit any booleans. To ensure that the conversion flow from toml to yaml explicitly mentions all boolean values.

Booleans really want to be pointers with omitempty IMO, else we don't have a way to tell between an explicitly set false value or implicitly set false just by the fact that false is the zero value

Would making them all pointers be complex for the conversion flow do you think?

pkg/apis/options/secret_source.go

github-actions · 2024-10-20T00:09:08Z

This pull request has been inactive for 60 days. If the pull request is still relevant please comment to re-activate the pull request. If no action is taken within 7 days, the pull request will be marked closed.

dolmen · 2025-05-26T13:21:27Z

I've developped independently #3064 which just handles the upgrade of viper+mapstructure. I think it would be worth doing that to keep this (2628) focused on behaviour changes.

tuunit · 2025-08-12T06:11:55Z

/e2e

Copilot

Pull Request Overview

This PR introduces mapstructure as the decoder for YAML configuration parsing, replacing the previous ghodss/yaml library. The change enables better handling of default values, type conversion, and field mapping while preserving existing configuration behavior.

Key changes:

Replaces options.Duration wrapper type with native time.Duration throughout the codebase
Implements mapstructure-based YAML parsing with decode hooks for duration and byte conversion
Updates struct tags from json to yaml across all configuration structs

Reviewed Changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/apis/options/load.go	Implements new LoadYAML function using mapstructure decoder with custom hooks
pkg/apis/options/hooks.go	Adds decode hooks for duration parsing and string-to-bytes conversion
pkg/apis/options/upstreams.go	Updates FlushInterval and Timeout from Duration wrapper to time.Duration
pkg/apis/options/*.go	Changes struct tags from json to yaml across configuration structs
pkg/validation/upstreams.go	Updates duration comparison logic for new time.Duration type
pkg/upstream/http.go	Removes Duration wrapper usage in favor of direct time.Duration
main.go	Updates configuration loading flow and YAML marshaling logic

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.}

pkg/apis/options/providers.go

JoelSpeed · 2025-08-19T10:29:50Z

pkg/apis/options/secret_source.go

+	// FromEnv expects the name of an environment variable.
+	FromEnv string `yaml:"fromEnv,omitempty"`


Not blocking for this PR, but I thought now we have env substitution, we won't need this right?

I would like to do a cleanup and review of our options API after we have the full alpha config support done

JoelSpeed · 2025-08-19T10:30:55Z

pkg/apis/options/providers.go

 	IssuerURL string `json:"issuerURL,omitempty"`
 	// InsecureAllowUnverifiedEmail prevents failures if an email address in an id_token is not verified
 	// default set to 'false'
-	InsecureAllowUnverifiedEmail bool `json:"insecureAllowUnverifiedEmail,omitempty"`


Booleans really want to be pointers with omitempty IMO, else we don't have a way to tell between an explicitly set false value or implicitly set false just by the fact that false is the zero value

Would making them all pointers be complex for the conversion flow do you think?

main.go

JoelSpeed · 2025-08-19T10:44:38Z

pkg/apis/options/load.go

+		),
+		Metadata:             nil,    // Don't track any metadata
+		Result:               result, // Decode the result into the prefilled options
+		TagName:              "yaml", // Parse all fields that use the json tag


Is there a reason to switch to yaml instead of json? Is some other library relying on yaml?

Yes the yaml parser expects it:
https://pkg.go.dev/gopkg.in/yaml.v3

Ack, can you just explain in the comment here why we use the yaml tag

Some yaml libraries like the json tag instead

JoelSpeed · 2025-08-19T15:08:11Z

pkg/util/ptr/ptr.go

@@ -0,0 +1,5 @@
+package ptr


Could create generics here?

Also, ptr.Deref which allows you to provide a default value if the pointer is nil?

E.g. https://pkg.go.dev/k8s.io/[email protected]/ptr#To

JoelSpeed · 2025-08-21T13:39:23Z

pkg/util/ptr/ptr.go

+func Int(v int) *int          { return &v }
+
+// Ptr generically returns a pointer to the given value.
+func Ptr[T any](v T) *T {


Maybe To to make it read as ptr.To "pointer to"?

JoelSpeed · 2025-08-21T13:39:34Z

pkg/util/ptr/ptr.go

+func Bool(v bool) *bool       { return &v }
+func String(v string) *string { return &v }
+func Int(v int) *int          { return &v }


We shouldn't need these anymore right?

JoelSpeed · 2025-08-21T13:40:04Z

pkg/util/ptr/ptr.go

+}
+
+// Deref returns the value of the pointer or def(ault) if nil.
+func Deref[T any](p *T, def T) T {


I saw lots of places where this isn't currently used where it could be, lets try and be exhaustive within the current PR and hopefully avoid panics

main_test.go

JoelSpeed · 2025-08-21T13:41:40Z

pkg/middleware/headers.go

 	headersToStrip := []string{}
 	for _, header := range headers {
-		if !header.PreserveRequestValue {
+		if !(*header.PreserveRequestValue) {


JoelSpeed · 2025-08-21T13:41:53Z

pkg/upstream/http.go

 	// Change default duration for waiting for an upstream response
 	if upstream.Timeout != nil {
-		transport.ResponseHeaderTimeout = upstream.Timeout.Duration()
+		transport.ResponseHeaderTimeout = *upstream.Timeout


JoelSpeed · 2025-08-21T13:42:09Z

pkg/upstream/proxy.go

 	}

-	if upstreams.ProxyRawPath {
+	if *upstreams.ProxyRawPath {


remove color output in tests for better readability in github actions bugfix: remove google as default provider for alpha options fix conversion flow for toml to yaml revert ginkgo color deactivation revert claim- and secret source back to pointers regenerate alpha config

Signed-off-by: Jan Larwig <[email protected]>

github-actions bot added go tests labels May 4, 2024

tuunit mentioned this pull request May 4, 2024

[WIP] feat: full support of all options in alpha configuration #2279

Closed

9 tasks

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch from 710f3a7 to 72dff61 Compare May 4, 2024 14:44

github-actions bot added the docs label May 4, 2024

tuunit mentioned this pull request May 4, 2024

structured config #2: legacy file structure refactoring and cookie support #2629

Open

4 tasks

tuunit changed the title ~~introduce mapstructure decoder for yaml parsing~~ structured logging #1: introduce mapstructure decoder for yaml parsing May 4, 2024

tuunit mentioned this pull request May 4, 2024

structured config #3: support for proxy options #2630

Open

4 tasks

tuunit changed the title ~~structured logging #1: introduce mapstructure decoder for yaml parsing~~ structured config #1: introduce mapstructure decoder for yaml parsing May 4, 2024

This was referenced May 5, 2024

structured config #4: support for session and probes #2632

Open

structured config #5: support logging #2633

Open

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch from 39b625e to 3fb3c1f Compare May 9, 2024 22:00

tuunit marked this pull request as ready for review May 10, 2024 07:34

tuunit requested a review from a team as a code owner May 10, 2024 07:34

tuunit commented May 10, 2024

View reviewed changes

pkg/apis/options/header.go Outdated Show resolved Hide resolved

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch from 7f8def6 to cd566ee Compare May 12, 2024 17:40

github-actions bot added the Stale label Jul 12, 2024

JoelSpeed reviewed Jul 14, 2024

View reviewed changes

github-actions bot removed the Stale label Jul 15, 2024

tuunit changed the base branch from master to release/v8.0.0 August 20, 2024 08:44

tuunit mentioned this pull request Oct 1, 2024

updated removed options docs to include http_server #2776

Closed

4 tasks

github-actions bot added the Stale label Oct 20, 2024

github-actions bot closed this Nov 4, 2024

tuunit reopened this Jan 4, 2025

tuunit added this to the v8.0.0 milestone Jan 4, 2025

github-actions bot removed the Stale label Jan 5, 2025

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch from 4941bf1 to 629ac24 Compare February 1, 2025 09:20

github-actions bot added the dependencies Pull requests that update a dependency file label Feb 1, 2025

dolmen mentioned this pull request May 26, 2025

chore(deps): upgrade github.com/spf13/viper to v1.20.1 #3064

Closed

4 tasks

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch 3 times, most recently from 4158184 to f46e023 Compare July 26, 2025 15:03

github-actions bot added the e2e/failed label Aug 12, 2025

tuunit requested a review from Copilot August 13, 2025 06:57

Copilot AI reviewed Aug 13, 2025

View reviewed changes

pkg/apis/options/providers.go Outdated Show resolved Hide resolved

pkg/apis/options/providers.go Outdated Show resolved Hide resolved

pkg/apis/options/providers.go Outdated Show resolved Hide resolved

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch 2 times, most recently from a471d71 to bc1d897 Compare August 18, 2025 19:37

JoelSpeed reviewed Aug 19, 2025

View reviewed changes

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch from bc1d897 to d2f72ef Compare August 19, 2025 10:59

github-actions bot added the provider label Aug 19, 2025

JoelSpeed reviewed Aug 19, 2025

View reviewed changes

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch from f84ed08 to 179e390 Compare August 20, 2025 18:00

JoelSpeed reviewed Aug 21, 2025

View reviewed changes

tuunit added 12 commits August 27, 2025 09:15

apply review suggestions

202de3a

add duration test

11f1b9e

use official upstream yaml library v3

37019fc

fix alpha config example

8b3451a

resolve cipher deprecation and update mapstructures v2

514296a

Signed-off-by: Jan Larwig <[email protected]>

fix alpha config

a646d9d

Signed-off-by: Jan Larwig <[email protected]>

revert: secrets as []byte instead of string

1d73f14

Signed-off-by: Jan Larwig <[email protected]>

fix merge problems and test cases

fa2587a

Signed-off-by: Jan Larwig <[email protected]>

fix test setup and add local image build make target

7e070d1

Signed-off-by: Jan Larwig <[email protected]>

return nil directly

3a52960

Signed-off-by: Jan Larwig <[email protected]>

feat: migrate all alpha config booleans to pointers

9db7738

Signed-off-by: Jan Larwig <[email protected]>

tuunit force-pushed the use-mapstructures-for-parsing-and-merging branch from 179e390 to 9db7738 Compare August 27, 2025 07:16

		// FromEnv expects the name of an environment variable.
		FromEnv string `yaml:"fromEnv,omitempty"`

Uh oh!

structured config #1: introduce mapstructure decoder for yaml parsing #2628

Are you sure you want to change the base?

structured config #1: introduce mapstructure decoder for yaml parsing #2628

Uh oh!

Conversation

tuunit commented May 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Checklist:

Uh oh!

Uh oh!

github-actions bot commented Jul 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 20, 2024

Uh oh!

dolmen commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tuunit commented Aug 12, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuunit Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

tuunit commented May 4, 2024 •

edited

Loading

dolmen commented May 26, 2025 •

edited

Loading

tuunit Aug 19, 2025 •

edited

Loading