Note: this proposal has been updated (as v0.2) after the review and discusion on the first version(as v0.1). And also updated (as v0.3) after long time review and discusion on version v0.2.

Please go to:

v0.3: #2988 (comment)
v0.2: #2988 (comment)

==========

This is the first version (as v0.1) of this proposal.

In what area(s)?

/area runtime

Proposal Background

Dapr plans to add new "configuration" building block to support dynamic configuration, and I was given the task of designing the configuration api.

Here is an issue to collect and discuss the features that this configuration building block should have.

#2941

During the process of designing the configuration api, we find that:

1. We want to define a API to satisfy all the requirements of dynamic configuration from applications in various scenarios

2. But in reality, there are various configuration products with different feature, and there is no common configuration model and feature list, which leads us to a dilemma when we define the configuration api:

   • If we define the configuration api according to the maximum set of features, we find that the components implementing the configuration api will not be able to provide all the features defined in this configuration api. What's worse is that some components will have large gaps, resulting in a lack of basic usability and portability is not guaranteed.

   • If we define the configuration api according to the minimum set of features, we find that although all components can provide the features defined by the configuration api, but there are few features defined in the configuration api : users have to use metadata and so on to do customized extension, which seriously affects portability.

We have to face this reality:

There is no configuration api that meets all requirements and is fully supported by all components at the same time!

This happens not only in this new configuration api, but also in the exist api. For example, the state api defines the ExecuteStateTransaction() method, which allows multiple state write operations to be executed as one transaction. However, this method is not supported in many state stores, and in fact the user must be careful to select the state component must when this feature is needed. Also, the consistency / concurrency parameters defined by the state api may not be fully supported in every component.

The configuration api is more complexity than the state api: more features, more complex models, and especially public cloud vendors often offer more advanced features than open source production.

To solve this problem, we consider introducing "hierarchical capability model" in the design of the configuration api and evaluating the "maturity of configuration components" according to this hierarchical capability model.

Hierarchical Capability Model

Considering the usage of configuration capabilities:

• The basic and common features are frequently used in most user cases
• Advanced features are used in a few user cases
• Some unique features should be used in fewer scenarios

Therefore, we can consider layering configuration-related features according to user cases. From bottom to top:

• The most common and basic features are at the bottom. Level0 represents the necessary functions that must be supported.
• Adding new level to provide more advanced features one by one. This gradually improves the configuration model and api details: more fields, more parameters and more methods.

In current configuration api design, we have defined three levels covering the most common used functions and user cases.

Level-0

Level-0 covers the basic function to meet the most basic user cases:

The configuration model of level-0 is very simple with only two concepts: configuration items and the application to which they belong.

Configuration Item

Configuration item is the basic unit of configuration.

In principle, a configuration item is a key-value pair with some attributes:

• key: the key of the configuration item, the type is fixed to string
• content: the value of the configuration item, the type is fixed to string
• metadata: optional, the type is map<string, string>, for custom extensions, like in other dapr api

the Application to which configuration items belong

Configuration items should belong to applications, which means that:

• Any configuration item must belong to an application
• One configuration item can only belong to one application
• Configuration items are unique within an application: e.g., under level-0, the key must be unique
• Different applications are allowed to have the same configuration items. In other words, configuration items are isolated by applications.

Special note on the AppID field of configuration item: in principle, each configuration item should have an appid field to mark which app the configuration item belongs to, but for security reasons, the app id is obtained in a different way than other parameters:

• dapr will get app id of the application itself

• For security reasons, applications are not allowed to access the configuration of other applications. So even the client passes the app id, dapr can not trust it and need to check the validity of the app id

• For the administrative api (save/delete), the app id passed by the api will be trusted only after the administrative privileges are verified, and at this time the app id will be a parameter of the request level and will not be set in each configuration item.

User Cases of Level-0

Here is the list of user cases for Level-0 of configuration api:

• Add/modify/delete a single configuration item of the specified app (single)
• Add/modify/delete multiple configuration items of the specified app (bulk)
• Query a single configuration item of the specified app (single)
• Query multiple configuration items of the specified app (bulk)
• Query all configuration items of the specified app (full)
• Subscribe to update operations for a single configuration item of the specified app (single)
• Subscribe to update operations for multiple configuration items of the specified app (bulk)
• Subscribe to update operations for all configuration items of the specified app (full)

Level-1

Level-1 adds the concept of "configuration group" to Level-0:

The background to introduce "configuration group":

• There will be a very large number of configuration items for an application, the number may be tens or even hundreds, and we need a convenient way to organize configuration items.
• Some configuration items usually have a clear context of association, such as mysql configuration items, rocketmq configuration items, configuration items for some business logic. These configuration items are often used together, for example mysql related configuration items: server address, port, username, password, they are used together as a collection whether they are created, modified, deleted, read, or listened for changes.

Characteristics of configuration group:

• Configuration group is only used to organize and manage a number of close configuration items in the same application
• Configuration group does not affect the relationship between configuration items and applications, nor does it affect the uniqueness of configuration items
• An application can have more than one configuration group, a configuration group can have more than one configuration item
• Each configuration item should belong to a configuration group (default group if not), and can only belong to a configuration group (if you need more flexible organization, please use the tag in level-2)
• If the configuration item's group is not explicitly specified, it will be assigned to the default group (e.g., group name "default")

Benefits of introducing configuration group:

• All keys under a group can be read and listened at once
• The difference compared to batch operation with multiple keys is that configuration group can be used without specifying each key individually: the mapping between configuration group and keys is implemented by the component.

Here is the list of user cases for Level-1 of configuration api (only list new added user case to level-0):

• Query all configuration items of specified app and group (group)
• Subscribe to update operations for all configuration items of specified app and group (group)

Level-2

Level-2 adds the concepts of "label" and "tag" to Level-1:

Label

The background to introduce label is to achieve environment isolation:

• The application needs to set different configuration values for the same configuration item (key) in different environments, such as "DEV" / "TESTING" / "PRODUCTION" / "STAGING" etc.

Characteristics of label:

• Label is used to do environmental isolation: the same key with different label allows different values to be set
• Label will affect the uniqueness of the configuration item in the application, after introducing label, the unique identifier of the configuration item in the given application is changed from the original "key" to "key + label"
• Label is allowed to be empty, i.e. no Label is set
• The type of label is string: for the time being, only one label is supported (because the complexity of supporting multiple labels will increase significantly).

Tag

The background to introduce tag is to allow for more flexible organization of configuration items:

• Supports advanced features, such as filtering by tag
• Makes it easy to manage configuration items in different dimensions: more flexible than group. Configuration item can only belong to one group, but it can have multiple tags.
• Multiple tags can be used at the same time

Relationship between Tag and other features:

• group support: for components that natively support tag, group can be implemented via tag, i.e. using a tag named "group".
• encryption support: tag can be used to mark a configuration item as requiring encryption

Characteristics of of Tag:

• Tag is used to help organize configuration items in a more flexible way. Unlike label that does environmental isolation, tags do not affect the uniqueness of configuration items: the uniqueness of a configuration item is still key + label
• Tag is allowed to be empty, i.e., no tag is set
• The type of tag is map<string, string>, which supports multiple tags for a single configuration item

Here is the list of user cases for level-2 of configuration api after introducing label and filter:

• Add/modify/delete a single configuration item of the specified app (single): support group/label/tags
• Add/modify/delete multiple configuration items of the specified app (bulk): as above
• Query a single configuration item of the specified app (single): support seperated by label and filtered by tags
• Query multiple configuration items of the specified app (bulk): as above
• Query multiple configuration items of the specified app and group (group): as above
• Query all configuration items of the specified app (full): as above
• Subscribe to update operations for a single configuration item of the specified app (single): support seperated by label and filtered by tags
• Subscribe to update operations for multiple configuration items of the specified app (bulk): as above
• Subscribe to update operations for all configuration items of the specified app and group (group): as above
• Subscribe to update operations for all configuration items of the specified app (full): as above

Higher than Level-2

On top of level-2, we can support other user cases of configuration, such as

• level-3 - Configuration encryption: channel encryption + automatic encryption and decryption of specific configuration item content
• Level-4 - Configuration cache: in case sidecar or underling componnent is unavailable
• level-5 - Configuration item history: save the history of the configuration item, and can query the history of a specific version

These features are not planned in current proposal, and can be added later if needed.

Maturity Evaluation of Configuration Components

For above level0 - level2 of configuration hierarchy capability model, we investigated the mainstream configuration implementation products to evaluate the level to support above configuration hierarchy capability model, which we call the "maturity evaluation of the configuration components" .

TBD: we need to complete this table. Update later.

Configuration API Definition (Draft)

we define the configuration api based on level-2.

Configuration Item Definition

Proto definition for Configuration item: (in dapr/proto/common/v1/comon.proto):

// ConfigurationItem represents a configuration item with key, content and other information.
message ConfigurationItem {
  // Required. The key of configuration item
  string key = 1;

  // The content of configuration item
  // Empty if the configuration is not set, including the case that the configuration is changed from value-set to value-not-set.
  string content = 2;

  // The group of configuration item.
  string group = 3;

  // The label of configuration item.
  string label = 4;

  // The tag list of configuration item.
  map<string,string> tags = 5;

  // The metadata which will be passed to configuration store component.
  map<string,string> metadata = 6;
}

Read Configuration and Subscribe to Updates

There are two ways to read the configuration and subscribe to updates:

• app-callback: The app first reads the configuration, and when the configuration is updated, dapr notifies the app of the updated configuration items by app callback.
• grpc-stream: implemented via gRPC's stream

TBD: We need to make a decision here to choise one of above two.

Two proposal api designs are given below to help to make this decision.

proposal-1: app-callback

GetConfiguration() method with OnConfigurationEvent() method: GetConfiguration() returns the current value of configuration items directly, and then notifies updates via the OnConfigurationEvent() method of app callabck.

The proto definition of dapr GetConfiguration() method which reads the configuration (in dapr/proto/runtime/v1/dapr.proto):

service Dapr {
  // GetConfiguration gets configuration from configuration store.
  rpc GetConfiguration(GetConfigurationRequest) returns (GetConfigurationResponse) {}
}

// GetConfigurationRequest is the message to get a list of key-value configuration from specified configuration store.
message GetConfigurationRequest {
  // The name of configuration store.
  string store_name = 1;

  // The application id.
  // Only used for admin client, Ignored and reset by daprd for normal client
  string app_id = 2;

  // The group of keys.
  string group = 3;

  // The label for keys.
  string label = 4;

  // The keys to get.
  repeated string keys = 5;

  // The metadata which will be sent to configuration store components.
  map<string,string> metadata = 6;

  // Subscribes update event for given keys.
  // If true, when any configuration item in this request is updated, app will receive event by OnConfigurationEvent() of app callback
  bool subscribe_update = 7;
}

// GetConfigurationResponse is the response conveying the list of configuration values.
message GetConfigurationResponse {
  // The list of items containing configuration values.
  repeated ConfigurationItem items = 1;
}

The proto definition of OnConfigurationEvent() app callback method which notifies the configuration update events (in dapr/proto/runtime/v1/appcallback.proto):

service AppCallback {
  // Listens configuration update events from the configuration store.
  rpc OnConfigurationEvent(ConfigurationEventRequest) returns (google.protobuf.Empty) {}
}

// ConfigurationEventRequest represents update events of configuration items.
message ConfigurationEventRequest {
  // Required. The name of the configuration store component.
  string store_name = 1;

  // The application id.
  // Only used for admin client.
  string app_id = 2;

  // Required. The updated configuration items.
  repeated common.v1.ConfigurationItem items = 3;
}

The advantage of this proposal is that it is easy to implement regardless of HTTP1.1 or gRPC.

proposal-2: grpc-stream

The proto definition of SubscribeConfiguration() dapr method to read and monitor the configuration (in dapr/proto/runtime/v1/dapr.proto):

service Dapr {
  // SubscribeConfiguration gets configuration from configuration store and subscribe the updates.
  rpc SubscribeConfiguration(stream SubscribeConfigurationRequest) returns (stream SubscribeConfigurationResponse) {}
}

// SubscribeConfigurationRequest is the message to get a list of key-value configuration from specified configuration store.
message SubscribeConfigurationRequest {
  // The name of configuration store.
  string store_name = 1;

  // The application id which
  // Only used for admin, Ignored and reset for normal client
  string app_id = 2;

  // The group of keys.
  string group = 3;

  // The label for keys.
  string label = 4;

  // The keys to get.
  repeated string keys = 5;

  // The metadata which will be sent to configuration store components.
  map<string,string> metadata = 6;
}

// SubscribeConfigurationResponse is the response conveying the list of configuration values.
message SubscribeConfigurationResponse {
  // The name of configuration store.
  string store_name = 1;

  // The application id.
  // Only used for admin client.
  string app_id = 2;

  // The list of items containing configuration values.
  repeated ConfigurationItem items = 3;
}

Save Configuration

The proto definition of SaveConfiguration() method to save (including add and modify) the configuration (in dapr/proto/runtime/v1/dapr.proto):

service Dapr {
  // SaveConfiguration saves configuration into configuration store.
  rpc SaveConfiguration(SaveConfigurationRequest) returns (google.protobuf.Empty) {}
}

// SaveConfigurationRequest is the message to save a list of key-value configuration into specified configuration store.
message SaveConfigurationRequest {
  // The name of configuration store.
  string store_name = 1;

  // The application id which
  // Only used for admin, ignored and reset for normal client
  string app_id = 2;

  // The list of configuration items to save.
  // To delete a exist item, set the key (also label) and let content to be empty
  repeated ConfigurationItem items = 3;

  // The metadata which will be sent to configuration store components.
  map<string,string> metadata = 4;
}

When saving a configuration item, if you need to delete an existing configuration item (change to the default value), you can set the content of this configuration item to empty.

Considering that configuration update is an administrative operation with low frequency, we do not introducing concurrency control and consistency control (as in dapr state api).

Delete Configuration

The proto definition of DeleteConfiguration() method to delete the configuration (in dapr/proto/runtime/v1/dapr.proto):

service Dapr {
  // DeleteConfiguration deletes configuration from configuration store.
  rpc DeleteConfiguration(DeleteConfigurationRequest) returns (google.protobuf.Empty) {}
}

// DeleteConfigurationRequest is the message to delete a list of key-value configuration from specified configuration store.
message DeleteConfigurationRequest {
  // The name of configuration store.
  string store_name = 1;

  // The application id which
  // Only used for admin, Ignored and reset for normal client
  string app_id = 2;

  // The group of keys.
  string group = 3;

  // The label for keys.
  string label = 4;

  // The keys to get.
  repeated string keys = 5;

  // The metadata which will be sent to configuration store components.
  map<string,string> metadata = 6;
}

Also considering that configuration delete is an administrative operation with low frequency, we do not introducing concurrency control and consistency control (as in dapr state api).

Design outside of the Configuration API

The following will not be reflected in the api (http and grpc) provided by dapr, but is still a part of the design of the configuration building block.

Default Valule of Configuration Item

In theory, each configuration item should be allowed not to be configured, so we need the way to save and provide the default values.

How to Represent the Default Value

The underlying storage of default value is determined by the component implementation: it is recommended (but not mandatory) to store without a key, it is allowed if the content is saved as null (does not distinguish between null and "" empty strings, both are considered null).

When dapr calls component to do a query, if the configuration item does not have a value, the component is allowed to return with either the key not present or a null for the corresponding value. And dapr should unify to "the key does not exist" before returning.

Monitoring is different: suppose a key has a value before, and later it is modified by removing the value and using the default value instead. In this case, the notification semantics should be explicitly "the value of the key is changed to be not set, please use the default value", and we need to provide a configuration item of "content is empty" for this key in the notification message.

How to Get the Default Value

How do we get the default value of a configuration item when it is not set?

The recommended approach is to solve this at the SDK level: the user needs to provide the default value when calling to get the content of a specific configuration item.

The method of SDK should be similar to this:

value1 := client.getConfiguration("key-1", "defaultValue")

Configuration Aggregation

Darp's configuration building block should support multiple configuration sources and can aggregate configurations from those sources.

Multiple Configuration Sources

Configuration source is the component that can implement the configuration api:

• Local configuration files
• Various configuration management systems: diamond, nacos, apollo, etc.
• Command line parameters: command line parameters of the application can only be implemented in SDK; command line parameters of daprd can be supported by dapr component
• Environment variables: application environment variables can only be implemented in SDK; daprd environment variables can be supported by dapr component

Each configuration source can be supported by a configuration component.

Aggregate Configurations from Multiple Configuration Sources

A configuration aggregation component should be built into daprd:

• Support for aggregation of configuration items from multiple configuration sources
• Support priority : configuration items from high-level configuration sources can override configuration items from low-level configuration sources
• Optional: Aggregation is optional. User can directly use components for a single configuration source without aggregation.

The timing for configuration aggregation to take effect:

• Be transparent to the application
  • Multiple configuration sources should also be transparent to the application
  • The user does not need to know the source of the configuration and which source is actually in effect. The users simply use the final configuration in effect.
  • Therefore, it only affects the implementation of the components, not the definition of the configuration api
• Any changes to the configuration at any source may affect the finalized configuration: reading should return the affected value immediately , listening needs to notify all subscribers.

User case for configuration aggregation:

• Multiple source aggregation: e.g. distributed configuration management system + local configuration files
• Multiple level aggregation: instance-level configuration + App-level configuration