Context

VMOperator merges routes in user-provided VMAlertManagerConfig into the routing tree in a shared VMAlertManager.
When a new top-level route for a new VMAlertManagerConfig is built, the namespace label of VMAlertManagerConfig is used to make a label matcher for incoming alerts. This requires that alerts have a namespace label that refers to the namespace of a VMAlertManagerConfig that contains desired routes.

• There is an option to disable this dynamic routing rule: disableNamespaceMatcher
• There is an option to add enforced matchers that are static at time of VMAlertMnager creation: enforcedTopRouteMatchers
• There are no options to customise the dynamic matcher.

Problem

The namespace label is hard-coded in VMOperator.
This works if the user-owned alert rules set the namespace label to the namespace of a VMAlertManagerConfig that contains desired routes.
But, the namespace label name is (ironically) not namespaced, and is widely used in other contexts (like Kubernetes), therefore it is ambiguous and overloaded. It can be set by users and often appears in source metrics (used in alert rules) to mean other types of namespaces in the infrastructure or business domains. This means alert routing does not work, or the namespace label from source telemetry has to be overwritten.

Proposal

The namespace label in this context should mean "the namespace of the VMAlertManagerConfig to which the alert should be routed", not "the namespace provided by the alert expression". We need to avoid conflicts with labels from source metrics.
Option 1: A better label name would be something like "alertmanagerconfig_namespace"... but backwards compatibility is needed.
Option 2: Making it configurable in the operator would be much better, with the default being namespace for compatibility.
Option 3: Allow an override for the label name in the VMAlertManagerConfig. For example, a field called topRouteNamespaceLabel could be provided, with the default being namespace for compatibility.
edit:
Option 4: Make it configurable in each VMAlertmanager - best for flexibility and security