Alertmanager(kind: Alertmanager) is running in prometh namespace.
We have access to push alert rule definitions(kind: PrometheusRule) to prometh namespace, with additional labels in alert payload(sourceNamespace & destinationNamespace), as shown below:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: service-prometheus
role: alert-rules
app: kube-prometheus-stack
release: monitoring-prom
name: rule_a
namespace: monitoring
spec:
groups:
- name: rule_a_alert
rules:
- alert: usage_exceed
expr: salesforce_api_usage > 100000
labels:
severity: urgent
sourceNamespace: app-dev
destinationNamespace: promethValue of sourceNamespace indicate the namespace where application pods are running.
We have alert manager config(kind: AlertmanagerConfig) that match with rule definition(above) using below syntax:
route:
routes:
- match:
severity: urgentbut we do not have access to push alert manager config into prometh namespace. we have access to push alert manager config only into app-dev
Is it possible to start alert manager, by enabling it to register new alertmanager configs from a new sourceNamespace in runtime? with alertmanager.alertmanagerSpec.alertmanagerConfigNamespaceSelector field
Related discussion: https://issueexplorer.com/issue/prometheus-operator/prometheus-operator/4263