Flux monitoring docs lost prometheus alert ReconciliationFailure during transition to kube-state-metrics #1686
Description
When we created the new Flux Custom Metrics guide in the Monitoring docs, we lost something:
Buried at the bottom of this metrics section, there was an example similar to in the Flagger docs where we show if a Canary fails, how to build an alert so that someone is able to intervene. The link above has a similar ReconciliationFailure alert that isn't in the Flux docs anymore, we should figure out where to put it back, I'm sure many people have used it!
At the time I found the doc less than helpful because it doesn't make very clear how to add a new alert to Prometheus with the kube-prometheus-stack chart. I did this:
I don't suggest we do that. It must be possible to create a new PrometheusRule resource alongside of the Flux Monitoring deployment of kube-prometheus-stack. I couldn't figure it out in limited time, so I went with "let's rewrite all of the default alerts, but add one more" because of the difficulty of patching an array of values in Helm values. 😬
This might be a good place to provide another example of how to install a chart that manages CRDs and a CRD alongside of it.
I see how we lost this now, it was buried at the bottom of the Metrics section, and the only relevant bit left, without a subheading of its own. Let's add it back? (Just wanted to document this because I have limited time again today, and the day is almost over!)