Scenario: when creating a `ServiceMonitor`, the Prometheus config never gets refreshed.
*Check Prometheus operator logs, config reloader logs, Prometheus logs
* Make sure the labelselector specified in your Prometheus instance actually selects the ServiceMonitor you are intending to select and that the namespace of the two objects is the same.
* Check Prometheus config on `/config` in the Prometheus UI. The ServiceMonitor name is encoded in the config, so you should be able to find it if the config was successfully applied.
* If you can find the ServiceMonitor, you are either not selecting the Service you are intending to select, or your service is not selecting any Pods.
* If you cannot find your ServiceMonitor in the config, make sure Prometheus successfully reloaded, and possibly use the metrics it exposes to check when it was last reloaded. As mentioned, you should have already received an alert if Prometheus could not be reloaded successfully for a period of time (longer than 10 minutes I believe).
* Next make sure the Prometheus Operator doesn't have any errors in its logs. Unfortunately validations for CRDs are very new, and we haven't implemented them yet so it is possible to create ServiceMonitors with invalid syntax.
I'm pretty sure in your second bullet point that you're trying to get people to check their ServiceMonitorSelector under the prometheus pod spec, but that's not necessarily clear unless the person knows that the ServiceMonitorSelector is a LabelSelector. Maybe more useful if it's rephrased? (I lost several hours to this)
Please sign in to leave a comment.