Remove host from Grafana
Hi
I'm using Grafana + Prometheus + Node Exporter with Node Exporter Full Dashboard and would like to remove an old host from Grafana Dashboard.
Whenever I delete a series, the data is removed from Prometheus but in the Grafana Dashboard, the host remains there.
These are the commands I use:
curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="my.instance.tld:9100",job="job_name"}'
curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
How can I remove the host from Grafana Dashboard?
Thanks
Comments
Without knowing your config, I'm guessing the dashboard populates the host field with a variable.
Check the settings for the dashboard to see how it generates those variables, my guess is that it's trough a Prometheus query and that the host still exists in the database.
Or you simply have not updated the variable after removing the host from Prometheus, there is a button for it in the dashboard variables settings.
Unless you hardcoded hosts in the Grafana dashboard, it's most likely still living somewhere in Prometheus.
Perhaps you forgot to disable the job for that host and it's still scraping?
I'm testing Grafana, just added a new target and after an hour, I removed from Prometheus configuration.
I'm using the default Prometheus configuration.
if someone is interested, it looks like the only way to do this is to wait up to the retention period (15 days by default)
After removing the instance, I forced two hours and the host was purged from Prometheus.
If you need to configure retention to a value other than 15d, you'll need to set
--storage.tsdb.retention.timein the system's service on ExecStarthttps://prometheus.io/docs/prometheus/latest/storage/#compaction