recently i started pushing metrics to prometheus but whenever i sent a sample query to prometheus to see the data points, i would get none.
There were no errors in prometheus and i was scratching my head …. grrr…
I was on Mac OS and using a Docker image of Prometheus (version=1.7.1)
When i went into the docker container and checked the dateTime, walaaaaaaaaaaaa
There was a Time mismatch between host and Docker container.
#___7_day__difference__ –> Possible docker bug !!!
since i was searching in the current timebucket, and the datapoints were going to time bucket 7 days ago, i was not able to find em.
Restarting docker for Mac fixed the issue.
Recently the mysql community got an awesome monitoring solution for mysql
You can actually monitor Amazon RDS instance with the same steps mentioned in the above post but with a few changes:
The monitoring framework consists of 4 components:
- Prometheus server on port 9090
- Grafana server on port 3000
- MySQL exporter process which connects to the MySQL Server
- Node Exporter process which gets system metrics of the box hosting MySQL
- Create an RDS instance.
- create an amazon ec2-instance. (this will host all the 4 components)
Component 3 & 4: – Node Exporter & MySQL exporter process :
Amazon _DOES_NOT_ allow us to install anything on the RDS box.
So, I am sorry we will not be able get the System metrics of RDS – please rely on cloudwatch / Rds console for load averages, cpu usage , io etc etc.
So Follow the steps as mentioned in the nice post BUT make the following changes,
- Install the Node exporter & MySQL exporter processes on the ec-instance ,
- So the ‘/opt/prometheus/prometheus.yml’ file will look like
- i.e. you are now monitoring system metrics of the ec2-instance not RDS box !
cat << EOF > /opt/prometheus/prometheus.yml global: scrape_interval: 5s evaluation_interval: 5s scrape_configs: - job_name: linux target_groups: - targets: ['localhost:9100'] labels: alias: db1 - job_name: mysql target_groups: - targets: ['localhost:9104'] labels: alias: db1 EOF
But we need to tell the MySQL exporter to pull from RDS endpoint, so the
my.cnf file for MySQL exporter should be as follows:
[root@centos7 prometheus_exporters]# cat << EOF > .my.cnf [client] user=prom password=abc123 host=amazon-rds-instance.amazonaws.com EOF
Component 1 & 2: – Grafana & Prometheus:
Just Follow the steps as mentioned in the nice post .
And walllaaaah…. you should be able graph Amazon RDS metrics 🙂