Prometheus+Grafana monitors Linux hosts and databases

Monitoring overview

  • Monitoring is to help us in real-time to monitor or detect whether the services we deploy are running normally

Commonly used monitoring

  • zabbix
  • Advantages: relatively complete components, disadvantages: low performance (the bottom layer of zabbix is ​​Msql database)
  • When zabbix monitors large clusters, how to optimize (how to optimize the database)
  • When the data in a table in the database exceeds 2000w, the performance of the database drops sharply
  • Alibaba Cloud Cloud Monitoring
  • Tencent Cloud Blue Whale Monitoring
  • Prometheus
  • The performance is relatively high, the bottom layer is used (time series database, the performance of time series database is much higher than the performance of Mysql database)
  • Native support for monitoring containers

Pronesius monitoring process

Prometheus monitoring is divided into two situations

Services carrying metrics interface (kubernetes, ETCD, Docker)

For services that do not carry metrics interface (Nginx, mysql, Linux host), for services that do not carry metrics interface, we need to install an exporter plug-in

Deploy Prometheus

Official website: https://prometheus.io/

Download link: https://github.com/prometheus/prometheus/releases/download/v2.27.1/prometheus-2.27.1.linux-amd64.tar.gz

1. Experimental environment

主机名					外网IP地址					内网IP地址					配置
Prometheus				192.168.15.71				172.16.1.71					内存4G
Web01					192.168.15.7				172.16.1.7					

2. Install Prometheus software

- Prometheus主机
1. 安装Prometheus软件包
rz -E prometheus-2.25.0.linux-amd64.tar.gz
tar -xf prometheus-2.25.0.linux-amd64.tar.gz -C /usr/local/

2. 建立Prometheus超链接
ln -s /usr/local/prometheus-2.25.0.linux-amd64 /usr/local/prometheus

3. 将Prometheus添加至系统环境变量
vim /etc/profile
...
export PROMETHEUS_HOME=/usr/local/prometheus
PATH=$PATH:$PROMETHEUS_HOME
export PATH

4. 重载系统环境变量文件
source /etc/profile

5. 查看Prometheus的版本信息
prometheus --version			# 如果可以查看到Prometheus的版本信息,那么就表示Prometheus安装完成了

6. 启动Prometheus服务
prometheus --config.file=/usr/local/prometheus/prometheus.yml 

3. Test Prometheus's monitoring local state

Insert picture description here


Insert picture description here

Use Prometheus to monitor Linux hosts

Linux host download exporter plug-in link:
https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz

1. Install the exporter plug-in on the Linux host

- Web01主机
1. Linux主机安装exporter插件
rz -E node_exporter-1.1.2.linux-amd64.tar.gz
tar -xf node_exporter-1.1.2.linux-amd64.tar.gz -C /usr/local/

2. 建立node_exporter超链接
ln -s /usr/local/node_exporter-1.1.2.linux-amd64/ /usr/local/node_exporter
ls /usr/local/node_exporter
# LICENSE  node_exporter(执行文件)  NOTICE

3. 将node_exporter执行文件目录添加至环境变量
vim /etc/profile
....
export NODE_EXPORTER=/usr/local/node_exporter
export PATH=$PATH:$NODE_EXPORTER

4. 重载系统环境变量文件
source /etc/profile

5. 启动node_exporter服务
node_exporter

6. 查看node_exporter监听的端口号
netstat -lntp
# tcp6       0      0 :::9100                 :::*                    LISTEN      1701/node_exporter

2. Add the exporter plug-in of the Linux host to the Prometheus software service

- Prometheus主机
1. 修改Prometheus配置文件
vim /usr/local/prometheus/prometheus.yml
scrape_configs:
... 
  - job_name: 'Linux Node'					# 添加监控项的名字
    static_configs:							
      - targets: ['172.16.1.7:9100']		# 监控主机的ip地址和端口号

2. 启动Prometheus服务
prometheus --config.file=/usr/local/prometheus/prometheus.yml 

3. Test Prometheus's monitoring Linux host status

Insert picture description here


Insert picture description here

Use Grafana to display data

Grafana is the best product in the industry for data display. Official website: https://grafana.com/

- Prometheus主机
1. 安装Grafana软件
rz -E grafana-7.3.6-1.x86_64.rpm
yum -y install grafana-7.3.6-1.x86_64.rpm

2. 启动Grafana软件服务
systemctl start grafana-server.service 
netstat -lntp			# 查看grafana服务监听的端口号
# tcp6       0      0 :::3000                 :::*                    LISTEN      1997/grafana-server 

Log in to Grafana Showcase

Insert picture description here


Insert picture description here


Insert picture description here


Insert picture description here

Monitor single status information of Linux host

Insert picture description here


Insert picture description here
Insert picture description here

Monitor the overall status information of the Linux host

Prometheus various display page status codes: https://grafana.com/grafana/dashboards?dataSource=prometheus

Insert picture description here


Insert picture description here

Use Prometheus to monitor Mysql database

Prometheus monitoring Mysql database official website tutorial: https://github.com/prometheus/mysqld_exporter

lab environment

主机名					外网IP地址					内网IP地址					配置
Prometheus				192.168.15.71				172.16.1.71					内存4G
db01					192.168.15.51				172.16.1.51		

1. Install the exporter plugin on the Mysql host

- db01主机
1. db01主机安装exporter插件
rz -E mysqld_exporter-0.12.1.linux-amd64.tar.gz
tar -xf mysqld_exporter-0.12.1.linux-amd64.tar.gz -C /usr/local/

2. 建立mysql_exporter超链接
ln -s /usr/local/mysqld_exporter-0.12.1.linux-amd64 /usr/local/mysqld_exporter
ls /usr/local/mysqld_exporter
# LICENSE  mysqld_exporter(执行文件)  NOTICE

3. 创建一个数据库用户
mysql -uroot -p123
CREATE USER 'exporter'@'%' IDENTIFIED BY '123';			# 创建一个数据库用户
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';			# 给创建的数据库用户分配一个权限
use mysql;					# 切换到mysql库中
select Host,User from user;			# 查看创建的用户信息
#  Host       User     
#  %          exporter 

4. 创建数据库用户和密码文件
vim /usr/local/mysqld_exporter/.my.cnf
[client]
host=172.16.1.51
user=exporter
password=123

5. 启动mysqld_exporter服务
cd /usr/local/mysqld_exporter/
./mysqld_exporter --config.my-cnf="/usr/local/mysqld_exporter/.my.cnf"
# INFO[0000] Listening on :9104                            source="mysqld_exporter.go:283"

如果启动mysqld_exporter服务出现 
Access denied for user ‘root‘@‘db01‘(using password: YES) source=“exporter.go:146“报错,
请参考下一篇博文解决方法

2. Add the exporter plug-in of the Mysql host to the Prometheus software service

- Prometheus主机
1. 修改Prometheus配置文件
vim /usr/local/prometheus/prometheus.yml
scrape_configs:
... 
  - job_name: 'Mysql Server'					# 添加监控项的名字
    static_configs:							
      - targets: ['172.16.1.51:9104']		# 监控主机的ip地址和端口号

2. 启动Prometheus服务
prometheus --config.file=/usr/local/prometheus/prometheus.yml 

3. Test Prometheus monitoring Mysql host status

Insert picture description here


Insert picture description here

4. Mysql host status is added to Garafana display

Insert picture description here


Insert picture description here


Insert picture description here
Insert picture description here


Insert picture description here


Insert picture description here