linux12 -prometheus monitoring --> brief description and installation

Article Directory

Prometheus monitoring

监控就是实时的帮助我们来监控或者探测我们部署的服务是否正常运行。

1. Commonly used monitoring

- zabbix
  - 组件比较全,缺点:性能不高
  - 当zabbix监控大型集群的时候,怎么优化
  - 当数据库中一个表中数据超过2000w的时候,数据库的性能急剧下降
  
- 阿里云云监控
- 腾讯云蓝鲸监控
- 普罗米修斯
  - 性能比较高,底层使用(时序数据库)
  - 原生支持监控容器

Two, Promethus monitoring system introduction

能够安装prometheus服务器
能够通过安装node_exporter监控远程linux
能够通过安装mysqld_exporter监控远程mysql数据库
能够安装grafana
能够在grafana添加prometheus数据源
能够在grafana添加监控cpu负载的图形
能够在grafana图形显示mysql监控数据

# 官网: https://prometheus.io/
下载链接: https://github.com/prometheus/prometheus/releases/download/v2.27.1/prometheus-2.27.1.linux-amd64.tar.gz
Prometheus monitoring process
Prometheus monitoring is divided into two situations:
1. Services carrying metrics interface (kubernetes, ETCD, Docker)
2. For services that do not carry metrics interface (Nginx, mysql, Linux host), for services that do not carry metrics interface, we need to install an export plug-in

1. General description

Prometheus(由go语言(golang)开发)是一套开源的监控&报警&时间序列数据库的组合。适合监控docker容器。因为kubernetes(俗称k8s)的流行带动了prometheus的发展。
Insert picture description here
Time series data
时间序列数据(TimeSeries Data) : 按照时间顺序记录系统、设备状态变化的数据被称为时序数据。
1. Features
# 性能好

关系型数据库对于大规模数据的处理性能糟糕。NOSQL可以比较好的处理大规模数据,让依然比不上时间序列数据库。

# 成本低

# 高效的压缩算法,节省存储空间,有效降低IO

Prometheus有着非常高效的时间序列数据存储方法,每个采样数据仅仅占用3.5byte左右空间,上百万条时间序列,30秒间隔,保留60天,大概花了200多G(来自官方数据)
2. Features
多维度数据模型
灵活的查询语言(PromQL)
不依赖分布式存储,单个服务器节点是自治的
以HTTP方式,通过pull模型拉去时间序列数据
也可以通过中间网关支持push模型
通过服务发现或者静态配置,来发现目标服务对象
支持多种多样的图表和界面展示
3. Environment settings
Host nameipservice
Prometheus172.16.1.125Grafana, prometheus
web01172.16.1.7node_exporter
web02172.16.1.7node_exporter
web03172.16.1.7node_exporter
db01172.16.1.51mysqld_exporter
4. Prometheus server deployment
- 从 https://prometheus.io/download/ 下载相应版本,安装到服务器上官网提供的是二进制版,解压就能用,不需要编译。
# 方式一:prometheus手动安装启动
# 1、下载
[[email protected] opt]# wget https://github.com/prometheus/prometheus/releases/download/v2.27.1/prometheus-2.27.1.linux-amd64.tar.gz

# 2、解压
[[email protected] opt]# tar -xf prometheus-2.25.0.linux-amd64.tar.gz -C /usr/local/
[[email protected] ~]# cd /usr/local/

# 3、建立超链接
[[email protected] local]# ln -s /usr/local/prometheus-2.25.0.linux-amd64 /usr/local/prometheus
[[email protected] local]# ll
lrwxrwxrwx  1 root root  30 Jun  2 09:49 prometheus -> prometheus-2.25.0.linux-amd64/

# 4、创建环境变量
[[email protected] local]# cat >> /etc/profile.d/prometheus.sh <<EOF
export PROMETHEUS_HOME=/usr/local/prometheus
PATH=$PATH:$PROMETHEUS_HOME
export PATH
EOF
[[email protected] ~]# source /etc/profile

# 5、测试安装成功
[[email protected] ~]# prometheus --version
prometheus, version 2.25.0 (branch: HEAD, revision: a6be548dbc17780d562a39c0e4bd0bd4c00ad6e2)
  build user:       [email protected]
  build date:       20210217-14:17:24
  go version:       go1.15.8
  platform:         linux/amd64
  
# 6、查看配置文件(prometheus.yml)
[[email protected] ~]# cd /usr/local/prometheus
[[email protected] prometheus]# ll
total 167984
drwxr-xr-x 2 3434 3434       38 Feb 18 00:11 console_libraries
drwxr-xr-x 2 3434 3434      173 Feb 18 00:11 consoles
-rw-r--r-- 1 3434 3434    11357 Feb 18 00:11 LICENSE
-rw-r--r-- 1 3434 3434     3420 Feb 18 00:11 NOTICE
-rwxr-xr-x 1 3434 3434 91044140 Feb 17 22:19 prometheus
-rw-r--r-- 1 3434 3434     1170 Jun  2 18:16 prometheus.yml
-rwxr-xr-x 1 3434 3434 80948693 Feb 17 22:21 promtool

# 7、启动prometheus
[[email protected] ~]# prometheus --config.file=/usr/local/prometheus/prometheus.yml


# 方式二:prometheus的systemd脚本执行安装启动
[[email protected] files]# cat prometheus.sh
echo "1.下载"
cd /opt/ &&\

wget https://github.com/prometheus/prometheus/releases/download/v2.27.1/prometheus-2.27.1.linux-amd64.tar.gz &&\

echo "2.解压"
tar -xf /opt/prometheus-2.27.1.linux-amd64.tar.gz -C /usr/local/ &&\

echo "3. 建立超链接"
ln -s /usr/local/prometheus-2.27.1.linux-amd64 /usr/local/prometheus &&\

echo "4.创建环境变量"
echo "export PATH=$PATH:/usr/local/prometheus/" >> /etc/profile.d/prometheus.sh &&\

echo "5.加载环境变量"
source /etc/profile &&\

echo "6.创建promethets的systemd启动文件"
cat >>/usr/lib/systemd/system/prometheus.service <<EOF
[Unit]
Description=https://prometheus.io

[Service]    
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml

[Install]
WantedBy=multi-user.target  
EOF
echo "7.启动promethets"
systemctl daemon-reload &&\
systemctl enable --now prometheus.service

# 访问
http://192.168.16.125:9090/
通过`http://服务器IP:9090/metrics`可以查看到监控的数据,在web主界面可以通过关键字查询监控项 

5. Use Garafana to display data

Garafana is a good product for data display in the industry
Download website: https://grafana.com/get/?plcmt=top-nav&cta=downloads
[[email protected] opt]# rz -E
[[email protected] opt]# ll
-rw-r--r-- 1 root root 53727312 Jun  2 11:21 grafana-7.3.6-1.x86_64.rpm

[[email protected] opt]# yum install -y grafana-7.3.6-1.x86_64.rpm

[[email protected] opt]# systemctl start grafana-server.service

[[email protected] opt]# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name                       
tcp6       0      0 :::3000                 :::*                    LISTEN      10137/grafana-serve 

#访问
HTTPS://172.16.1.81:3000
Insert picture description here
6. Test the connection

Enter: http://ip:9090 on the browser, the following page appears and it means the startup is successful.

[External link image transfer failed. The source site may have an anti-leech link mechanism. It is recommended to save the image and upload it directly (img-OqVJWV9i-1622889846353) (C:\Users\17155\Desktop\download image\1622598243380.png)]

7. Web client deployment

1. Install the node_exporter component

# 在被web集群上安装node_exporte脚本执行安装组件

# 在web集群三台机器分别执行该脚本
[[email protected]]# cat node.sh  
echo "1.下载node包"
wget https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz

echo "2.解压"
tar -xf node_exporter-1.1.2.linux-amd64.tar.gz -C /usr/local/
echo "3. 建立超链接"
ln -s /usr/local/node_exporter-1.1.2.linux-amd64/ /usr/local/node_exporter &&\

echo "4.创建systemd服务"
cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=This is prometheus node exporter
After=node_exporter.service

[Service]
Type=simple
ExecStart=/usr/local/node_exporter/node_exporter
ExecReload=/bin/kill -HUP
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
echo "5.启动node_exporter服务"
systemctl daemon-reload &&\
systemctl enable --now node_exporter.service

参数:
	--web.listen-address=":9100" 	#修改默认端口,防止冲突
	--web.telemetry-path="/metrics" #获取metric信息的url,默认是/metrics,若需要修改则通过此参数
    --log.level="info" 			    #设置日志级别
    --log.format="logger:stderr"     #设置打印日志的格式,若有自动化日志提取工具可以使用这个参数规范日志打印的格式
2. Go back to the configuration file of the prometheus server and add the configuration section of the monitored machine
在主配置文件最后加上下面三行
  - job_name: 'agent01'
    static_configs:
      - targets: ['172.16.1.7:9100']

改完配置文件后,重启服务
[[email protected] opt]# /usr/local/prometheus-2.25.0/prometheus --config.file="/usr/local/prometheus-2.25.0/prometheus.yml" &

回到web管理界面 --> 点Status --> 点Targets --> 可以看到多了两台监控目标

8.Monitor MySQL

1. Install the mysqld_exporter component
# 在db01机器上安装mysqld_exporter组件
# 方式一: mysqld_exporter手动安装组件
# 1、上传解压包
[[email protected] opt]# rz -E
-rw-r--r-- 1 root root 7121565 Apr 22 21:33 mysqld_exporter-0.12.1.linux-amd64.tar.gz
[[email protected] opt]# tar -xf mysqld_exporter-0.12.1.linux-amd64.tar.gz -C /usr/local/
[[email protected] opt]# cd /usr/local/
[[email protected] local]# ll
drwxr-xr-x  2 3434 3434 58 Jul 29  2019 mysqld_exporter-0.12.1.linux-amd64
# 2、软连接
[[email protected] local]# ln -s /usr/local/mysqld_exporter-0.12.1.linux-amd64/ /usr/local/mysqld_exporter
lrwxrwxrwx  1 root root 46 Jun  2 17:47 mysqld_exporter -> /usr/local/mysqld_exporter-0.12.1.linux-amd64/

# 3、创建用户并授权以及查看用户权限
[[email protected] ~]# mysql -uroot -p123

MariaDB [(none)]> create user 'root'@'%' identified by '123';
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> grant process, replication client, select on *.* to 'root'@'%';
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mysql]> select host,user from user;
+------------+----------+
| host       | user     |
+------------+----------+
| %          | root |
| 127.0.0.1  | root     |
| 172.16.1.% | www      |
| ::1        | root     |
| db01       |          |
| db01       | root     |
| localhost  |          |
| localhost  | root     |
+------------+----------+
8 rows in set (0.00 sec)
# 4、添加文件
# 创建一个mariadb配置文件,写上连接的用户名与密码(和上面的授权的用户名 和密码要对应)
[[email protected] mysqld_exporter]# vim .my.cnf
[client]
host=172.16.1.51
user=root
password=123
# 5、启动并查看监控是否成功(9104端口)
[[email protected] mysqld_exporter]# ./mysqld_exporter  --config.my-cnf="/usr/local/mysqld_exporter/.my.cnf"


# 方式二: mysqld_exporter脚本安装组件
[[email protected] files]# cat mysqld_exporter.sh 
echo "1.上传解压包 ”
[[email protected] opt]# rz -E
-rw-r--r-- 1 root root 7121565 Apr 22 21:33 mysqld_exporter-0.12.1.linux-amd64.tar.gz
echo "2.解压"
[[email protected] opt]# tar -xf mysqld_exporter-0.12.1.linux-amd64.tar.gz -C /usr/local/
[[email protected] opt]# cd /usr/local/
[[email protected] local]# ll
drwxr-xr-x  2 3434 3434 58 Jul 29  2019 mysqld_exporter-0.12.1.linux-amd64
echo "3. 建立超链接"
ln -s /usr/local/mysqld_exporter-0.12.1.linux-amd64/ /usr/local/mysqld_exporter &&\
echo "4.编辑my.cnf"

cat >> /usr/local/mysqld_exporter/.my.cnf <<EOF
[client]
host=172.16.1.51
user=root
password=123
EOF

echo "5.创建systemdqldmysqld_exporter.service务"
cat >> /usr/lib/systemd/system/mysqld_exporter.service <<EOF
[Unit]
Description=Prometheus

[Service]
Environment=DATA_SOURCE_NAME=root:[email protected](172.16.1.51:3306)/
ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/.my.cnf --web.listen-address=:9104
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

echo "6.启动node_exporter服务"
systemctl daemon-reload &&\
systemctl enable --now mysqld_exporter.service

# 普罗米修斯是通过mysql_exporter来找mariadb进行获取数据的,所以要给它授权(用户名可自定义,与.my.cnf对应)
# (注意:授权ip为localhost,因为不是prometheus服务器来直接找mariadb 获取数据,而是prometheus服务器找mysql_exporter,mysql_exporter 再找mariadb。所以这个localhost是指的mysql_exporter的IP)
2. Go back to the configuration file of the prometheus server and add the configuration section of the monitored machine
[[email protected] ~]# vim /usr/local/prometheus/prometheus.yml 
  - job_name: 'Mysql server'
    static_configs:
      - targets:
        - "172.16.1.51:9104"
#重载prometheus
[[email protected] ~]# prometheus --config.file=/usr/local/prometheus/prometheus.yml
Insert picture description here