(Linux系统的日志管理)
一、systemd-journald服务
1.systemd-journald介绍
# 系统时间日志的核心服务
# 收集包括来自内核，早期引导阶段啊，进程启动，运行时的标准输出和错误输出，以及syslog时间等
# 保存在二进制的日志文件中，具有易失性，重启后不保存
2.系统日志默认保存路径
保存目录为/run/log/journal
[root@tianyi 4f596c775d924b618367d1c448fd5578]# pwd
/run/log/journal/4f596c775d924b618367d1c448fd5578
[root@tianyi 4f596c775d924b618367d1c448fd5578]# ls
system@f51ef91c90ff4fd48133fe727841edb8-000000000011b8bf-0005c7be331e995d.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000011d6a0-0005c7c4724c57e6.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000011f42f-0005c7cab33cc55b.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000121201-0005c7cf0099fa95.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000122fca-0005c7db1bc72be2.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000124d63-0005c7e173a7688d.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000126bf3-0005c7e70d3d55d0.journal
system@f51ef91c90ff4fd48133fe727841edb8-0000000000128a4c-0005c7ebf2c2f08f.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000012a845-0005c7ef788f2d0b.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000012c602-0005c7f31a53c98f.journal
system@f51ef91c90ff4fd48133fe727841edb8-000000000012e3d8-0005c7f8cdf9eace.journal
system.journal
3.systemd-journald的配置文件修改
①配置文件目录
/etc/systemd/journald.conf
②配置文件修改
persistent: 将日志存储在/var/log/journal目录中，若该目录不存在，则systemd-journald服务自动创建。 volatile： 将日志存储在易失性的目录/run/log/journal目录中，若该目录不存在，则systemd-journald服务自动创建。 auto： 如果/var/log/journal目录存在，那么rsyslog会使用持久存储，否则为易失性存储，此为默认配置。
[root@tianyi systemd]# grep auto journald.conf
#Storage=auto
4.检索日志消息
①journalctl命令的相关选项
-n 指定显示末尾几条消息，默认最后10条日志消息
-f 与tail -f 命令相似
-p 指定显示某个优先级以上的日志
--since或者--until 限制特定的时间段，时间格式为“YYYY-MM-DD hh:mm:ss”等
-o verbose 查看日志的详细信息
_PID 进程的PID
_UID 运行该进程的用户ID
_SYSTEMD_UNIT 启动该进程的systemd单元
_COMM 指定命令的名称
_EXE 京城的可执行文件的路径
②示例
[root@tianyi systemd]# journalctl -n 5 _SYSTEMD_UNIT=sshd.service
-- Logs begin at Fri 2021-07-23 06:56:38 CST, end at Mon 2021-07-26 23:20:45 CST. --
Jul 26 23:20:33 tianyi sshd[10011]: pam_unix(sshd:auth): check pass; user unknown
Jul 26 23:20:33 tianyi sshd[10011]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.117.46.219
Jul 26 23:20:36 tianyi sshd[10011]: Failed password for invalid user vbox from 1.117.46.219 port 41028 ssh2
Jul 26 23:20:36 tianyi sshd[10011]: Received disconnect from 1.117.46.219 port 41028:11: Bye Bye [preauth]
Jul 26 23:20:36 tianyi sshd[10011]: Disconnected from invalid user vbox 1.117.46.219 port 41028 [preauth
二、rsyslog服务
1.日志保存相关路径
/var/log/message 系统绝大多数的日志文件都会记录在该文件中 /var/log/secure 记录与安全相关的日志，比如创建用户，修改密码，切换用户等操作 /var/log/cron 记录与计划任务相关 /var/log/maillog 记录与邮件相关的日志 /var/log/boot.log 记录与启动相关的
2.日志消息的类型
lpr: 打印相关的日志 auth：认证相关的日志 user：用户相关的日志 cron：计划任务相关的日志 kern：内核相关的日志 mail：邮件相关的日志 daemon:系统服务相关的日志 authpri：授权相关的日志 security：安全相关的日志 local0-local7:自定义相关的日志信息
3.日志消息优先级
| 优先级 | 代码 | 严重性 |
| none | 无 | 不记录任何信息 |
| emerg | 内核崩溃等严重信息 | |
| alert | 1 | 需要立即修改的信息 |
| crit | 2 | 严重错误级别 |
| err | 3 | 错误级别 |
| warning | 4 | 警告级别 |
| notice | 5 | 具有重要性的普通信息 |
| info | 6 | 一般信息 |
| warning | 7 | 调试信息 |
4.自定义日志规则
①日志定义的格式
facility.priority Target
mail.info /var/log/maillog 比指定级别更高的日志级别，包括级别自身，保存到 /var/log/maillog
mail.=info /var/log/maillog 明确指定级别为info,保存到/var/log/maillog
mail.！info /var/log/maillog 除了指定的日志级别以外的级别，都保存到/var/log/maillog
*.info info /var/log/maillog 所有类型的info级别都保存到/var/log/maillog
mail.* /var/log/maillog mail类型的所有级别日志都保存到/var/log/maillog
三、日志查看
[root@tianyi log]# grep -E -C 5 '(err|not|no)' messages
Jul 25 05:44:39 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device/start timed out.
Jul 25 05:44:39 tianyi systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device.
Jul 25 05:44:39 tianyi systemd[1]: Dependency failed for /dev/disk/by-uuid/c30fcda5-d830-4c90-b818-831e33389b2e.
Jul 25 05:44:39 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.swap: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.swap/start failed with result 'dependency'.
Jul 25 05:44:39 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device/start failed with result 'timeout'.
Jul 25 07:40:57 tianyi rsyslogd[1750]: imjournal: sd_journal_get_cursor() failed: Cannot assign requested address [v8.37.0-9.el8]
Jul 25 07:40:57 tianyi rsyslogd[1750]: imjournal: journal reloaded... [v8.37.0-9.el8 try http://www.rsyslog.com/e/0 ]
Jul 25 07:41:56 tianyi systemd[1]: Starting dnf makecache...
Jul 25 07:42:02 tianyi dnf[6483]: Docker CE Stable - x86_64 690 B/s | 3.5 kB 00:05
Jul 25 07:42:07 tianyi dnf[6483]: Zabbix 590 B/s | 2.9 kB 00:05
Jul 25 07:42:12 tianyi dnf[6483]: huawei-AppStream 875 B/s | 4.3 kB 00:05
--
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info> [1627190262.1609] dhcp4 (eth0): hostname 'host-192-168-1-209'
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info> [1627190262.1609] dhcp4 (eth0): gateway 192.168.1.1
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info> [1627190262.1609] dhcp4 (eth0): static route 169.254.169.254/32 gw 192.168.1.254
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info> [1627190262.1609] dhcp4 (eth0): mtu 1500
Jul 25 13:17:42 tianyi NetworkManager[1936]: <info> [1627190262.1611] dhcp4 (eth0): state changed bound -> bound
Jul 25 13:17:42 tianyi dbus-daemon[614]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.33' (uid=0 pid=1936 comm="/usr/sbin/NetworkManager --no-daemon ")
Jul 25 13:17:42 tianyi systemd[1]: Starting Network Manager Script Dispatcher Service...
Jul 25 13:17:42 tianyi dbus-daemon[614]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Jul 25 13:17:42 tianyi systemd[1]: Started Network Manager Script Dispatcher Service.
Jul 25 13:17:42 tianyi nm-dispatcher[11673]: req:1 'dhcp4-change' [eth0]: new request (5 scripts)
Jul 25 13:17:42 tianyi nm-dispatcher[11673]: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
Jul 25 13:19:12 tianyi systemd[1]: dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device: Job dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device/start timed out.
Jul 25 13:19:12 tianyi systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-c30fcda5\x2dd830\x2d4c90\x2db818\x2d831e33389b2e.device.
四、日志服务器搭建
1.日志服务器介绍
rstslog是一个C/S架构，可以基于TCP和UDP工作，默认监听端口为514
2.服务端配置
①修改服务端配置文件
[root@IT-01 log]# vim /etc/rsyslog.conf
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
②重启服务及防火墙关闭
[root@IT-01 log]# systemctl restart rsyslog.service
[root@IT-01 log]# systemctl stop firewalld
3.客户端配置
①修改配置文件
vim /etc/rsyslog.conf
*.info @192.168.200.129
②重启服务
[root@node1 ~]# systemctl restart rsyslog.service
4.测试日志
服务端查看
[root@IT-01 log]# tail -n 5 /var/log/messages
Jul 27 00:03:02 node1 dnf[3099]: Repository AppStream is listed more than once in the configuration
Jul 27 00:03:02 node1 dnf[3099]: Repository BaseOS is listed more than once in the configuration
Jul 27 00:03:02 node1 dnf[3099]: Repository AppStream is listed more than once in the configuration
Jul 27 00:03:02 node1 dnf[3099]: Repository BaseOS is listed more than once in the configuration
Jul 27 00:03:13 node1 dnf[3099]: CentOS-8 - AppStream - mirrors.aliyun.com 0.0 B/s | 0 B 00:10
五、日志轮询
1.logrotate介绍
logrotate工具会轮询日志文件，防止日志文件占用过多的系统空间。
配置文件
[root@IT-01 log]# /etc/logrotate.d/*
[root@IT-01 log]# /etc/logrotate.conf
2.日志目录查看
[root@IT-01 log]# ls
anaconda cron firewalld maillog-20210623 qemu-ga spooler-20210725 vmware-network.9.log
audit cron-20210623 gdm maillog-20210707 rhsm sssd vmware-network.log
boot.log cron-20210707 glusterfs maillog-20210717 samba swtpm vmware-vgauthsvc.log.0
boot.log-20210506 cron-20210717 hawkey.log maillog-20210725 secure tuned vmware-vmsvc-root.log
boot.log-20210507 cron-20210725 hawkey.log-20210623 messages secure-20210623 vmware vmware-vmtoolsd-root.log
boot.log-20210508 cups hawkey.log-20210707 messages-20210623 secure-20210707 vmware-network.1.log vmware-vmusr-root.log
boot.log-20210511 dnf.librepo.log hawkey.log-20210717 messages-20210707 secure-20210717 vmware-network.2.log wtmp
boot.log-20210531 dnf.librepo.log-20210623 hawkey.log-20210725 messages-20210717 secure-20210725 vmware-network.3.log zabbix
boot.log-20210613 dnf.librepo.log-20210707 httpd messages-20210725 speech-dispatcher vmware-network.4.log
boot.log-20210722 dnf.librepo.log-20210717 lastlog mysql spooler vmware-network.5.log
btmp dnf.librepo.log-20210725 libvirt php-fpm spooler-20210623 vmware-network.6.log
btmp-20210707 dnf.log mail ppp spooler-20210707 vmware-network.7.log
chrony dnf.rpm.log maillog private spooler-20210717 vmware-network.8.log