Linux日志系统

日志系统负责记录系统运行过程中内核产生的各种信息，并分别将它们存放到不同的欸之文件中，以便系统管理源进行故障排查、异常跟踪。
REHL8使用rsylog作为日志服务程序。

介绍

syslog是一个综合的日志记录系统，主要功能是为了方便管理日志和分类存放系统日志。
使用grep命令查看日志很费劲，rsylog是syslog的替代产品。Redhat和Fedora使用rsylog替换了syslog

rsylog配置文件及语法

rsylog默认配置文件为/etc/rsyslog.conf，定义了系统中需要监听的事件和对应的日志文件的保存位置。

例子

*.info;mail.none;authpriv.none;cron.none                /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure
# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog
# Log cron stuff
cron.*                                                  /var/log/cron
# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler
# Save boot messages also to boot.log
local7.*

每一行由两部分组成：
第一部分是一个或多个“设备”，设备后面跟一些空格字符
第二部分是一个“操作动作”

设备

设备本身分为两个字段，之间用一个小数点“.”分割。
前一个字段表示一项服务
后一个字段是一个优先级
通过设备将不同类型的消息发送到不同的地方。
在同一个rsyslog配置行上允许出现一个以上设备，但是必须使用“;”把他们分隔开
Linux操作系统可以识别的设备。

优先级

优先级代表消息的紧急程度。
优先级定义消息的紧急程序，按严重程度按高到底排列如表

优先级限定符

使用三种限定符对优先级进行修饰：

• 星号（*） 表示对应服务生成的所有日志消息都发送到操作动作到指定的地点
• 等号（=） 表示只把对应服务生成的本优先级的日志消息都发送到操作动作指定的地点
• 叹号（！） 表示把对应服务生成的所有日志消息都发生到操作动作指定的地点，但是本优先级的消息不包括在内，类似于编程语言中“非”运算符的用法

  操作动作

  日志信息可以分别记录到多个文件中，还可以发送到命名管道、其他程序，甚至远程主机
  常见的动作
  File：指定日志文件的绝对路径
  terminal或print：发送到串行或并行设备标志符。eg：/dev/ttyS2
  @host：远程的日志服务器
  Username：发送信息到本机的指定用户终端中，前提是该用户必须已经登录到系统中
  named pipe：发送到预先使用mkfifo命令来创建FIFO文件的绝对路径

  logrotate

  logrotate可以自动完成日志的压缩、备份、删除等工作，并可以设置为定时任务，eg：每日、每周或每月处理。

  参数

  | 参数 | 说明 | | —- | —- | | -d | 详细显示指令执行过程，便于调试或了解程序执行的情况 | | -f | 强行启动记录文件维护操作 | | -s | 使用指定的状态文件 | | -v | 在执行日志滚动时显示详细信息 | | -？ | 显示帮助信息 |

配置文件

主配置文件
/etc/logrotate.conf 和 /etc/logrotate.d

[root@lhuan ~]# cat /etc/logrotate.conf 
# see "man logrotate" for details
#使用man logrotate查看更多帮助消息

# rotate log files weekly
#每周轮转
weekly

#保存过去4周的文件
# keep 4 weeks worth of backlogs
rotate 4

#轮转后创建新的空日志文件
# create new (empty) log files after rotating old ones
create

#轮转的文件以日志结尾，eg：messages-20201012
# use date as a suffix of the rotated file
dateext

#如果需要将轮转后日志压缩，可以去掉此行的注释
# uncomment this if you want your log files compressed
#compress

#其他配置可以放到此文件夹中
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

#一些系统日志的轮转规则
# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

配置文件参数

systemd日志

systemd主要是服务管理所产生的日志和系统日志

配置文件

/etc/systemd/journald.conf

[root@lhuan ~]# cat /etc/systemd/journald.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

journalctl

1.journalctl 展示的是一个交互式界面，用空格翻页，q键退出

[root@lhuan ~]# journalctl 
-- Logs begin at Sat 2020-02-22 18:16:12 CST, end at Wed 2020-10-14 00:08:26 CST. --
Feb 22 18:16:12 VM_0_3_centos sshd[12331]: Received disconnect from 193.112.97.106 port 33710:11: Bye Bye
Feb 22 18:16:12 VM_0_3_centos sshd[12331]: Disconnected from 193.112.97.106 port 33710
Feb 22 18:24:57 VM_0_3_centos passwd[13514]: pam_unix(passwd:chauthtok): password changed for ftpuser
Feb 22 18:25:08 VM_0_3_centos passwd[13553]: pam_unix(passwd:chauthtok): authentication failure; logname=ftpuser uid=1000 euid=0 tty=pts/1 ruser= rhos
Feb 22 18:25:15 VM_0_3_centos passwd[13565]: pam_unix(passwd:chauthtok): authentication failure; logname=ftpuser uid=1000 euid=0 tty=pts/1 ruser= rhos
Feb 22 18:25:20 VM_0_3_centos passwd[13577]: pam_unix(passwd:chauthtok): authentication failure; logname=ftpuser uid=1000 euid=0 tty=pts/1 ruser= rhos
-- Reboot --

查看内核日志 -k

[root@lhuan ~]# journalctl  -k
-- Logs begin at Sat 2020-02-22 18:16:12 CST, end at Wed 2020-10-14 00:09:25 CST. --
Sep 03 15:21:11 VM_69_195_centos kernel: Initializing cgroup subsys cpuset
Sep 03 15:21:11 VM_69_195_centos kernel: Initializing cgroup subsys cpu
Sep 03 15:21:11 VM_69_195_centos kernel: Initializing cgroup subsys cpuacct
Sep 03 15:21:11 VM_69_195_centos kernel: Linux version 3.10.0-1062.9.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Re
Sep 03 15:21:11 VM_69_195_centos kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1062.9.1.el7.x86_64 root=UUID=4b499d76-769a-40a0-93dc-4a31a59ad
Sep 03 15:21:11 VM_69_195_centos kernel: e820: BIOS-provided physical RAM map:
Sep 03 15:21:11 VM_69_195_centos kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Sep 03 15:21:11 VM_69_195_centos kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Sep 03 15:21:11 VM_69_195_centos kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved

查看系统本次启动的日志 -b

[root@lhuan ~]# journalctl  -b
-- Logs begin at Sat 2020-02-22 18:16:12 CST, end at Wed 2020-10-14 00:09:51 CST. --
Sep 03 15:21:11 VM_69_195_centos systemd-journal[89]: Runtime journal is using 8.0M (max allowed 91.8M, trying to leave 137.8M free of 910.9M availabl
Sep 03 15:21:11 VM_69_195_centos kernel: Initializing cgroup subsys cpuset
Sep 03 15:21:11 VM_69_195_centos kernel: Initializing cgroup subsys cpu
Sep 03 15:21:11 VM_69_195_centos kernel: Initializing cgroup subsys cpuacct
Sep 03 15:21:11 VM_69_195_centos kernel: Linux version 3.10.0-1062.9.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Re
Sep 03 15:21:11 VM_69_195_centos kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1062.9.1.el7.x86_64 root=UUID=4b499d76-769a-40a0-93dc-4a31a59ad
Sep 03 15:21:11 VM_69_195_centos kernel: e820: BIOS-provided physical RAM map:
Sep 03 15:21:11 VM_69_195_centos kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable

查看指定时间的日志

root@lhuan ~]# journalctl  --since yesterday
-- Logs begin at Sat 2020-02-22 18:16:12 CST, end at Wed 2020-10-14 00:13:31 CST. --
Oct 13 00:00:01 lhuan systemd[1]: Started Session 57686 of user root.
Oct 13 00:00:01 lhuan systemd[1]: Started Session 57687 of user root.
Oct 13 00:00:01 lhuan CROND[15527]: (root) CMD (/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &)
Oct 13 00:00:01 lhuan CROND[15528]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1 &)


[root@lhuan ~]# journalctl  --since "2020-09-09" --until "2 day ago"
-- Logs begin at Sat 2020-02-22 18:16:12 CST, end at Wed 2020-10-14 00:14:45 CST. --
Sep 09 00:00:01 lhuan systemd[1]: Started Session 7856 of user root.
Sep 09 00:00:01 lhuan CROND[25107]: (root) CMD (/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &)
Sep 09 00:00:01 lhuan systemd[1]: Started Session 7857 of user root.
Sep 09 00:00:01 lhuan CROND[25108]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1 &)
Sep 09 00:00:21 lhuan sshd[25160]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.85.53.58  user=root
Sep 09 00:00:21 lhuan sshd[25160]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Sep 09 00:00:23 lhuan sshd[25160]: Failed password for root from 111.85.53.58 port 39765 ssh2


[root@lhuan ~]# journalctl  --since  "30 min ago"
-- Logs begin at Sat 2020-02-22 18:16:12 CST, end at Wed 2020-10-14 00:15:19 CST. --
Oct 13 23:45:19 lhuan sshd[4811]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.230.157.219  user=root
Oct 13 23:45:19 lhuan sshd[4811]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Oct 13 23:45:21 lhuan sshd[4811]: Failed password for root from 111.230.157.219 port 50858 ssh2
Oct 13 23:45:21 lhuan sshd[4811]: Received disconnect from 111.230.157.219 port 50858:11: Bye Bye [preauth]
Oct 13 23:45:21 lhuan sshd[4811]: Disconnected from 111.230.157.219 port 50858 [preauth]
Oct 13 23:45:42 lhuan sshd[4889]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=118.25.185.106  user=root

查看某个服务产生的日志

[root@lhuan ~]# journalctl  -u nginx.service
-- Logs begin at Sat 2020-02-22 18:16:12 CST, end at Wed 2020-10-14 00:16:06 CST. --
Sep 03 15:20:43 lhuan systemd[1]: Stopping nginx - high performance web server...
Sep 03 15:20:43 lhuan systemd[1]: Stopped nginx - high performance web server.
-- Reboot --
Sep 03 15:21:24 lhuan systemd[1]: Starting nginx - high performance web server...
Sep 03 15:21:25 lhuan nginx[1342]: nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
Sep 03 15:21:25 lhuan systemd[1]: Can't open PID file /var/run/nginx.pid (yet?) after start: No such file or directory
Sep 03 15:21:25 lhuan systemd[1]: Started nginx - high performance web server.

滚动显示某个服务产生的日志

[root@lhuan ~]# journalctl  -u nginx.service -f

利用日志定位问题

查看系统登录日志

[root@lhuan ~]# lastlog
Username         Port     From             Latest
root             pts/1    223.104.145.42   Tue Oct 13 23:40:11 +0800 2020
bin                                        **Never logged in**
daemon                                     **Never logged in**
adm                                        **Never logged in**
lp                                         **Never logged in**
sync                                       **Never logged in**
shutdown                                   **Never logged in**
halt                                       **Never logged in**
mail

查看历史命令

[root@lhuan ~]# history | grep nginx
    2  2020-02-22 00:09:31 vi /etc/yum.repos.d/nginx.repo
    4  2020-02-22 00:12:04 vim /etc/yum.repos.d/nginx.repo
    5  2020-02-22 00:14:12 yum install -y nginx
    6  2020-02-22 00:23:04 vim /etc/nginx/nginx.conf
    7  2020-02-22 00:28:07 vim /etc/nginx/nginx.conf
    8  2020-02-22 00:31:07 vim /etc/nginx/nginx.conf
    9  2020-02-22 00:33:34 systemctl start nginx
   10  2020-02-22 00:33:47 systemctl enable  nginx
   24  2020-02-22 01:53:23 echo "<?php phpinfo(); ?>" >> /usr/share/nginx/html/index.php
   25  2020-02-22 01:53:33 systemctl restart nginx
   32  2020-02-22 14:27:20 rm -rf /usr/share/nginx/html/index.php

查看系统日志

[root@lhuan ~]# head -n  10 /var/log/messages  

Dec 25 21:03:33 VM_0_3_centos rsyslogd: imjournal: journal reloaded... [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/0 ]
Dec 25 21:03:33 VM_0_3_centos systemd: Stopped target Cloud-init target.
Dec 25 21:03:33 VM_0_3_centos ntpd[551]: ntpd exiting on signal 15
Dec 25 21:03:33 VM_0_3_centos systemd: Stopping RPC bind service...
Dec 25 21:03:33 VM_0_3_centos systemd: Stopped Dump dmesg to /var/log/dmesg.
Dec 25 21:03:33 VM_0_3_centos systemd: Stopping Authorization Manager...
Dec 25 21:03:33 VM_0_3_centos acpid: exiting
Dec 25 21:03:33 VM_0_3_centos systemd: Stopped target Multi-User System.
Dec 25 21:03:33 VM_0_3_centos systemd: Stopping Network Time Service...