日志系统 - FELK - 《运维笔记 - 教程》

01
02
03
04 ES安装
05 logstash安装

01

环境说明：
centos7
关闭iptables(systemctl stop firewalld, systemctl disable firewalld)
关闭selinux
yum install net-tools vim -y

1: 讲解的流程
先讲elk： logstash -> elasticsearch -> kibana #elk搭建讲解
引入filebeat讲解：filebeat -> elasticsearch -> kibana #轻客户端
filebeat + logstash: filebeat -> logstash -> elasticsearch -> kibana #轻客户端+正则
引入redis: filebeat -> redis(缓存) -> logstash -> elasticsearch -> kibana #f引入缓存

2: yum安装jdk1.8 Java 8 is required for Logstash 6.x and 5.x.(https://www.elastic.co/downloads/logstash)
yum install java-1.8.0-openjdk -y
elasticsearch、logstash依赖于java环境

3: 使用官方的二进制包解压安装
先下载java linux 64位tar.gz包
java 1.8的下载链接： http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

4: 解压，移动(即安装)
cd /usr/local/src/
tar -zxvf jdk-8u162-linux-x64.tar.gz
mv jdk1.8.0_162 /usr/local/
用全路径验证java是否安装成功
/usr/local/jdk1.8.0_162/bin/java -version

5: 配置java环境变量
vim /etc/profile加入
export JAVA_HOME=/usr/local/jdk1.8.0_162/
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
#环境变量生效
source /etc/profile

6: java版本查看
java -version

02

1: kibana下载地址(kibana主要用来展现数据，它本身不存储数据) https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz

2: 准备工作，添加elk用户，用elk用户来启动elk
useradd elk
usermod -s /sbin/nologin elk #不让elk用户来登录系统
解压安装kibana:
tar -zxf kibana-6.2.3-linux-x86_64.tar.gz
mv kibana-6.2.3-linux-x86_64 /usr/local/kibana-6.2.3

3: kibana配置文件
vim /usr/local/kibana-6.2.3/config/kibana.yml修改：
server.port: 5601
server.host: “0.0.0.0”(监听在所有网卡，有风险)
#elasticsearch.url: “http://localhost:9200“ (默认是连接elasticsearch的9200端口)
#elasticsearch.username: “user” (配置连接elasticsearch的用户名和密码)
#elasticsearch.password: “pass”

4: 把kibana目录改为elk用户
chown -R elk:elk /usr/local/kibana-6.2.3/

5: 新增启动脚本vim /usr/local/kibana-6.2.3/bin/start.sh
nohup /usr/local/kibana-6.2.3/bin/kibana >>/tmp/kibana.log 2>>/tmp/kibana.log &
chmod a+x /usr/local/kibana-6.2.3/bin/start.sh

6: 用普通用户启动
su -s /bin/bash elk ‘/usr/local/kibana-6.2.3/bin/start.sh’
netstat -tulnp查看端口
访问kibana，如有防火墙需要放开tcp 5601端口

03

默认的kibana是没有任何的权限控制，先把kibana改到监听127.0.0.1，借助nginx来限制访问
1:借助nginx来限制访问，控制源ip的访问
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main ‘$remote_addr - $remote_user [$time_local] “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
‘“$http_user_agent” “$http_x_forwarded_for”‘;
sendfile on;
keepalive_timeout 65;
server {
listen 5609; #访问端口变为5609
access_log /usr/local/nginx/logs/kibana_access.log main;
error_log /usr/local/nginx/logs/kibana_error.log error;
location / {
allow 127.0.0.1;
deny all;
proxy_pass http://127.0.0.1:5601;
}
}
}
可以在日志里面找到源ip地址：tail -f /usr/local/nginx/logs/kibana_access.log

再把源ip地址加入到白名单中
location / {
allow 127.0.0.1;
allow ip;
deny all;
proxy_pass http://127.0.0.1:5601;
}
访问kibana，http://ip:5609

2: 如果ip经常变化，就会很麻烦。nginx支持简单的用户名密码认证。
location / {
auth_basic “elk auth”;
auth_basic_user_file /usr/local/nginx/conf/htpasswd;
proxy_pass http://127.0.0.1:5601;
}
设置访问的用户为elk，密码为elkpass（加密过的）
printf “elk:$(openssl passwd -1 elkpass)\n” >/usr/local/nginx/conf/htpasswd

3: nginx源码编译安装脚本
if [ -d “/usr/local/nginx/“ ];then
echo “nginx is install”
exit 1
else
echo “nginx in not install”
fi
for softpack in wget tar gcc gcc-c++ make pcre pcre-devel zlib zlib-devel openssl openssl-devel;do
soft_result=rpm -qa $softpack
if [ -z “$soft_result” ];then
echo “${softpack} is not exist,install it”
yum -y install ${softpack}
else
echo “${softpack} is exist”
fi
done

cd /usr/local/src wget ‘http://nginx.org/download/nginx-1.12.2.tar.gz‘
tar -zxvf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure —prefix=/usr/local/nginx —with-http_ssl_module —with-stream —with-stream_ssl_module
make
make install
exit 0
Nginx链接环境
ln -sf /usr/local/nginx/sbin/nginx /usr/local/bin/

04 ES安装

elasticsearch未安装之前，kibana网页上报错，提示找不到elasticsearch。
1: elasticsearch的下载地址(elasticsearch主要用来存储数据，供kibana调取并进行展现) https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz
解压安装：
cd /usr/local/src/
tar -zxf elasticsearch-6.2.3.tar.gz
mv elasticsearch-6.2.3 /usr/local/

2: elasticsearch配置
vim /usr/local/elasticsearch-6.2.3/config/elasticsearch.yml 修改：
path.data: /usr/local/elasticsearch-6.2.3/data
path.logs: /usr/local/elasticsearch-6.2.3/logs
network.host: 127.0.0.1 #监听本机，先监听内网
http.port: 9200

3: 把elasticsearch目录的用户和属主都更新为elk
chown -R elk:elk /usr/local/elasticsearch-6.2.3/

4: 更改jvm的内存限制(看个人配置)
vim /usr/local/elasticsearch-6.2.3/config/jvm.options
-Xms100M
-Xmx100M

5: 编辑elasticsearch启动脚本，使用-d进行后台启动。elasticsearch
vim /usr/local/elasticsearch-6.2.3/bin/start.sh
/usr/local/elasticsearch-6.2.3/bin/elasticsearch -d
chmod a+x /usr/local/elasticsearch-6.2.3/bin/start.sh

6: 启动elasticsearch
su -s /bin/bash elk ‘/usr/local/elasticsearch-6.2.3/bin/start.sh’
观察日志
观察kibana网页，看下还会不会报elasticsearch的错误

7: elasticsearch如果监听在非127.0.0.1，需要配置内核参数等
vim /usr/local/elasticsearch-6.2.3/config/elasticsearch.yml
network.host: 0.0.0.0 #监听外网
vim /etc/security/limits.conf
(处理max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536])
soft nofile 65536
hard nofile 65536

vim /etc/security/limits.d/20-nproc.conf
(处理max number of threads [3885] for user [elk] is too low, increase to at least [4096])
soft nproc 10240
hard nproc 10240
sysctl.conf添加
(处理max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144])
vm.max_map_count = 262144 #需要运行sysctl -p生效

访问elasticsearch：
https://ip:9092

8: 由于讲课只有一台机器，就监听在127.0.0.1比较安全点。如果有内网又有外网的话，建议是监听在内网

05 logstash安装

1: logstash的下载地址(用来读取日志，正则分析日志，发送给elasticsearch数据库) https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz
解压安装：
tar -zxf logstash-6.2.3.tar.gz
mv logstash-6.2.3 /usr/local/
ll -h /usr/local/logstash-6.2.3
2: 更改logstash jvm配置vim /usr/local/logstash-6.2.3/config/jvm.options
-Xms150M
-Xmx150M
3: logstash配置 vim /usr/local/logstash-6.2.3/config/logstash.conf (新建的)
input {
file {
path => “/usr/local/nginx/logs/kibana_access.log” #读取这个日子文件
}
}
output {
elasticsearch {
hosts => [“http://127.0.0.1:9200“] #将读取的日志文件发送给elasticsearch
}
}
4: logstash的启动脚本：
vim /usr/local/logstash-6.2.3/bin/start.sh
nohup /usr/local/logstash-6.2.3/bin/logstash -f /usr/local/logstash-6.2.3/config/logstash.conf >>/tmp/logstash.log 2>>/tmp/logstash.log &
chmod a+x /usr/local/logstash-6.2.3/bin/start.sh
5: 启动logstash
/usr/local/logstash-6.2.3/bin/start.sh
启动成功
有图。。没上传上来
logstash的启动时间会有点慢，等启动过后查看kibana的界面，会有可以创建索引的地方。