linux大杂烩 - 云雁项目部署文档 - 《运维大世界》

date: 2020-11-24title: 云雁项目部署文档 #标题
tags: 工作 #标签
categories: 工作大杂烩 # 分类
password: jianzhao87.
abstract: 有东西被加密了, 请输入密码查看。
message: 您好, 这里需要密码。
wrong_pass_message: 抱歉, 这个密码看着不太对, 请再试试。

date: 2020-11-24title: 云雁项目部署文档 #标题
tags: 工作 #标签
categories: 工作大杂烩 # 分类
password: jianzhao87.
abstract: 有东西被加密了, 请输入密码查看。
message: 您好, 这里需要密码。
wrong_pass_message: 抱歉, 这个密码看着不太对, 请再试试。

交维文档，特此记录。

kafka+zookeeper

环境准备

需要三台机器，IP随实际来变，我这里的IP分别为：192.168.20.5/6/10

配置jdk

自行去oracle官网下载java包 jdk-8u261-linux-x64.tar.gz ，然后上传至各个服务器(每台机器都要配置jdk环境)。

$ mkdir /apps/usr -p && yum -y install rsync  && systemctl stop firewalld && systemctl disable firewalld && setenforce 0
$ tar zxf jdk-8u261-linux-x64.tar.gz -C /apps/usr/
$ ln -sf /apps/usr/jdk1.8.0_261 /apps/usr/jdk
$ cat >> /etc/profile << EOF
export JAVA_HOME=/apps/usr/jdk
export CLASSPATH=\$JAVA_HOME/lib
export PATH=\$JAVA_HOME/bin:\$PATH
EOF
$ source /etc/profile
$ java -version      # 查看版本信息
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)

配置zookeeper

以下操作只需在192.168.20.10上执行，然后将修改好的配置文件发送至其他节点。

cd /tmp/
wget http://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar zxf zookeeper-3.4.14.tar.gz -C /apps/usr/
ln -sf /apps/usr/zookeeper-3.4.14 /apps/usr/zk
# 配置
mkdir /apps/usr/zk/data
echo 1 > /apps/usr/zk/data/myid
$ cat /apps/usr/zk/conf/zoo.cfg     # 编写配置文件如下
dataDir=/apps/usr/zk/data
clientPort=2181
maxClientCnxns=0
tickTime=2000
initLimit=10
syncLimit=5
quorumListenOnAllIPs=true
server.1=192.168.20.5:2888:3888
server.2=192.168.20.6:2888:3888
server.3=192.168.20.10:2888:3888
# 注意，server.x ：x表示节点id，必须和实际IP对应上，如果不对应，最后查看集群状态时，大概率会看到如下：
# $ /apps/usr/zk/bin/zkServer.sh status
# ZooKeeper JMX enabled by default
# Using config: /apps/usr/zk/bin/../conf/zoo.cfg
# Error contacting service. It is probably not running.  # 尽管端口都在监听，但状态是没运行
# 现在接着配置
# 将zookeeper目录发送至其他节点
$ cd /apps/usr/
$ for i in `seq 5 6`;do rsync -az /apps/usr/zookeeper-3.4.14 192.168.20.${i}:/apps/usr/;done
# 启动192.168.20.10节点的zookeeper
$ /apps/usr/zk/bin/zkServer.sh start
# 只有leader才会监听28888
ss -lnpt | egrep '2181|3888|2888'

修改192.168.20.5节点的配置文件

ln -sf /apps/usr/zookeeper-3.4.14/ /apps/usr/zk
echo 2 > /apps/usr/zk/data/myid

修改192.168.20.6节点的配置文件

ln -sf /apps/usr/zookeeper-3.4.14/ /apps/usr/zk
echo 3 > /apps/usr/zk/data/myid

执行以下指令启动各个zookeeper节点

$ /apps/usr/zk/bin/zkServer.sh status    # 查看集群状态（应该有一个leader，两个follower）
ZooKeeper JMX enabled by default
Using config: /apps/usr/zk/bin/../conf/zoo.cfg
Mode: follower
$ /apps/usr/zk/bin/zkServer.sh start
ss -lnpt | egrep '2181|3888|2888'      # 确定端口在监听
# 别忘了，只有leader才会监听2888

至此，zookeeper集群部署完成。

部署kafka集群

接下来的操作还是在任意一台进行即可。我这里在192.168.20.10这台机器进行配置。

$ cd /tmp/
$ wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.2/kafka_2.11-2.2.2.tgz
# 解压
$ tar zxf kafka_2.11-2.2.2.tgz -C /apps/usr/
$ ln -sf /apps/usr/kafka_2.11-2.2.2 /apps/usr/kafka
$ cat /apps/usr/kafka/config/server.properties     # kafka配置文件如下
broker.id=0
delete.topic.enable=true
auto.create.topics.enable=false
auto.leader.rebalance.enable=true
default.replication.factor=3
num.replica.fetchers=3
listeners=PLAINTEXT://:9092
num.network.threads=33
num.io.threads=64
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/apps/usr/kafka/logs
num.partitions=10
num.recovery.threads.per.data.dir=3
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=48
#log.retention.bytes=1610612736000
log.retention.bytes=3072000000
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.20.5:2181,192.168.20.6:2181,192.168.20.10:2181
zookeeper.connection.timeout.ms=100000
controlled.shutdown.enable=true
$ mkdir /apps/usr/kafka/logs -p      # 创建所需目录
# 将修改好的kafka目录发送至其他节点
$ cd /apps/usr/
$ for i in 5 6;do rsync -avz kafka_2.11-2.2.2 192.168.20.${i}:/apps/usr/;done

配置其他节点

需要修改其他节点的kafka id号，避免冲突。

配置192.168.20.5

ln -sf /apps/usr/kafka_2.11-2.2.2 /apps/usr/kafka
sed -i 's#broker.id=0#broker.id=1#g' /apps/usr/kafka/config/server.properties

配置192.168.20.6

ln -sf /apps/usr/kafka_2.11-2.2.2 /apps/usr/kafka
sed -i 's#broker.id=0#broker.id=2#g' /apps/usr/kafka/config/server.properties

启动kafka

所有节点执行如下指令，启动kafka集群。

$ cd /apps/usr/kafka/bin/
$ ./kafka-server-start.sh -daemon ../config/server.properties

测试kafka可用性

# 在任意一个节点测试即可
$ cd /apps/usr/kafka
# 创建一个topic（--zookeeper指定任意一个zookeeper节点监听地址即可）
$ bin/kafka-topics.sh --create --zookeeper 192.168.20.5:2181 --replication-factor 1 --partitions 3  --topic test
Created topic test.
# 查看topic列表
$ bin/kafka-topics.sh --zookeeper 192.168.20.5:2181  --list
test
# 查看topic详细信息
$ bin/kafka-topics.sh --zookeeper 192.168.20.5:2181  --topic test --describe
Topic:test    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: test    Partition: 0    Leader: 1    Replicas: 1    Isr: 1
    Topic: test    Partition: 1    Leader: 2    Replicas: 2    Isr: 2
    Topic: test    Partition: 2    Leader: 1    Replicas: 1    Isr: 1

至此，kafka集群部署完成。

部署ElasticSearch集群

环境准备

OS	hostname	IP	role
centos 7.5	es-master	192.168.20.2	master
centos 7.5	es-slave1	192.168.20.3	slave1
centos 7.5	es-slave2	192.168.20.4	slave2
centos 7.5	es-slave3	192.168.20.5	slave3

配置jdk环境

自行去oracle官网下载java包 jdk-8u261-linux-x64.tar.gz ，然后上传至各个服务器(每台机器都要配置jdk环境)。

$ mkdir /apps/usr -p && yum -y install rsync  && systemctl stop firewalld && systemctl disable firewalld && setenforce 0
$ tar zxf jdk-8u261-linux-x64.tar.gz -C /apps/usr/
$ ln -sf /apps/usr/jdk1.8.0_261 /apps/usr/jdk
$ cat >> /etc/profile << EOF
export JAVA_HOME=/apps/usr/jdk
export CLASSPATH=\$JAVA_HOME/lib
export PATH=\$JAVA_HOME/bin:\$PATH
EOF
$ source /etc/profile
$ java -version      # 查看版本信息
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)

配置es-master

以下操作在es-master （192.168.20.2）上执行。

部署ES

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.2.tar.gz
$ tar zxf elasticsearch-6.6.2.tar.gz -C /apps/usr/
$ ln -sf /apps/usr/elasticsearch-6.6.2 /apps/usr/es-master
cat > /apps/usr/es-master/config/elasticsearch.yml << EOF
cluster.name: yypt-cluster
node.master: true
node.data: false
node.ingest: true
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
node.name: master
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
network.host: 192.168.20.2
discovery.zen.ping.unicast.hosts: ["192.168.20.2:9300"]
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
EOF
# 部分配置解释如下：
# cluster.name：自定义，参与集群的节点cluster.name 必须一致
# node.name： 自定义，参与集群的节点 node.name 不可冲突
# network.host：指定本机IP，不可以写 0.0.0.0
# discovery.zen.ping.unicast.hosts：指定master节点的IP+9300端口
# discovery.zen.minimum_master_nodes：master节点数量
# node.max_local_storage_nodes：存储节点数量

优化ES

修改jvm虚拟机使用内存。

$ vim /apps/usr/es-master/config/jvm.options
-Xms1g
-Xmx1g
# 将Xms和Xmx两个值设置为一致(一般为物理内存的50%)

将修改后的es目录发送至其他节点

$ for i in $(seq 3 5);do rsync -az /apps/usr/elasticsearch-6.6.2 192.168.20.${i}:/apps/usr/;done

配置es-slave1

$ ln -sf /apps/usr/elasticsearch-6.6.2 /apps/usr/es-slave1
$ cat > /apps/usr/es-slave1/config/elasticsearch.yml << EOF
cluster.name: yypt-cluster
node.master: false
node.data: true
node.ingest: false
http.port: 9200
transport.tcp.port: 9300
node.name: slave1
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
network.host: 192.168.20.3
discovery.zen.ping.unicast.hosts: ["192.168.20.2:9300"]
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
EOF
# 部分配置解释如下：
# cluster.name：自定义，参与集群的节点cluster.name 必须一致
# node.name： 自定义，参与集群的节点 node.name 不可冲突
# network.host：指定本机IP，不可以写 0.0.0.0
# discovery.zen.ping.unicast.hosts：指定master节点的IP+9300端口
# discovery.zen.minimum_master_nodes：master节点数量
# node.max_local_storage_nodes：存储节点数量

配置es-slave2

$ ln -sf /apps/usr/elasticsearch-6.6.2 /apps/usr/es-slave2
$ cat > /apps/usr/es-slave2/config/elasticsearch.yml << EOF
cluster.name: yypt-cluster
node.master: false
node.data: true
node.ingest: false
http.port: 9200
transport.tcp.port: 9300
node.name: slave2
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
network.host: 192.168.20.4
discovery.zen.ping.unicast.hosts: ["192.168.20.2:9300"]
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
EOF
# 部分配置解释如下：
# cluster.name：自定义，参与集群的节点cluster.name 必须一致
# node.name： 自定义，参与集群的节点 node.name 不可冲突
# network.host：指定本机IP，不可以写 0.0.0.0
# discovery.zen.ping.unicast.hosts：指定master节点的IP+9300端口
# discovery.zen.minimum_master_nodes：master节点数量
# node.max_local_storage_nodes：存储节点数量

配置es-slave3

$ ln -sf /apps/usr/elasticsearch-6.6.2 /apps/usr/es-slave3
$ cat > /apps/usr/es-slave3/config/elasticsearch.yml << EOF
cluster.name: yypt-cluster
node.master: false
node.data: true
node.ingest: false
http.port: 9200
transport.tcp.port: 9300
node.name: slave3
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
network.host: 192.168.20.5
discovery.zen.ping.unicast.hosts: ["192.168.20.2:9300"]
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
EOF

# 部分配置解释如下：
# cluster.name：自定义，参与集群的节点cluster.name 必须一致
# node.name： 自定义，参与集群的节点 node.name 不可冲突
# network.host：指定本机IP，不可以写 0.0.0.0
# discovery.zen.ping.unicast.hosts：指定master节点的IP+9300端口
# discovery.zen.minimum_master_nodes：master节点数量
# node.max_local_storage_nodes：存储节点数量

针对服务器优化操作

创建相应的目录

创建你在es配置文件中指定的数据目录及日志目录。所有机器上必须做。

$ mkdir -p /data/elasticsearch/{data,logs}

创建es启动用户

$ useradd es && echo 'i8h2f!zxZH98' | passwd --stdin es
chown -R es.es /apps/usr/elasticsearch-6.6.2  /data/elasticsearch

配置虚拟内存（官方推荐）

es默认使用mmapfs目录来存储索引。mmap计数的默认操作系统限制可能太低，这可能导致内存异常。

$ cat >> /etc/sysctl.conf << EOF
vm.max_map_count=262144
vm.swappiness = 0
EOF


$ sysctl -p        # 刷新配置
vm.max_map_count = 262144
vm.swappiness = 0

修改打开文件数、进程数等限制

$ mv /etc/security/limits.conf{,.bak}

$ cat >> /etc/security/limits.conf  << EOF
*                -       nofile          655360
*                -       memlock         unlimited
*                -       stack           655360
*                -       nproc           unlimited
EOF

启动es集群

# 启动master节点
$ su -s /bin/bash -c "/apps/usr/es-master/bin/elasticsearch -d" es

$ ss -lnput | grep 9.00      # 确定端口在监听
tcp    LISTEN     0      128       [::ffff:192.168.20.2]:9200               [::]:*                   users:(("java",pid=23199,fd=205))
tcp    LISTEN     0      128       [::ffff:192.168.20.2]:9300               [::]:*                   users:(("java",pid=23199,fd=191))


# 启动slave1节点
$ su -s /bin/bash -c "/apps/usr/es-slave1/bin/elasticsearch -d" es

$ ss -lnptu | grep 9.00      # 确定端口在监听
tcp    LISTEN     0      128       [::ffff:192.168.20.3]:9200               [::]:*                   users:(("java",pid=22686,fd=227))
tcp    LISTEN     0      128       [::ffff:192.168.20.3]:9300               [::]:*                   users:(("java",pid=22686,fd=191))



# 启动slave2节点
$ su -s /bin/bash -c "/apps/usr/es-slave2/bin/elasticsearch -d" es
$ ss -lnptu | grep 9.00      # 确定端口在监听
tcp    LISTEN     0      128       [::ffff:192.168.20.4]:9200               [::]:*                   users:(("java",pid=22293,fd=251))
tcp    LISTEN     0      128       [::ffff:192.168.20.4]:9300               [::]:*                   users:(("java",pid=22293,fd=191))

# 启动slave3节点
$ su -s /bin/bash -c "/apps/usr/es-slave3/bin/elasticsearch -d" es

$ ss -lnptu | grep 9.00      # 确定端口在监听
tcp    LISTEN     0      128       [::ffff:192.168.20.5]:9200               [::]:*                   users:(("java",pid=31353,fd=275))
tcp    LISTEN     0      128       [::ffff:192.168.20.5]:9300               [::]:*                   users:(("java",pid=31353,fd=191))

验证集群

# 访问各个节点的9200端口，对比cluster_uuid一致即表示集群没有问题
$ for i in $(seq 2 5);do curl 192.168.20.${i}:9200 -s | grep cluster_uuid;done
  "cluster_uuid" : "y5PJVhmGQrmEDcHWZPaCng",
  "cluster_uuid" : "y5PJVhmGQrmEDcHWZPaCng",
  "cluster_uuid" : "y5PJVhmGQrmEDcHWZPaCng",
  "cluster_uuid" : "y5PJVhmGQrmEDcHWZPaCng",

部署k8s 1.13.4集群

环境准备

OS	IP	hostname
Centos 7.5	192.168.20.2	master
Centos 7.5	192.168.20.3	node01
Centos 7.5	192.168.20.4	node02
Centos 7.5	192.168.20.5	node03

注：上述主机名即为集群中的角色，也就是说192.168.20.2为k8s中的master节点，其余为k8s中的node节点。每个主机，最少2G内存，CPU不得低于2核心。

安装前准备

接下来的操作，若无特殊说明，均需在所有节点上执行。

修改hostname，并配置解析

$ hostnamectl set-hostname your-new-host-name
$ echo "127.0.0.1   $(hostname)" >> /etc/hosts

检查网络

必须保证所有节点上 Kubernetes 所使用的 IP 地址必须可以互通（无需 NAT 映射、无安全组或防火墙隔离）

是否安装k8s集群的硬性要求

必须满足以下几个硬性要求：

cpu核心数不低于2，内存不低于2G。
hostname不是localhost，且不包含下划线、小数点、大写字母。
任意节点都有固定的内网IP地址。
任意节点上kubelet使用的IP地址在同一局域网内，没有网络策略隔离。
任意节点不会直接使用 docker run 或 docker-compose 运行容器。

安装docker 18.06

# 卸载旧版本docker
$ yum remove -y docker \
docker-client \
docker-client-latest \
docker-ce-cli \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine

# 设置 yum repository
$ yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# 安装 docker
$ yum -y install docker-ce-18.06.0.ce-3.el7
# 修改 docker 镜像下载源
$ [ -d /etc/docker ] || mkdir /etc/docker
echo "{\"registry-mirrors\": [\"https://registry.cn-hangzhou.aliyuncs.com\"]}" >> /etc/docker/daemon.json

# 修改docker的根目录
$ mkdir -p /etc/systemd/system/docker.service.d
$ cat > /etc/systemd/system/docker.service.d/devicemapper.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd  --graph=/data/lib/docker
EOF


# 启动docker
$ systemctl enable docker && systemctl start docker

# 确认 docker 版本及镜像下载地址
$ docker version
$ docker info | grep -A1 'Registry Mirrors:'
 Registry Mirrors:
  https://registry.cn-hangzhou.aliyuncs.com/

调整系统其他配置

# 安装nfs-utils
$ yum install -y nfs-utils wget

# 关闭防火墙
$ systemctl stop firewalld
systemctl disable firewalld

# 关闭SeLinux
$ setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config


# 禁用swap分区
$ swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab


# 调整内核参数
# 如果有配置，则修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.disable_ipv6.*#net.ipv6.conf.all.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.default.disable_ipv6.*#net.ipv6.conf.default.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.lo.disable_ipv6.*#net.ipv6.conf.lo.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.forwarding.*#net.ipv6.conf.all.forwarding=1#g"  /etc/sysctl.conf

# 如果没有，则追加
cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.all.forwarding = 1
EOF


# 执行命令以应用
$ sysctl -p

部署k8s

配置k8s的yum源

$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

卸载旧版本

$ yum remove -y kubelet kubeadm kubectl

安装kubelet、kubeadm、kubectl

$ yum install -y kubelet-1.13.4  kubectl-1.13.4 kubeadm-1.13.4 kubernetes-cni-0.6.0

启动kubelet

systemctl daemon-reload
systemctl enable kubelet 
systemctl start kubelet

注：如果此时执行systemctl status kubelet命令，会得到 kubelet 启动失败的错误提示，请忽略此错误，因为必须完成后续步骤中 kubeadm init 的操作，kubelet 才能正常启动。

错误如下：

云雁项目部署文档 - 图1

初始化master节点

注：接下来的操作，只需要在master节点上进行配置。

执行初始化master

$ kubeadm init --kubernetes-version=v1.13.4 \
--image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
--service-cidr=10.10.0.0/16 \
--pod-network-cidr=10.20.0.0/16 --apiserver-advertise-address=0.0.0.0

配置kubectl

rm -rf /root/.kube/
mkdir /root/.kube/
cp -i /etc/kubernetes/admin.conf /root/.kube/config

安装flannel网络插件

$ cat > flannel.yaml << EOF
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.20.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: registry.cn-shenzhen.aliyuncs.com/cp_m/flannel:v0.10.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: registry.cn-shenzhen.aliyuncs.com/cp_m/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
EOF

# 创建flannel网络
$ kubectl apply -f flannel.yaml

初始化worker节点

获取join命令参数

# 在 master 节点上执行
$ kubeadm token create --print-join-command

# 输出如下：
# 我们需要的就是下面kubeadm join指令，每个集群初始化出来的指令都不一样，请复制自己master输出的指令
kubeadm join 192.168.20.2:6443 --token vidxau.rlm1lhne3kam2cuk --discovery-token-ca-cert-hash sha256:e66891446ae7335434fb6b54fec4f87583f55441b411a7401c4e1bca2e7d0ded

# 该 token 的有效时间为 2 个小时，2小时内，您可以使用此 token 初始化任意数量的 worker 节点。

初始化worker节点

所有worker节点都执行下述指令，以便加入k8s集群。

# 只在 worker 节点执行
kubeadm join 192.168.20.2:6443 --token vidxau.rlm1lhne3kam2cuk --discovery-token-ca-cert-hash sha256:e66891446ae7335434fb6b54fec4f87583f55441b411a7401c4e1bca2e7d0ded

最终，在master上查看所有pod状态，全为running即可，如下：

kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-89cc84847-gplkm            1/1     Running   0          9m51s
kube-system   coredns-89cc84847-j4clc            1/1     Running   0          9m51s
kube-system   etcd-master01                      1/1     Running   0          9m
kube-system   kube-apiserver-master01            1/1     Running   0          8m58s
kube-system   kube-controller-manager-master01   1/1     Running   0          8m58s
kube-system   kube-flannel-ds-amd64-8gldg        1/1     Running   1          3m23s
kube-system   kube-flannel-ds-amd64-9wlqg        1/1     Running   0          4m59s
kube-system   kube-flannel-ds-amd64-hzfqm        1/1     Running   1          3m25s
kube-system   kube-flannel-ds-amd64-kpjzs        1/1     Running   0          7m27s
kube-system   kube-proxy-fc94p                   1/1     Running   0          9m51s
kube-system   kube-proxy-gnwf4                   1/1     Running   0          3m25s
kube-system   kube-proxy-stmlc                   1/1     Running   0          3m23s
kube-system   kube-proxy-x7mm7                   1/1     Running   0          4m59s
kube-system   kube-scheduler-master01            1/1     Running   0          8m48s

查看node状态全为Ready：

$ kubectl get nodes
NAME       STATUS   ROLES    AGE     VERSION
master01   Ready    master   9m57s   v1.13.4
node01     Ready    <none>   4m46s   v1.13.4
node02     Ready    <none>   3m10s   v1.13.4
node03     Ready    <none>   3m12s   v1.13.4

部署k8s管理工具——kuboard

兼容性如下：

Kubernetes 版本	Kuboard 版本	兼容性	说明
v1.18	v1.0.x，v2.0.x	😄	已验证
v1.17	v1.0.x，v2.0.x	😄	已验证
v1.16	v1.0.x，v2.0.x	😄	已验证
v1.15	v1.0.x，v2.0.x	😄	已验证
v1.14	v1.0.x，v2.0.x	😄	已验证
v1.13	v1.0.x，v2.0.x	😄	已验证
v1.12	v1.0.x，v2.0.x	😐	Kuboard 不支持 Kubernetes v1.12
v1.11	v1.0.x，v2.0.x	😐	Kuboard 不支持 Kubernetes v1.11

安装kuboard

# 安装
$ kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml
$ kubectl apply -f https://addons.kuboard.cn/metrics-server/0.3.6/metrics-server.yaml

# 查看kuboard运行状态,直到状态为running
$ watch kubectl get pods -l k8s.kuboard.cn/name=kuboard -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE
kuboard-7bb89b4cc4-99l9n   1/1     Running   0          2m22s

获取Token

Token有两种类型，分别是管理员用户、只读用户的Token，这里将分别写下两种token的查看方式。

获取管理员用户的Token

此Token拥有 ClusterAdmin 的权限，可以执行所有操作。

# 执行命令如下
$ echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-user | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d)

# 输出如下（复制输出内容）
eyJhbGciOiJSUzI1NiIsImtpZCI6IllWaUVoOXZtUUNlS25JaVpPZVhReGhnNzZjcExzSFVfZy1ycGJiZkp0ZEUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJvYXJkLXVzZXItdG9rZW4tajRidHgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoia3Vib2FyZC11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjMxNDM3ODQtMWM4My00NzRiLWExMGUtMDE0ZjY4MDA1NGUyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmt1Ym9hcmQtdXNlciJ9.i6z0pUpyYdBlfwkMEH4FTjsz550J3fGn_e9uNrke_TpLasvvQlqfwQ8_kUOrjXyztW42ANcTZTVZ2z8lE7tsf53FpKwDnhZ3Swt0pcC018Cm-4q0YhkHqJlh6l4c6DiPqqPetYfvOOfXee-f9EbuQVCMVLqDfqZKwDM8Qa3gt81tk4oFwlDfahNdUmL0UQXXm_2LXQHqcXJ5slc7V1NDngou8aSxA1GgfFsQB8SxIjYlatD5mG_ZyIpX9gPZxiDWgOJDCogwIhrJGa2vDPnhZSayqJwZrIvsS2WLejeh5w7D0p6ZdNg53R_pPjOt5NLgeJZQCF3pvvo-touWNbnTPA

获取只读用户的Token

只读用户拥有的权限如下：

view 可查看名称空间的内容
system:node 可查看节点信息
system:persistent-volume-provisioner 可查看存储类和存储卷声明的信息

适用场景：

只读用户不能对集群的配置执行修改操作，非常适用于将开发环境中的 Kuboard 只读权限分发给开发者，以便开发者可以便捷地诊断问题。

# 执行命令如下
$ echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-viewer | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d)

# 复制输出的Token信息
eyJhbGciOiJSUzI1NiIsImtpZCI6IllWaUVoOXZtUUNlS25JaVpPZVhReGhnNzZjcExzSFVfZy1ycGJiZkp0ZEUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJvYXJkLXZpZXdlci10b2tlbi1tNzRkbCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJvYXJkLXZpZXdlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNjMTI0NmExLTgzZTktNGZlZC1hNGQ2LTRkMjYzNzQ1YjBjZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJvYXJkLXZpZXdlciJ9.jpuVX2aYmpycjcXxTnMCnJre_F10OHiMZPgcOxnPJ3B_5ZeDHzj__06MCn2IF0x8yZZmJIqqr57HIWU5eGKk1jqxuC1OHCQGcG4RGL0e0hCcW-MW2nRT6YMWTiqGMGQ9Ooydt3pLaTEdJAg0ZnMXY5eoBIk9TX_UzoQASH0kkBnjqFDwaJ8-O6bybVB1cNQIlUug2mSgY2sWn-JvyA9doeq7qjS5mXrO_rGbGwUR2HL_KWNeFd9Ha_GWpnSHMdARMTCgDHiV_sznRFG72puWgg-J_2oSEC8pM8-VpBRSxzHbgEI5-ANP2fwjpcJDT8SdBHC5XntGT3ctA66de2T8vQ

访问Kuboard

Kuboard Service 使用了 NodePort 的方式暴露服务，NodePort 为 32567；可以按如下方式访问 Kuboard：

http://任意一个Worker节点的IP地址:32567/

访问登录页面：

云雁项目部署文档 - 图2

登录后的首页如下：

云雁项目部署文档 - 图3

不要以为就这些东西，很多地方都是可以点击进去的，如下：

云雁项目部署文档 - 图4

云雁项目部署文档 - 图5

部署traefik

镜像文件提取链接如下：
链接：https://pan.baidu.com/s/1lTUYouhnnzRulFjmbNF4PQ
提取码：1233

上传traefik镜像文件至服务器，然后运行如下指令：

$ docker load < traefik.tgz

# 编写yaml文件，注意：yaml文件中部署daemonset资源对象那段，需要更改下pod的亲和度，以便可以运行在指定机器上
# 具体请看yaml文件中的注释
$ cat > traefix.yaml << EOF
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-config
  namespace: kube-system
data:
  traefik.toml: |-
      defaultEntryPoints = ["http", "https"]
      [api]
      [kubernetes]
      logLevel = "DEBUG"
      [entryPoints]
        [entryPoints.http]
        address = ":80"

        [entryPoints.https]
        address = ":443"
          [entryPoints.https.tls]
             [[entryPoints.https.tls.certificates]]
             certFile = "/ssl/3777538/tls.crt"
             keyFile = "/ssl/3777538/tls.key"
      [traefikLog]
        filePath = "/data/traefik.log"
        format   = "json"
      [accessLog]
        filePath = "/data/access.log"
        format = "json"
        [accessLog.filters]
          #statusCodes = ["200", "300-302"]
          retryAttempts = true
          minDuration = "10ms"
        [accessLog.fields]
          defaultMode = "keep"
          [accessLog.fields.names]
            "ClientUsername" = "drop"
            "TESTDATE" = "StartUTC"
            # ...
          [accessLog.fields.headers]
            defaultMode = "keep"
            [accessLog.fields.headers.names]
              "User-Agent" = "redact"
              "Authorization" = "drop"
              "Content-Type" = "keep"
              # ...  


---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: hub.i139.cn/rdc-commons/official-traefik:v1.7.24
        name: traefik-ingress-lb
        resources:
          limits:
            cpu: 2
            memory: 2000Mi
          requests:
            cpu: 100m
            memory: 200Mi
        env:
        - name: TZ
          value: Asia/shanghai
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443          
        - name: admin
          containerPort: 8080
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        volumeMounts:
          - mountPath: /config
            name: traefik-config
#          - mountPath: /ssl/3777538
#            name: ssl-cert
          - mountPath: /data
            name: traefik-log
          - name: tz-config
            mountPath: /etc/localtime
            readOnly: true
        args:
        - --configFile=/config/traefik.toml
      volumes:
        - name: traefik-config
          configMap:
            name: traefik-config
#        - name: ssl-cert
#          secret:
#            secretName: asp-jkcredit-com
        - name: traefik-log
          hostPath:
            path: /data/logs/traefik
        - name: tz-config
          hostPath:
            path: /etc/localtime
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
#      nodeSelector:
#        IngressProxy: "true"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:      # 这里的value值，需要更改为你想k8s集群中的主机名，你想让它运行在哪些机器上，就写上哪个机器的主机名即可
                - node01
                - master01

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
    - extensions
    resources:
    - ingresses/status
    verbs:
    - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  annotations:
    kubernetes.io/ingress.class: traefik
    #traefik.ingress.kubernetes.io/whitelist-source-range: "192.168.168.168/32"
    #traefik.ingress.kubernetes.io/auth-type: "basic"
    #traefik.ingress.kubernetes.io/auth-secret: "traefik-ingress-admin-ui-secret"
  namespace: kube-system
spec:
  rules:
  - host: asp.jkcredit.com
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web
EOF


$ kubectl apply -f traefix.yaml
$ kubectl get pods -n kube-system | grep traefik     # 确定pod运行成功
traefik-ingress-controller-8wrwx   1/1     Running   0          14m
traefik-ingress-controller-nnqfj   1/1     Running   0          14m

访问staefik的web UI界面（windows机器需要写hosts解析记录，格式为：“k8s节点IP asp.jkcredit.com”）：

云雁项目部署文档 - 图6

云雁项目部署文档

date: 2020-11-24title: 云雁项目部署文档 #标题tags: 工作 #标签categories: 工作大杂烩 # 分类password: jianzhao87.abstract: 有东西被加密了, 请输入密码查看。message: 您好, 这里需要密码。wrong_pass_message: 抱歉, 这个密码看着不太对, 请再试试。

kafka+zookeeper

环境准备

配置jdk

配置zookeeper

部署kafka集群

部署ElasticSearch集群

环境准备

配置jdk环境

配置es-master

配置es-slave1

配置es-slave2

配置es-slave3

针对服务器优化操作

启动es集群

验证集群

部署k8s 1.13.4集群

环境准备

安装前准备

修改hostname，并配置解析

检查网络

是否安装k8s集群的硬性要求

安装docker 18.06

调整系统其他配置

部署k8s

配置k8s的yum源

卸载旧版本

安装kubelet、kubeadm、kubectl

启动kubelet

初始化master节点

执行初始化master

配置kubectl

安装flannel网络插件

初始化worker节点

获取join命令参数

初始化worker节点

部署k8s管理工具——kuboard

安装kuboard

获取Token

获取管理员用户的Token

获取只读用户的Token

访问Kuboard

部署traefik