1. 组件介绍
1.1. 微服务
1.1.1. 介绍
微服务 (Microservices) 是一种软件架构风格，它是以专注于单一责任与功能的小型功能区块 (Small Building Blocks) 为基础，利用模块化的方式组合出复杂的大型应用程序，各功能区块使用与语言无关 (Language-Independent/Language agnostic) 的 API 集相互通信。微服务的优点：应用解耦，可以单独升级，甚至不同模块之间可以编程语言不同；横向扩展方便，高可用。缺点：架构复杂，运维成本高
1.1.2. Dubbo微服务框架
阿里巴巴在2011年开源了Dubbo框架，虽然在2013年停止更新，但在2017年9月又重启维护并发布了新版本。目前已有很多的公司将自己的业务建立在Dubbo之上，同时阿里云也推出了企业级分布式应用服务EDAS，为Dubbo提供应用托管。Dubbo采用Zookeeper作为注册中心，RPC作为服务调用方式，致力于提供高性能和透明化的RPC远程服务调用方案。
- Provider：服务提供方发布服务到服务注册中心
- Consumer：服务消费方从服务注册中心订阅服务
- Registry：注册中心通知消息调用方服务已注册
- Monitor：监控计数
- 服务消费方调用已经注册的可用服务(RPC invoke)
1.2. Devops流水线
1.2.1. Jenkins
Jenkins是一款由Java编写的开源的持续集成工具。它运行在Servlet容器中（例如Apache Tomcat）。它支持软件配置管理（SCM）工具（包括AccuRev SCM、CVS、Subversion、Git、Perforce、Clearcase和RTC），可以执行基于Apache Ant和Apache Maven的项目，以及任意的Shell脚本和Windows批处理命令。Jenkins是一个有状态的应用，如果托管在K8S中，只运行一个副本，且需要持久化存储。
官网：https://jenkins.io/zh/
下载地址：https://jenkins.io/zh/download/
官方文档：https://jenkins.io/zh/doc/
1.2.2. Maven
Apache Maven，是一个软件（特别是Java软件）项目管理及自动构建工具，由Apache软件基金会所提供。Maven也可被用于构建和管理各种项目，例如C#，Ruby，Scala和其他语言编写的项目。Maven曾是Jakarta项目的子项目，现为由Apache软件基金会主持的独立Apache项目。在Devops流水线中，参与Java程序构建工具常用Maven。
官网：https://maven.apache.org/index.html
下载地址：https://archive.apache.org/dist/maven/
基础教程：https://www.runoob.com/maven/maven-setup.html
1.3. 实验拓扑图
2. ZK集群部署
ZK集群是有状态的服务，其选择Leader的方式和ETCD类似，要求集群节点是不低于3的奇数个。
| 主机 | IP地址 | 角色 |
|---|---|---|
| hdss7-11 | 10.4.7.11 | zk1 |
| hdss7-12 | 10.4.7.12 | zk2 |
| hdss7-21 | 10.4.7.21 | zk3 |
2.1. 安装JDK
[root@hdss7-12 ~]# cd /opt/src/[root@hdss7-12 src]# tar -xf jdk-8u241-linux-x64.tar.gz -C /opt/release/[root@hdss7-12 src]# ln -s /opt/release/jdk1.8.0_241 /opt/apps/jdk[root@hdss7-12 src]# ls /opt/apps/jdk -llrwxrwxrwx 1 root root 25 Jan 31 11:36 /opt/apps/jdk -> /opt/release/jdk1.8.0_241[root@hdss7-12 src]# vim /etc/profile.d/java.shexport JAVA_HOME=/opt/apps/jdkexport PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar[root@hdss7-12 src]# source /etc/profile.d/java.sh[root@hdss7-12 src]# java -versionjava version "1.8.0_241"Java(TM) SE Runtime Environment (build 1.8.0_241-b07)Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)
2.2. 配置DNS解析
[root@hdss7-11 src]# vim /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
2020011305 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
monitor.api A 10.4.7.10
zk1 A 10.4.7.11
zk2 A 10.4.7.12
zk3 A 10.4.7.21
[root@hdss7-11 src]# systemctl restart named
[root@hdss7-11 src]# dig -t A zk3.od.com @10.4.7.11 +short
10.4.7.21
2.3. 安装ZK
[root@hdss7-12 src]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
[root@hdss7-12 src]# tar -xf zookeeper-3.4.14.tar.gz -C /opt/release/
[root@hdss7-12 src]# ln -s /opt/release/zookeeper-3.4.14 /opt/apps/zookeeper
[root@hdss7-12 src]# ll /opt/apps/zookeeper
lrwxrwxrwx 1 root root 29 Jan 31 11:52 /opt/apps/zookeeper -> /opt/release/zookeeper-3.4.14
[root@hdss7-12 src]# mkdir -p /data/zookeeper/data /data/zookeeper/logs
[root@hdss7-12 src]# vim /opt/apps/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zk1.od.com:2888:3888
server.2=zk2.od.com:2888:3888
server.3=zk3.od.com:2888:3888
[root@hdss7-11 ~]# echo 1 > /data/zookeeper/data/myid # 三台节点分别设为1，2，3
[root@hdss7-12 ~]# echo 2 > /data/zookeeper/data/myid
[root@hdss7-21 ~]# echo 3 > /data/zookeeper/data/myid
[root@hdss7-12 ~]# /opt/apps/zookeeper/bin/zkServer.sh --help
ZooKeeper JMX enabled by default
Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
Usage: /opt/apps/zookeeper/bin/zkServer.sh {start|start-foreground|stop|restart|status|upgrade|print-cmd}
[root@hdss7-12 ~]# /opt/apps/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hdss7-12 ~]# /opt/apps/zookeeper/bin/zkServer.sh status # 查看三个节点，其中一个为leader，其它为follower
ZooKeeper JMX enabled by default
Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
Mode: follower
-----
# 如果需要使用 supervisor 管理，需要调整启动脚本，比如配置Java环境变量
3. 交付Jenkins到K8S
3.1. 制作Jenkins镜像
# 准备基础镜像
[root@hdss7-200 ~]# docker pull jenkins/jenkins:2.190.3
[root@hdss7-200 ~]# docker image tag jenkins/jenkins:2.190.3 harbor.od.com/public/jenkins:v2.190.3
[root@hdss7-200 ~]# docker image push harbor.od.com/public/jenkins:v2.190.3
# 准备相关文件
[root@hdss7-200 docker_files]# ssh-keygen -t rsa -b 2048 -C "1659775014@qq.com" -N "" -f id_rsa
[root@hdss7-200 docker_files]# cp ~/.docker/config.json ./ # Docker登陆信息
# 该脚本就是在docker-ce源中安装了一个docker-ce-cli
# --mirror=Aliyun 指定使用阿里云的repo仓库
[root@hdss7-200 docker_files]# wget -O get-docker.sh https://get.docker.com
[root@hdss7-200 docker_files]# vim Dockerfile
# 修改默认的 Jenkins 镜像
FROM harbor.od.com/public/jenkins:v2.190.3
USER root
ADD id_rsa /root/.ssh/id_rsa
ADD config.json /root/.docker/config.json
ADD get-docker.sh /get-docker.sh
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone && echo " StrictHostKeyChecking no" >> /etc/ssh/ssh_config && sh /get-docker.sh ; rm -f get-docker.sh
[root@hdss7-200 docker_files]# docker image build -t harbor.od.com/infra/jenkins:v2.190.3 ./
# infra 是harbor的一个私有仓库，是infrastructure(基础设置)的缩写
[root@hdss7-200 docker_files]# docker image push harbor.od.com/infra/jenkins:v2.190.3
3.2. 准备NFS共享存储
NFS共享存储放在 hdss7-200 上，用于存储Jenkins持久化文件。所有Node和hdss7-200都需要安装
[root@hdss7-200 ~]# for i in 21 22 200;do ssh hdss7-$i "yum install -y nfs-utils";done
[root@hdss7-200 ~]# for i in 21 22 200;do echo -ne "hdss7-$i\t";ssh hdss7-$i "rpm -q nfs-utils >/dev/null 2>&1 && echo yes || echo no";done
[root@hdss7-200 ~]# systemctl start nfs ; systemctl enable nfs
[root@hdss7-200 ~]# mkdir -p /data/nfs-volume/jenkins_home
[root@hdss7-200 ~]# vim /etc/exports
/data/nfs-volume 10.4.7.0/24(rw,sync,no_root_squash)
[root@hdss7-200 ~]# systemctl reload nfs
[root@hdss7-200 ~]# showmount -e
Export list for hdss7-200.host.com:
/data/nfs-volume 10.4.7.0/24
3.3. 部署Jenkins
docker-registory 登陆密钥
# docker-registory 登陆密钥 # 使用 kubectl create secret docker-registry ... 生成的 apiVersion: v1 kind: Secret metadata: name: harbor namespace: infra type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: eyJhdXRocyI6eyJodHRwOi8vaGFyYm9yLm9kLmNvbS8iOnsidXNlcm5hbWUiOiJhZG1pbiIsInBhc3N3b3JkIjoiSGFyYm9yMTIzNDUiLCJhdXRoIjoiWVdSdGFXNDZTR0Z5WW05eU1USXpORFU9In19fQ==deployment.yaml
kind: Deployment apiVersion: extensions/v1beta1 metadata: name: jenkins namespace: infra labels: name: jenkins spec: replicas: 1 selector: matchLabels: name: jenkins template: metadata: labels: app: jenkins name: jenkins spec: volumes: - name: data nfs: server: hdss7-200 path: /data/nfs-volume/jenkins_home - name: docker hostPath: path: /run/docker.sock type: '' containers: - name: jenkins image: harbor.od.com/infra/jenkins:v2.190.3 ports: - containerPort: 8080 protocol: TCP env: - name: JAVA_OPTS value: -Xmx512m -Xms512m volumeMounts: - name: data mountPath: /var/jenkins_home - name: docker mountPath: /run/docker.sock imagePullSecrets: - name: harbor securityContext: runAsUser: 0 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 0 revisionHistoryLimit: 7 progressDeadlineSeconds: 600service.yaml
kind: Service apiVersion: v1 metadata: name: jenkins namespace: infra spec: ports: - protocol: TCP port: 80 targetPort: 8080 selector: app: jenkinsingress.yaml
kind: Ingress apiVersion: extensions/v1beta1 metadata: name: jenkins namespace: infra spec: rules: - host: jenkins.od.com http: paths: - path: / backend: serviceName: jenkins servicePort: 80
[root@hdss7-21 ~]# kubectl create namespace infra # 创建名称空间
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/secret.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/ingress.yaml
3.4. 配置DNS解析
[root@hdss7-11 ~]# vim /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
2020011306 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
k8s-yaml A 10.4.7.200
traefik A 10.4.7.10
dashboard A 10.4.7.10
monitor.api A 10.4.7.10
zk1 A 10.4.7.11
zk2 A 10.4.7.12
zk3 A 10.4.7.21
jenkins A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
3.5. 登陆Jenkins
[root@hdss7-21 ~]# kubectl get pod -n infra
NAME READY STATUS RESTARTS AGE
jenkins-54b8469cf9-mtkm5 1/1 Running 0 2m16s
[root@hdss7-21 ~]# kubectl log -f jenkins-54b8469cf9-mtkm5 -n infra # 取得初始化密码
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
04e1fec0d9db4283b6076532f857f137
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
# 检查Pod：运行用户是否为root、时区、是否连接到本地docker server、是否都能登陆harbor
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "ps aux|grep jenkins|grep -v grep"
root 1 0.0 0.0 1136 4 ? Ss 11:27 0:00 /sbin/tini -- /usr/local/bin/jenkins.sh
root 6 2.3 5.9 4100008 479344 ? Sl 11:27 0:49 java -Duser.home=/var/jenkins_home -Xmx512m -Xms512m -Djenkins.model.Jenkins.slaveAgentPort=50000 -jar /usr/share/jenkins/jenkins.war
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "date"
Sat Feb 1 12:02:28 CST 2020
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "docker container ps"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b76e1b0db2c4 add5fac61ae5 "/entrypoint.sh --ap…" Less than a second ago Up Less than a second k8s_traefik-ingress_traefik-ingress-vtlch_kube-system_da355707-a2d1-4f79-8dfd-3ebe36c6d31d_9
9e7b423da707 harbor.od.com/public/pause:latest "/pause" Less than a second ago Up Less than a second 0.0.0.0:81->80/tcp k8s_POD_traefik-ingress-vtlch_kube-system_da355707-a2d1-4f79-8dfd-3ebe36c6d31d_9
14f439b5bac9 ae5fe3d4bbb7 "/sbin/tini -- /usr/…" 34 minutes ago Up 34 minutes k8s_jenkins_jenkins-54b8469cf9-s6k4h_infra_9d14995c-7205-4b2c-98e9-de81a7e1845c_0
b4c79a78aee2 harbor.od.com/public/pause:latest "/pause" 34 minutes ago Up 34 minutes k8s_POD_jenkins-54b8469cf9-s6k4h_infra_9d14995c-7205-4b2c-98e9-de81a7e1845c_0
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "docker login harbor.od.com"
Authenticating with existing credentials...
Login Succeeded
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
# 确认是否能通过SSH方式连接到git仓库
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- /bin/sh -c "ssh -i /root/.ssh/id_rsa -T git@gitee.com"
Hi StanleyWang (DeployKey)! You've successfully authenticated, but GITEE.COM does not provide shell access.
Note: Perhaps the current use is DeployKey.
Note: DeployKey only supports pull/fetch operations
3.6. 配置Jenkins
3.6.1. 配置安全策略
3.6.2. 配置插件加速地址
地址：https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
# 修改下载地址
[root@hdss7-200 ~]# cd /data/nfs-volume/jenkins_home/updates
[root@hdss7-200 updates]# sed -i 's#http://updates.jenkins-ci.org/download#https://mirrors.tuna.tsinghua.edu.cn/jenkins#g;s#http://www.google.com#https://www.baidu.com#g' default.json
3.7. 安装Maven到Jenkins
Maven是提供给Jenkins使用，需要放到Jenkins的持久化目录中，直接将二进制包形式的Maven拷贝到Jenkins目录最方便。因此本次安装直接在 hdss7-200 操作。
在公司中，不同的项目对编译的JDK版本和Maven可能不同，可能需要多个版本的JDK和Maven组合使用，因此Maven目录名称就使用 maven-${maven_versin}-${jdk_version}格式。Maven的 bin/mvn 文件中可以定义 JAVA_HOME环境变量的值，不同的Maven可以使用不同的 JAVA_HOME 值。
3.7.1. 场景一
当Maven需求的jdk版本和jenkins一致时，不需要定义 bin/mvn 中JAVA_HOME。
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- java -version # 查看jenkins中jdk版本
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
[root@hdss7-200 src]# wget https://archive.apache.org/dist/maven/maven-3/3.6.2/binaries/apache-maven-3.6.2-bin.tar.gz
[root@hdss7-200 src]# tar -xf apache-maven-3.6.2-bin.tar.gz
[root@hdss7-200 src]# mv apache-maven-3.6.2 /data/nfs-volume/jenkins_home/maven-3.6.2-8u232 # 8u232是jenkins的jdk地址
# settings.xml 中 <mirrors></mirrors>标签中添加国内源
[root@hdss7-200 ~]# vim /data/nfs-volume/jenkins_home/maven-3.6.2-8u232/conf/settings.xml
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>*</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
3.7.2. 场景二
当Maven需要 oracle jdk-8u241 时：
[root@hdss7-200 src]# mkdir /data/nfs-volume/jenkins_home/jdk_versions
[root@hdss7-200 src]# tar -xf jdk-8u241-linux-x64.tar.gz -C /data/nfs-volume/jenkins_home/jdk_versions/
[root@hdss7-200 src]# cp -r /data/nfs-volume/jenkins_home/maven-3.6.2-8u232 /data/nfs-volume/jenkins_home/maven-3.6.2-8u241
[root@hdss7-200 src]# vim /data/nfs-volume/jenkins_home/maven-3.6.2-8u241/bin/mvn # 使用jenkins中绝对路径
JAVA_HOME='/var/jenkins_home/jdk_versions/jdk1.8.0_241'
3.8. 配置Jenkins流水线
# 参数
1. name: git_repo
type: string
description: 项目在git版本仓库的地址，如 https://gitee.com/xxx/dubbo-demo-service.git
2. name: app_name
type: string
description: 项目名称，如 dubbo-demo-service
3. name: git_ver
type: string
description: 项目在git仓库中对应的分支或者版本号
4. name: maven
type: choice
description: 编译时使用的maven目录中的版本号部分
5. name: mvn_cmd
type: string
default: mvn clean package -Dmaven.test.skip=true
description: 执行编译所用的指令
6. name: mvn_dir
type: string
default: ./
description: 在哪个目录执行编译，由开发同事提供
7. name: target_dir
type: string
default: ./target
description: 编译的jar/war文件存放目录，由开发同事提供
8. name: base_image
type: choice
default:
description: 项目使用的jre底包
9. name: image_name
type: string
description: docker镜像名称，如 app/dubbo-demo-service
10. name: add_tag
type: string
default:
description: 日期-时间，和git_ver拼在一起组成镜像的tag，如: 202002011001
pipeline {
agent any
stages {
stage('pull') { //get project code from repo
steps {
sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
}
}
stage('build') { //exec mvn cmd
steps {
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
}
}
stage('package') { //move jar file into project_dir
steps {
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
}
}
stage('image') { //build image and push to registry
steps {
writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/project_dir"""
sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
}
}
}
}
4. 制作JRE镜像底包
当前的底包只是一个最基础的底包，实际生成环境中，根据需要调整，或者根据当前的底包制作新的底包都可以。
# jre8:8u112 是java程序运行环境必要的程序
[root@hdss7-200 docker_files]# docker pull docker.io/stanleyws/jre8:8u112
[root@hdss7-200 docker_files]# docker image tag stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112
[root@hdss7-200 docker_files]# docker image push harbor.od.com/public/jre:8u112
# 监控agent和配置项
[root@hdss7-200 docker_files]# wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar
[root@hdss7-200 docker_files]# vim config.yml
rules:
- pattern: '.*'
# 默认启动脚本
[root@hdss7-200 docker_files]# vim entrypoint.sh
#!/bin/sh
# C_OPTS 和 JAR_BALL 由环境变量注入
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}
[root@hdss7-200 docker_files]# chmod +x entrypoint.sh
[root@hdss7-200 docker_files]# vim Dockerfile
FROM harbor.od.com/public/jre:8u112
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
ADD entrypoint.sh /entrypoint.sh
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone
WORKDIR /opt/project_dir
CMD ["/entrypoint.sh"]
[root@hdss7-200 docker_files]# docker image build -t harbor.od.com/base/jre:8u112 ./
[root@hdss7-200 docker_files]# docker image push harbor.od.com/base/jre:8u112
5. 构建和交付dubbo
5.1. 交付Provider到K8S
5.1.1. CI流水线构建
# 编译内容存放路径，数字目录表示某一个app编译的序号，每个目录下有自己的 Dockerfile
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- ls -l /var/jenkins_home/workspace/dubbo-demo/dubbo-demo-service
drwxr-xr-x 6 root root 101 Feb 2 11:51 1
drwxr-xr-x 6 root root 119 Feb 2 12:33 2
drwxr-xr-x 6 root root 101 Feb 2 12:35 3
drwxr-xr-x 6 root root 119 Feb 2 12:35 4
drwxr-xr-x 6 root root 119 Feb 2 14:16 5
drwxr-xr-x 6 root root 119 Feb 2 14:19 6
drwxr-xr-x 6 root root 119 Feb 2 14:28 7
# 第一次编译时会下载很多的第三方库文件，速度较慢，可以将下载后第三方库持久化，避免重启pod后速度变慢
# 第三方库的缓存目录在: /root/.m2/repository
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- ls -a /root/.m2/repository
5.1.2. 交付provider到k8s
# 准备工作
[root@hdss7-21 ~]# kubectl create namespace app
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username='app_view' --docker-password='Harbor12345' --docker-server='http://harbor.od.com/' -n app
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: app
labels:
name: dubbo-demo-service
tier: provider
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
version: master_20200202_1233
spec:
containers:
- name: dubbo-demo-service
image: harbor.od.com/app/dubbo-demo-service:master_20200202_1233
env:
- name: JAR_BALL
value: dubbo-server.jar
imagePullSecrets:
- name: harbor
terminationGracePeriodSeconds: 30
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-service/deployment.yaml
[root@hdss7-21 ~]# kubectl get pod -n app -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dubbo-demo-service-6949888ff6-b26t6 1/1 Running 0 10s 172.7.21.4 hdss7-21.host.com <none> <none>
[root@hdss7-21 ~]# kubectl logs dubbo-demo-service-6949888ff6-b26t6 -n app | tail -n 2
Dubbo server started
Dubbo 服务端已经启动
# 查看dubbo-demo-service是否连接到了ZK，通过以下方式查看
1. 通过 zkCli.sh 查看
[root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkCli.sh -server localhost:2181
[zk: localhost:2181(CONNECTED) 0] ls / # 确认存在 dubbo 即可
[dubbo, zookeeper]
[zk: localhost:2181(CONNECTED) 1] exit
2. 通过5.2中Monitor页面查看: Applications --> dubbo-demo-service
----
补充：dubbo-demo-service 连接zk的地址是写在配置文件中:
dubbo-server/src/main/java/config.properties:dubbo.registry=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
5.2. 交付Monitor到K8S
当前项目的Monitor非常丑陋，有很多不完善的地方，不太建议使用，以下只供参考：
5.2.1. 修改dubbo-monitor配置
[root@hdss7-200 src]# git clone https://github.com/Jeromefromcn/dubbo-monitor.git
[root@hdss7-200 src]# cd dubbo-minitor
# 修改配置文件
[root@hdss7-200 dubbo-minitor]# vim dubbo-monitor-simple/conf/dubbo_origin.properties
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=dubbo-monitor
dubbo.application.owner=duduniao
dubbo.registry.address=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=logs/dubbo-monitor-simple.log
dubbo.log4j.level=WARN
# 修改启动脚本
[root@hdss7-200 dubbo-minitor]# tail dubbo-monitor-simple/bin/start.sh
# 修改 启动参数，并且让 java 进程在前台运行，删除java之后的所有行
if [ -n "$BITS" ]; then
JAVA_MEM_OPTS=" -server -Xmx256m -Xms256m -Xmn32m -XX:PermSize=16m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
JAVA_MEM_OPTS=" -server -Xms256m -Xmx256m -XX:PermSize=32m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi
echo -e "Starting the $SERVER_NAME ...\c"
exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1
[root@hdss7-200 dubbo-minitor]# docker build . -t harbor.od.com/infra/dubbo-monitor:latest
[root@hdss7-200 dubbo-minitor]# docker push harbor.od.com/infra/dubbo-monitor:latest
5.2.2. monitor的资源配置清单
[root@hdss7-200 ~]# vim /data/k8s-yaml/devops/dubbo-monitor/deployment.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
labels:
name: dubbo-monitor
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-monitor
template:
metadata:
labels:
app: dubbo-monitor
name: dubbo-monitor
spec:
containers:
- name: dubbo-monitor
image: harbor.od.com/infra/dubbo-monitor:latest
imagePullSecrets:
- name: harbor
terminationGracePeriodSeconds: 30
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
kind: Service
apiVersion: v1
metadata:
name: dubbo-monitor
namespace: infra
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: dubbo-monitor
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
spec:
rules:
- host: dubbo-monitor.od.com
http:
paths:
- path: /
backend:
serviceName: dubbo-monitor
servicePort: 80
5.2.3. 交付monitor
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/dubbo-monitor/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/dubbo-monitor/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/dubbo-monitor/ingress.yaml
[root@hdss7-11 ~]# vim /var/named/od.com.zone # 配置DNS解析
......
dubbo-monitor A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
5.3. 交付Consumer到K8S
5.3.1. CI流水线构建
5.3.2. 资源配置清单
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
labels:
name: dubbo-demo-consumer
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-consumer
template:
metadata:
labels:
app: dubbo-demo-consumer
name: dubbo-demo-consumer
spec:
containers:
- name: dubbo-demo-consumer
image: harbor.od.com/app/dubbo-demo-consumer:master_20200202_1630
env:
- name: JAR_BALL
value: dubbo-client.jar
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
terminationGracePeriodSeconds: 30
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
kind: Service
apiVersion: v1
metadata:
name: dubbo-demo-consumer
namespace: app
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: dubbo-demo-consumer
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
spec:
rules:
- host: demo.od.com
http:
paths:
- path: /
backend:
serviceName: dubbo-demo-consumer
servicePort: 80
5.3.3. 交付Consumer
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-consumer/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-consumer/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-consumer/ingress.yaml
[root@hdss7-21 ~]# kubectl get pod -n app
NAME READY STATUS RESTARTS AGE
dubbo-demo-consumer-694b5645f9-dc4dg 1/1 Running 0 65s
dubbo-demo-service-6949888ff6-b26t6 1/1 Running 0 144m
[root@hdss7-21 ~]# kubectl logs dubbo-demo-consumer-694b5645f9-dc4dg -n app | tail -n 2
Dubbo client started
Dubbo 消费者端启动
[root@hdss7-11 ~]# vim /var/named/od.com.zone
......
demo A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
若有收获，就点个赞吧
0 人点赞
