04-1-持续集成

1. 组件介绍

1.1. 微服务

1.1.1. 介绍

微服务 (Microservices) 是一种软件架构风格，它是以专注于单一责任与功能的小型功能区块 (Small Building Blocks) 为基础，利用模块化的方式组合出复杂的大型应用程序，各功能区块使用与语言无关 (Language-Independent/Language agnostic) 的 API 集相互通信。微服务的优点：应用解耦，可以单独升级，甚至不同模块之间可以编程语言不同；横向扩展方便，高可用。缺点：架构复杂，运维成本高

1.1.2. Dubbo微服务框架

阿里巴巴在2011年开源了Dubbo框架，虽然在2013年停止更新，但在2017年9月又重启维护并发布了新版本。目前已有很多的公司将自己的业务建立在Dubbo之上，同时阿里云也推出了企业级分布式应用服务EDAS，为Dubbo提供应用托管。Dubbo采用Zookeeper作为注册中心，RPC作为服务调用方式，致力于提供高性能和透明化的RPC远程服务调用方案。

  • Provider：服务提供方发布服务到服务注册中心
  • Consumer：服务消费方从服务注册中心订阅服务
  • Registry：注册中心通知消息调用方服务已注册
  • Monitor：监控计数
  • 服务消费方调用已经注册的可用服务(RPC invoke)

04-1-持续集成 - 图1

1.2. Devops流水线

1.2.1. Jenkins

Jenkins是一款由Java编写的开源的持续集成工具。它运行在Servlet容器中（例如Apache Tomcat）。它支持软件配置管理（SCM）工具（包括AccuRev SCM、CVS、Subversion、Git、Perforce、Clearcase和RTC），可以执行基于Apache Ant和Apache Maven的项目，以及任意的Shell脚本和Windows批处理命令。Jenkins是一个有状态的应用，如果托管在K8S中，只运行一个副本，且需要持久化存储。

官网：https://jenkins.io/zh/

下载地址：https://jenkins.io/zh/download/

官方文档：https://jenkins.io/zh/doc/

1.2.2. Maven

Apache Maven，是一个软件（特别是Java软件）项目管理及自动构建工具，由Apache软件基金会所提供。Maven也可被用于构建和管理各种项目，例如C#，Ruby，Scala和其他语言编写的项目。Maven曾是Jakarta项目的子项目，现为由Apache软件基金会主持的独立Apache项目。在Devops流水线中，参与Java程序构建工具常用Maven。

官网：https://maven.apache.org/index.html

下载地址：https://archive.apache.org/dist/maven/

基础教程：https://www.runoob.com/maven/maven-setup.html

1.3. 实验拓扑图

04-1-持续集成 - 图2

04-1-持续集成 - 图3

2. ZK集群部署

ZK集群是有状态的服务，其选择Leader的方式和ETCD类似，要求集群节点是不低于3的奇数个。

主机 IP地址 角色
hdss7-11 10.4.7.11 zk1
hdss7-12 10.4.7.12 zk2
hdss7-21 10.4.7.21 zk3

2.1. 安装JDK

  1. [root@hdss7-12 ~]# cd /opt/src/
  2. [root@hdss7-12 src]# tar -xf jdk-8u241-linux-x64.tar.gz -C /opt/release/
  3. [root@hdss7-12 src]# ln -s /opt/release/jdk1.8.0_241 /opt/apps/jdk
  4. [root@hdss7-12 src]# ls /opt/apps/jdk -l
  5. lrwxrwxrwx 1 root root 25 Jan 31 11:36 /opt/apps/jdk -> /opt/release/jdk1.8.0_241
  7. [root@hdss7-12 src]# vim /etc/profile.d/java.sh
  8. export JAVA_HOME=/opt/apps/jdk
  9. export PATH=$JAVA_HOME/bin:$PATH
  10. export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
  11. [root@hdss7-12 src]# source /etc/profile.d/java.sh 
  12. [root@hdss7-12 src]# java -version
  13. java version "1.8.0_241"
  14. Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
  15. Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

2.2. 配置DNS解析

[root@hdss7-11 src]# vim /var/named/od.com.zone 
$ORIGIN od.com.
$TTL 600    ; 10 minutes
@           IN SOA  dns.od.com. dnsadmin.od.com. (
                2020011305 ; serial
                10800      ; refresh (3 hours)
                900        ; retry (15 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
                NS   dns.od.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11
harbor             A    10.4.7.200
k8s-yaml           A    10.4.7.200
traefik            A    10.4.7.10
dashboard          A    10.4.7.10
monitor.api        A    10.4.7.10
zk1                A    10.4.7.11
zk2                A    10.4.7.12
zk3                A    10.4.7.21
[root@hdss7-11 src]# systemctl restart named

[root@hdss7-11 src]# dig -t A zk3.od.com @10.4.7.11 +short
10.4.7.21

2.3. 安装ZK

[root@hdss7-12 src]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
[root@hdss7-12 src]# tar -xf zookeeper-3.4.14.tar.gz -C /opt/release/
[root@hdss7-12 src]# ln -s /opt/release/zookeeper-3.4.14 /opt/apps/zookeeper
[root@hdss7-12 src]# ll /opt/apps/zookeeper
lrwxrwxrwx 1 root root 29 Jan 31 11:52 /opt/apps/zookeeper -> /opt/release/zookeeper-3.4.14
[root@hdss7-12 src]# mkdir -p /data/zookeeper/data /data/zookeeper/logs

[root@hdss7-12 src]# vim /opt/apps/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zk1.od.com:2888:3888
server.2=zk2.od.com:2888:3888
server.3=zk3.od.com:2888:3888
[root@hdss7-11 ~]# echo 1 > /data/zookeeper/data/myid # 三台节点分别设为1，2，3
[root@hdss7-12 ~]# echo 2 > /data/zookeeper/data/myid
[root@hdss7-21 ~]# echo 3 > /data/zookeeper/data/myid

[root@hdss7-12 ~]# /opt/apps/zookeeper/bin/zkServer.sh --help
ZooKeeper JMX enabled by default
Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
Usage: /opt/apps/zookeeper/bin/zkServer.sh {start|start-foreground|stop|restart|status|upgrade|print-cmd}
[root@hdss7-12 ~]# /opt/apps/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hdss7-12 ~]# /opt/apps/zookeeper/bin/zkServer.sh status  # 查看三个节点，其中一个为leader，其它为follower
ZooKeeper JMX enabled by default
Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
Mode: follower
-----
# 如果需要使用 supervisor 管理，需要调整启动脚本，比如配置Java环境变量

3. 交付Jenkins到K8S

3.1. 制作Jenkins镜像

# 准备基础镜像
[root@hdss7-200 ~]# docker pull jenkins/jenkins:2.190.3 
[root@hdss7-200 ~]# docker image tag jenkins/jenkins:2.190.3 harbor.od.com/public/jenkins:v2.190.3
[root@hdss7-200 ~]# docker image push harbor.od.com/public/jenkins:v2.190.3

# 准备相关文件
[root@hdss7-200 docker_files]# ssh-keygen -t rsa -b 2048 -C "1659775014@qq.com" -N "" -f id_rsa
[root@hdss7-200 docker_files]# cp ~/.docker/config.json ./  # Docker登陆信息
# 该脚本就是在docker-ce源中安装了一个docker-ce-cli
# --mirror=Aliyun 指定使用阿里云的repo仓库
[root@hdss7-200 docker_files]# wget -O get-docker.sh https://get.docker.com 

[root@hdss7-200 docker_files]# vim Dockerfile 
# 修改默认的 Jenkins 镜像
FROM harbor.od.com/public/jenkins:v2.190.3
USER root 
ADD id_rsa /root/.ssh/id_rsa
ADD config.json /root/.docker/config.json
ADD get-docker.sh /get-docker.sh
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone && echo "   StrictHostKeyChecking no" >> /etc/ssh/ssh_config && sh /get-docker.sh ; rm -f get-docker.sh 
[root@hdss7-200 docker_files]# docker image build -t harbor.od.com/infra/jenkins:v2.190.3 ./

# infra 是harbor的一个私有仓库，是infrastructure(基础设置)的缩写
[root@hdss7-200 docker_files]# docker image push harbor.od.com/infra/jenkins:v2.190.3

3.2. 准备NFS共享存储

NFS共享存储放在 hdss7-200 上，用于存储Jenkins持久化文件。所有Node和hdss7-200都需要安装

[root@hdss7-200 ~]# for i in 21 22 200;do ssh hdss7-$i "yum install -y nfs-utils";done
[root@hdss7-200 ~]# for i in 21 22 200;do echo -ne "hdss7-$i\t";ssh hdss7-$i "rpm -q nfs-utils >/dev/null 2>&1 && echo yes || echo no";done
[root@hdss7-200 ~]# systemctl start nfs ; systemctl enable nfs

[root@hdss7-200 ~]# mkdir -p /data/nfs-volume/jenkins_home
[root@hdss7-200 ~]# vim /etc/exports
/data/nfs-volume  10.4.7.0/24(rw,sync,no_root_squash)
[root@hdss7-200 ~]# systemctl reload nfs
[root@hdss7-200 ~]# showmount -e 
Export list for hdss7-200.host.com:
/data/nfs-volume 10.4.7.0/24

3.3. 部署Jenkins

  • docker-registory 登陆密钥
# docker-registory 登陆密钥
# 使用 kubectl create secret docker-registry ... 生成的
apiVersion: v1
kind: Secret
metadata:
  name: harbor
  namespace: infra
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwOi8vaGFyYm9yLm9kLmNvbS8iOnsidXNlcm5hbWUiOiJhZG1pbiIsInBhc3N3b3JkIjoiSGFyYm9yMTIzNDUiLCJhdXRoIjoiWVdSdGFXNDZTR0Z5WW05eU1USXpORFU9In19fQ==
  • deployment.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: jenkins
  namespace: infra
  labels: 
    name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: jenkins
  template:
    metadata:
      labels: 
        app: jenkins 
        name: jenkins
    spec:
      volumes:
      - name: data
        nfs: 
          server: hdss7-200
          path: /data/nfs-volume/jenkins_home
      - name: docker
        hostPath: 
          path: /run/docker.sock
          type: ''
      containers:
      - name: jenkins
        image: harbor.od.com/infra/jenkins:v2.190.3
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: JAVA_OPTS
          value: -Xmx512m -Xms512m
        volumeMounts:
        - name: data
          mountPath: /var/jenkins_home
        - name: docker
          mountPath: /run/docker.sock
      imagePullSecrets:
      - name: harbor
      securityContext: 
        runAsUser: 0
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 0
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
  • service.yaml
kind: Service
apiVersion: v1
metadata: 
  name: jenkins
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: jenkins
  • ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: jenkins
  namespace: infra
spec:
  rules:
  - host: jenkins.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: jenkins
          servicePort: 80

[root@hdss7-21 ~]# kubectl create namespace infra  # 创建名称空间
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/secret.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/jenkins/ingress.yaml

3.4. 配置DNS解析

[root@hdss7-11 ~]# vim /var/named/od.com.zone 
$ORIGIN od.com.
$TTL 600    ; 10 minutes
@           IN SOA  dns.od.com. dnsadmin.od.com. (
                2020011306 ; serial
                10800      ; refresh (3 hours)
                900        ; retry (15 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
                NS   dns.od.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11
harbor             A    10.4.7.200
k8s-yaml           A    10.4.7.200
traefik            A    10.4.7.10
dashboard          A    10.4.7.10
monitor.api        A    10.4.7.10
zk1                A    10.4.7.11
zk2                A    10.4.7.12
zk3                A    10.4.7.21
jenkins            A    10.4.7.10
[root@hdss7-11 ~]# systemctl restart named

3.5. 登陆Jenkins

[root@hdss7-21 ~]# kubectl get pod -n infra
NAME                       READY   STATUS    RESTARTS   AGE
jenkins-54b8469cf9-mtkm5   1/1     Running   0          2m16s
[root@hdss7-21 ~]# kubectl log -f jenkins-54b8469cf9-mtkm5  -n infra # 取得初始化密码
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

04e1fec0d9db4283b6076532f857f137

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

# 检查Pod：运行用户是否为root、时区、是否连接到本地docker server、是否都能登陆harbor
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "ps aux|grep jenkins|grep -v grep"
root          1  0.0  0.0   1136     4 ?        Ss   11:27   0:00 /sbin/tini -- /usr/local/bin/jenkins.sh
root          6  2.3  5.9 4100008 479344 ?      Sl   11:27   0:49 java -Duser.home=/var/jenkins_home -Xmx512m -Xms512m -Djenkins.model.Jenkins.slaveAgentPort=50000 -jar /usr/share/jenkins/jenkins.war
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "date"
Sat Feb  1 12:02:28 CST 2020
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "docker container ps"
CONTAINER ID        IMAGE                               COMMAND                  CREATED                  STATUS                  PORTS                NAMES
b76e1b0db2c4        add5fac61ae5                        "/entrypoint.sh --ap…"   Less than a second ago   Up Less than a second                        k8s_traefik-ingress_traefik-ingress-vtlch_kube-system_da355707-a2d1-4f79-8dfd-3ebe36c6d31d_9
9e7b423da707        harbor.od.com/public/pause:latest   "/pause"                 Less than a second ago   Up Less than a second   0.0.0.0:81->80/tcp   k8s_POD_traefik-ingress-vtlch_kube-system_da355707-a2d1-4f79-8dfd-3ebe36c6d31d_9
14f439b5bac9        ae5fe3d4bbb7                        "/sbin/tini -- /usr/…"   34 minutes ago           Up 34 minutes                                k8s_jenkins_jenkins-54b8469cf9-s6k4h_infra_9d14995c-7205-4b2c-98e9-de81a7e1845c_0
b4c79a78aee2        harbor.od.com/public/pause:latest   "/pause"                 34 minutes ago           Up 34 minutes                                k8s_POD_jenkins-54b8469cf9-s6k4h_infra_9d14995c-7205-4b2c-98e9-de81a7e1845c_0
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-s6k4h -n infra -- /bin/sh -c "docker login harbor.od.com"
Authenticating with existing credentials...
Login Succeeded
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

# 确认是否能通过SSH方式连接到git仓库
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- /bin/sh -c "ssh -i /root/.ssh/id_rsa -T git@gitee.com"
Hi StanleyWang (DeployKey)! You've successfully authenticated, but GITEE.COM does not provide shell access.
Note: Perhaps the current use is DeployKey.
Note: DeployKey only supports pull/fetch operations

04-1-持续集成 - 图4

3.6. 配置Jenkins

3.6.1. 配置安全策略

04-1-持续集成 - 图5

04-1-持续集成 - 图6

3.6.2. 配置插件加速地址

04-1-持续集成 - 图7

地址：https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json

04-1-持续集成 - 图8

# 修改下载地址
[root@hdss7-200 ~]# cd /data/nfs-volume/jenkins_home/updates
[root@hdss7-200 updates]# sed -i 's#http://updates.jenkins-ci.org/download#https://mirrors.tuna.tsinghua.edu.cn/jenkins#g;s#http://www.google.com#https://www.baidu.com#g' default.json

04-1-持续集成 - 图9

3.7. 安装Maven到Jenkins

Maven是提供给Jenkins使用，需要放到Jenkins的持久化目录中，直接将二进制包形式的Maven拷贝到Jenkins目录最方便。因此本次安装直接在 hdss7-200 操作。

在公司中，不同的项目对编译的JDK版本和Maven可能不同，可能需要多个版本的JDK和Maven组合使用，因此Maven目录名称就使用 maven-04-1-持续集成 - 图10{jdk_version}格式。Maven的 bin/mvn 文件中可以定义 JAVA_HOME环境变量的值，不同的Maven可以使用不同的 JAVA_HOME 值。

3.7.1. 场景一

当Maven需求的jdk版本和jenkins一致时，不需要定义 bin/mvn 中JAVA_HOME。

[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- java -version # 查看jenkins中jdk版本
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

[root@hdss7-200 src]# wget https://archive.apache.org/dist/maven/maven-3/3.6.2/binaries/apache-maven-3.6.2-bin.tar.gz
[root@hdss7-200 src]# tar -xf apache-maven-3.6.2-bin.tar.gz
[root@hdss7-200 src]# mv apache-maven-3.6.2 /data/nfs-volume/jenkins_home/maven-3.6.2-8u232 # 8u232是jenkins的jdk地址

# settings.xml 中 <mirrors></mirrors>标签中添加国内源
[root@hdss7-200 ~]# vim /data/nfs-volume/jenkins_home/maven-3.6.2-8u232/conf/settings.xml
<mirror>
  <id>nexus-aliyun</id>
  <mirrorOf>*</mirrorOf>
  <name>Nexus aliyun</name>
  <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

3.7.2. 场景二

当Maven需要 oracle jdk-8u241 时：

[root@hdss7-200 src]# mkdir /data/nfs-volume/jenkins_home/jdk_versions
[root@hdss7-200 src]# tar -xf jdk-8u241-linux-x64.tar.gz -C /data/nfs-volume/jenkins_home/jdk_versions/

[root@hdss7-200 src]# cp -r /data/nfs-volume/jenkins_home/maven-3.6.2-8u232 /data/nfs-volume/jenkins_home/maven-3.6.2-8u241
[root@hdss7-200 src]# vim /data/nfs-volume/jenkins_home/maven-3.6.2-8u241/bin/mvn # 使用jenkins中绝对路径
JAVA_HOME='/var/jenkins_home/jdk_versions/jdk1.8.0_241'

3.8. 配置Jenkins流水线

04-1-持续集成 - 图11

04-1-持续集成 - 图12

04-1-持续集成 - 图13

# 参数
1.  name: git_repo
    type: string
    description: 项目在git版本仓库的地址，如 https://gitee.com/xxx/dubbo-demo-service.git

2.  name: app_name
    type: string
    description: 项目名称，如 dubbo-demo-service  

3.  name: git_ver
    type: string
    description: 项目在git仓库中对应的分支或者版本号

4.  name: maven
    type: choice
    description: 编译时使用的maven目录中的版本号部分

5.  name: mvn_cmd
    type: string
    default: mvn clean package -Dmaven.test.skip=true
    description: 执行编译所用的指令

6.  name: mvn_dir
    type: string
    default: ./
    description: 在哪个目录执行编译，由开发同事提供

7.  name: target_dir
    type: string
    default: ./target
    description: 编译的jar/war文件存放目录，由开发同事提供

8.  name: base_image
    type: choice
    default:
    description: 项目使用的jre底包

9.  name: image_name
    type: string
    description: docker镜像名称，如 app/dubbo-demo-service 

10. name: add_tag
    type: string
    default:
    description: 日期-时间，和git_ver拼在一起组成镜像的tag，如: 202002011001

04-1-持续集成 - 图14

pipeline {
  agent any 
    stages {
      stage('pull') { //get project code from repo 
        steps {
          sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
        }
      }
      stage('build') { //exec mvn cmd
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
        }
      }
      stage('package') { //move jar file into project_dir
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
        }
      }
      stage('image') { //build image and push to registry
        steps {
          writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/project_dir"""
          sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
        }
      }
    }
}

4. 制作JRE镜像底包

当前的底包只是一个最基础的底包，实际生成环境中，根据需要调整，或者根据当前的底包制作新的底包都可以。

# jre8:8u112 是java程序运行环境必要的程序
[root@hdss7-200 docker_files]# docker pull docker.io/stanleyws/jre8:8u112
[root@hdss7-200 docker_files]# docker image tag stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112
[root@hdss7-200 docker_files]# docker image push harbor.od.com/public/jre:8u112

# 监控agent和配置项
[root@hdss7-200 docker_files]# wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar
[root@hdss7-200 docker_files]# vim config.yml  
rules:
  - pattern: '.*'

# 默认启动脚本
[root@hdss7-200 docker_files]# vim entrypoint.sh
#!/bin/sh
# C_OPTS 和 JAR_BALL 由环境变量注入
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}
[root@hdss7-200 docker_files]# chmod +x entrypoint.sh

[root@hdss7-200 docker_files]# vim Dockerfile
FROM harbor.od.com/public/jre:8u112
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
ADD entrypoint.sh /entrypoint.sh
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone
WORKDIR /opt/project_dir
CMD ["/entrypoint.sh"]
[root@hdss7-200 docker_files]# docker image build -t harbor.od.com/base/jre:8u112 ./
[root@hdss7-200 docker_files]# docker image push harbor.od.com/base/jre:8u112

5. 构建和交付dubbo

5.1. 交付Provider到K8S

5.1.1. CI流水线构建

04-1-持续集成 - 图15

04-1-持续集成 - 图16

# 编译内容存放路径，数字目录表示某一个app编译的序号，每个目录下有自己的 Dockerfile
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- ls -l /var/jenkins_home/workspace/dubbo-demo/dubbo-demo-service
drwxr-xr-x 6 root root 101 Feb  2 11:51 1
drwxr-xr-x 6 root root 119 Feb  2 12:33 2
drwxr-xr-x 6 root root 101 Feb  2 12:35 3
drwxr-xr-x 6 root root 119 Feb  2 12:35 4
drwxr-xr-x 6 root root 119 Feb  2 14:16 5
drwxr-xr-x 6 root root 119 Feb  2 14:19 6
drwxr-xr-x 6 root root 119 Feb  2 14:28 7
# 第一次编译时会下载很多的第三方库文件，速度较慢，可以将下载后第三方库持久化，避免重启pod后速度变慢
# 第三方库的缓存目录在: /root/.m2/repository
[root@hdss7-21 ~]# kubectl exec jenkins-54b8469cf9-v8g28 -n infra -- ls -a /root/.m2/repository

5.1.2. 交付provider到k8s

# 准备工作
[root@hdss7-21 ~]# kubectl create namespace app
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username='app_view' --docker-password='Harbor12345' --docker-server='http://harbor.od.com/' -n app

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-service
  namespace: app
  labels: 
    name: dubbo-demo-service
    tier: provider
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
        version: master_20200202_1233
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:master_20200202_1233
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
      imagePullSecrets:
      - name: harbor
      terminationGracePeriodSeconds: 30
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 0
      maxSurge: 1
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-service/deployment.yaml
[root@hdss7-21 ~]# kubectl get pod -n app -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
dubbo-demo-service-6949888ff6-b26t6   1/1     Running   0          10s   172.7.21.4   hdss7-21.host.com   <none>           <none>
[root@hdss7-21 ~]# kubectl logs dubbo-demo-service-6949888ff6-b26t6 -n app | tail -n 2
Dubbo server started
Dubbo 服务端已经启动

# 查看dubbo-demo-service是否连接到了ZK，通过以下方式查看
1. 通过 zkCli.sh 查看
[root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkCli.sh -server localhost:2181
[zk: localhost:2181(CONNECTED) 0] ls /  # 确认存在 dubbo 即可
[dubbo, zookeeper]
[zk: localhost:2181(CONNECTED) 1] exit
2. 通过5.2中Monitor页面查看: Applications --> dubbo-demo-service
----
补充：dubbo-demo-service 连接zk的地址是写在配置文件中:
dubbo-server/src/main/java/config.properties:dubbo.registry=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181

5.2. 交付Monitor到K8S

当前项目的Monitor非常丑陋，有很多不完善的地方，不太建议使用，以下只供参考：

5.2.1. 修改dubbo-monitor配置

[root@hdss7-200 src]# git clone https://github.com/Jeromefromcn/dubbo-monitor.git
[root@hdss7-200 src]# cd dubbo-minitor
# 修改配置文件
[root@hdss7-200 dubbo-minitor]# vim dubbo-monitor-simple/conf/dubbo_origin.properties
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=dubbo-monitor
dubbo.application.owner=duduniao
dubbo.registry.address=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=logs/dubbo-monitor-simple.log
dubbo.log4j.level=WARN
# 修改启动脚本
[root@hdss7-200 dubbo-minitor]# tail dubbo-monitor-simple/bin/start.sh
# 修改 启动参数，并且让 java 进程在前台运行，删除java之后的所有行
if [ -n "$BITS" ]; then
    JAVA_MEM_OPTS=" -server -Xmx256m -Xms256m -Xmn32m -XX:PermSize=16m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
    JAVA_MEM_OPTS=" -server -Xms256m -Xmx256m -XX:PermSize=32m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi

echo -e "Starting the $SERVER_NAME ...\c"
exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1
[root@hdss7-200 dubbo-minitor]# docker build . -t harbor.od.com/infra/dubbo-monitor:latest
[root@hdss7-200 dubbo-minitor]# docker push harbor.od.com/infra/dubbo-monitor:latest

5.2.2. monitor的资源配置清单

[root@hdss7-200 ~]# vim /data/k8s-yaml/devops/dubbo-monitor/deployment.yaml 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels: 
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-monitor
  template:
    metadata:
      labels: 
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
      imagePullSecrets:
      - name: harbor
      terminationGracePeriodSeconds: 30
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

kind: Service
apiVersion: v1
metadata: 
  name: dubbo-monitor
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: dubbo-monitor

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-monitor
  namespace: infra
spec:
  rules:
  - host: dubbo-monitor.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-monitor
          servicePort: 80

5.2.3. 交付monitor

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/dubbo-monitor/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/dubbo-monitor/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/dubbo-monitor/ingress.yaml

[root@hdss7-11 ~]# vim /var/named/od.com.zone  # 配置DNS解析
......
dubbo-monitor      A    10.4.7.10
[root@hdss7-11 ~]# systemctl restart named

04-1-持续集成 - 图17

5.3. 交付Consumer到K8S

5.3.1. CI流水线构建

04-1-持续集成 - 图18

5.3.2. 资源配置清单

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:master_20200202_1630
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      terminationGracePeriodSeconds: 30
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

kind: Service
apiVersion: v1
metadata: 
  name: dubbo-demo-consumer
  namespace: app
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-demo-consumer
  namespace: app
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-demo-consumer
          servicePort: 80

5.3.3. 交付Consumer

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-consumer/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-consumer/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/app/dubbo-demo-consumer/ingress.yaml
[root@hdss7-21 ~]# kubectl get pod -n app
NAME                                   READY   STATUS    RESTARTS   AGE
dubbo-demo-consumer-694b5645f9-dc4dg   1/1     Running   0          65s
dubbo-demo-service-6949888ff6-b26t6    1/1     Running   0          144m
[root@hdss7-21 ~]# kubectl logs dubbo-demo-consumer-694b5645f9-dc4dg -n app | tail -n 2
Dubbo client started
Dubbo 消费者端启动

[root@hdss7-11 ~]# vim /var/named/od.com.zone
......
demo               A    10.4.7.10
[root@hdss7-11 ~]# systemctl restart named

04-1-持续集成 - 图19

04-1-持续集成 - 图20