04-2-配置中心

1. 配置中心介绍

1.1. 配置管理方式

当前的微服务环境中，配置项都不是硬编码到代码中的，需要通过外部文件或者参数的方式来实现可动态调整的配置，目前主要有以下几种方式来管理：

• 数据库：部分开发会将配置写在数据库中，定期读取和应用
• 环境变量：这种是云原生环境中很常用的一种方式，通过在容器启动时对全局环境变量赋值的方式来完成
• 命令行参数：通过Pod的args或者docker容器的CMD来实现启动参数调整
• configmap：K8S集群中独有配置管理方案，参考：ConfigMap
• 存储卷：将公共配置放在共享存储中，然后容器启动时挂载配置文件，不建议使用

• 第三方配置中心：如 Apollo、XDiamond、Qconf等，这类配置中心操作根据方便

  1.2. Apollo

  Apollo（阿波罗）是携程框架部门研发的开源配置管理中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性。Apollo支持4个维度管理Key-Value格式的配置：Namespace(名称空间)、Cluster(集群)、Environment(环境)、Application(应用)。具体介绍参考：https://github.com/ctripcorp/apollo/wiki

  1.2.1. Apollo 特性

• 统一管理不同环境(支持有限的环境关键词)、不同集群的配置

• 配置修改实时生效
• 版本发布管理
• 灰度发布
• 权限管理、发布审核、操作审计
• 客户端配置信息监控

• 提供Java和.Net原生客户端，且支持HTTP接口

  1.2.2. Apollo 基础模型

  用户发布配置到 Apollo 配置中心 —> 通知客户端配置发生变化 —> 客户端拉取配置
  

  1.2.3. 整体架构

• Config Service提供配置的读取、推送等功能，服务对象是Apollo客户端

• Admin Service提供配置的修改、发布等功能，服务对象是Apollo Portal（管理界面）
• Config Service和Admin Service都是多实例、无状态部署，所以需要将自己注册到Eureka中并保持心跳
• 在Eureka之上我们架了一层Meta Server用于封装Eureka的服务发现接口
• Client通过域名访问Meta Server获取Config Service服务列表（IP+Port），而后直接通过IP+Port访问服务，同时在Client侧会做load balance、错误重试
• Portal通过域名访问Meta Server获取Admin Service服务列表（IP+Port），而后直接通过IP+Port访问服务，同时在Portal侧会做load balance、错误重试
• 为了简化部署，我们实际上会把Config Service、Eureka和Meta Server三个逻辑角色部署在同一个JVM进程中，因此必须要先部署 Config Service，否则其它服务交付会出错

选择Eureka的原因：

• 它提供了完整的Service Registry和Service Discovery实现
  • 首先是提供了完整的实现，并且也经受住了Netflix自己的生产环境考验，相对使用起来会比较省心。
• 和Spring Cloud无缝集成
  • 我们的项目本身就使用了Spring Cloud和Spring Boot，同时Spring Cloud还有一套非常完善的开源代码来整合Eureka，所以使用起来非常方便。
  • 另外，Eureka还支持在我们应用自身的容器中启动，也就是说我们的应用启动完之后，既充当了Eureka的角色，同时也是服务的提供者。这样就极大的提高了服务的可用性。
  • 这一点是我们选择Eureka而不是zk、etcd等的主要原因，为了提高配置中心的可用性和降低部署复杂度，我们需要尽可能地减少外部依赖。

• Open Source

  • 最后一点是开源，由于代码是开源的，所以非常便于我们了解它的实现原理和排查问题。

    1.2.4. 客户端实现方式

    

• 客户端和服务端保持了一个长连接，从而能第一时间获得配置更新的推送。

• 客户端还会定时从Apollo配置中心服务端拉取应用的最新配置。
  • 这是一个fallback机制，为了防止推送机制失效导致配置不更新
  • 客户端定时拉取会上报本地版本，所以一般情况下，对于定时拉取的操作，服务端都会返回304 - Not Modified
  • 定时频率默认为每5分钟拉取一次，客户端也可以通过在运行时指定System Property: apollo.refreshInterval来覆盖，单位为分钟。
• 客户端从Apollo配置中心服务端获取到应用的最新配置后，会保存在内存中
• 客户端会把从服务端获取到的配置在本地文件系统缓存一份
  • 在遇到服务不可用，或网络不通的时候，依然能从本地恢复配置
• 应用程序从Apollo客户端获取最新的配置、订阅配置更新通知

  1.2.5. 高可用场景

  | 场景 | 影响 | 降级 | 原因 | | —- | —- | —- | —- | | 某台config service下线 | 无影响 | | Config service无状态，客户端重连其它config service | | 所有config service下线 | 客户端无法读取最新配置，Portal无影响 | 客户端重启时,可以读取本地缓存配置文件 | | | 某台admin service下线 | 无影响 | | Admin service无状态，Portal重连其它admin service | | 所有admin service下线 | 客户端无影响，portal无法更新配置 | | | | 某台portal下线 | 无影响 | | Portal域名通过slb绑定多台服务器，重试后指向可用的服务器 | | 全部portal下线 | 客户端无影响，portal无法更新配置 | | | | 某个数据中心下线 | 无影响 | | 多数据中心部署，数据完全同步，Meta Server/Portal域名通过slb自动切换到其它存活的数据中心 |

1.3. 部署架构图

生产实践中，开发环境、测试环境、生产环境放到三个不同的K8S集群中，三个环境中使用三套不同的 admin service、config service和数据库，共用同一个Protal前端。从而实现了分环境的配置中心。本次实验环境资源有限，采用不同的K8S名称空间区分不同的环境。

2. 准备环境

2.1. 停止dubbo微服务集群

持续集成章节中的Provider、Consumer、Monitor全部停止掉。

2.2. 拆分zookeeper集群

为了模拟生产、测试、开发三种环境，将zookeeper集群拆分为独立的三个节点。

# hdss7-11,hdss7-12,hdss7-21 操作一致
[root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkServer.sh stop
[root@hdss7-11 ~]# rm -fr /data/zookeeper/data/* /data/zookeeper/logs/* # 删除历史数据
[root@hdss7-11 ~]# sed -i '/^server/d' /opt/apps/zookeeper/conf/zoo.cfg # 删除集群配置
[root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkServer.sh start
[root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkServer.sh status  # 启动后，所有节点都为 standalone 模式
ZooKeeper JMX enabled by default
Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
Mode: standalone

2.3. 创建名称空间

[root@hdss7-22 ~]# kubectl create namespace pro
[root@hdss7-22 ~]# kubectl create namespace fat
[root@hdss7-22 ~]# kubectl create namespace dev
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username=admin --docker-password='Harbor12345' --docker-server=http://harbor.od.com -n pro
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username=admin --docker-password='Harbor12345' --docker-server=http://harbor.od.com -n fat
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username=admin --docker-password='Harbor12345' --docker-server=http://harbor.od.com -n dev

3. 部署Config Service

3.1. 初始化数据库

3.1.1. 刷SQL脚本

# 数据库服务器: hdss7-150.host.com  MySQL 5.7
[root@hdss7-150 src]# wget -O config.sql https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql
# SQL脚本中的调整: 
# 1. 不同环境分库；
# 2. 不同环境中 ApolloConfigDB.ServerConfig 表中Eureka地址调整
[root@hdss7-150 src]# sed -i 's#http://localhost:8080/eureka/#http://config.od.com/eureka/#' config.sql
[root@hdss7-150 src]# sed 's/ApolloConfigDB/ApolloConfigFatDB/;s#http://config.od.com/eureka/#http://config-fat.od.com/eureka/#' config.sql > config-fat.sql
[root@hdss7-150 src]# sed 's/ApolloConfigDB/ApolloConfigDevDB/;s#http://config.od.com/eureka/#http://config-dev.od.com/eureka/#' config.sql > config-dev.sql

[root@hdss7-150 src]# mysql -uroot -p < config.sql 
[root@hdss7-150 src]# mysql -uroot -p < config-fat.sql 
[root@hdss7-150 src]# mysql -uroot -p < config-dev.sql

# 授权，实践中不同环境的库可以使用不同的用户名和密码
# 因为Pod连接MySQL是做了SNAT，所以需要使用Node IP授权，如果需要显示Pod地址，可以添加路由表
mysql> grant SELECT,INSERT,UPDATE,DELETE on ApolloConfigDB.* to apolloconfig@'10.4.7.%' identified by 'Centos.1992!@#' ;
mysql> grant SELECT,INSERT,UPDATE,DELETE on ApolloConfigFatDB.* to apolloconfig@'10.4.7.%' identified by 'Centos.1992!@#' ;
mysql> grant SELECT,INSERT,UPDATE,DELETE on ApolloConfigDevDB.* to apolloconfig@'10.4.7.%' identified by 'Centos.1992!@#' ;
mysql> show grants for apolloconfig@'10.4.7.%';
+--------------------------------------------------------------------------------------------+
| Grants for apolloconfig@10.4.7.%                                                           |
+--------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'apolloconfig'@'10.4.7.%'                                            |
| GRANT SELECT, INSERT, UPDATE, DELETE ON `ApolloConfigDB`.* TO 'apolloconfig'@'10.4.7.%'    |
| GRANT SELECT, INSERT, UPDATE, DELETE ON `ApolloConfigFatDB`.* TO 'apolloconfig'@'10.4.7.%' |
| GRANT SELECT, INSERT, UPDATE, DELETE ON `ApolloConfigDevDB`.* TO 'apolloconfig'@'10.4.7.%' |
+--------------------------------------------------------------------------------------------+

3.1.2. 配置DNS解析

[root@hdss7-11 ~]# vim /var/named/od.com.zone 
......
config             A    10.4.7.10
config-fat         A    10.4.7.10
config-dev         A    10.4.7.10
mysql              A    10.4.7.150
[root@hdss7-11 ~]# systemctl restart named

3.2. 制作Config Service镜像

# 下载页面: https://github.com/ctripcorp/apollo/releases/tag/v1.5.1
[root@hdss7-200 src]# unzip apollo-configservice-1.5.1-github.zip -d docker_files/apollo-configservice
[root@hdss7-200 src]# cd docker_files/apollo-configservice
[root@hdss7-200 apollo-configservice]# ls config/ -l # 以下配置采用 configmap 挂载
total 8
-rw-r--r-- 1 root root 289 Nov  9 23:21 application-github.properties # DB信息
-rw-r--r-- 1 root root  30 Apr 20  2017 app.properties # appid，jdkVersion信息

#!/bin/bash
# 修改 scripts/startup.sh; 其中内存参数需要根据情况调整，此处增加了一个APOLLO_CONFIG_SERVICE_NAME，没有通过环境变量传递
# https://github.com/ctripcorp/apollo/tree/1.5.1/scripts/apollo-on-kubernetes/apollo-config-server/scripts
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)

SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms6144m -Xmx6144m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

[root@hdss7-200 apollo-configservice]# vim Dockerfile
# jre:8u112 制作地址: https://www.yuque.com/duduniao/ww8pmw/gp8n04#JJ6SE
# config 通过configmap来挂载, *sources.jar为源码，scritps/shutdown.sh 不需要
FROM harbor.od.com/public/jre:8u112
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone && mkdir -p /apollo-configservice/scripts
ADD apollo-configservice-1.5.1.jar /apollo-configservice/apollo-configservice.jar
ADD scripts/ /apollo-configservice/scripts
CMD ["/apollo-configservice/scripts/startup.sh"]
[root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
[root@hdss7-200 apollo-configservice]# docker image push harbor.od.com/infra/apollo-configservice:v1.5.1

3.3. 资源配置清单

3.3.1. configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: pro
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = Centos.1992!@#
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003171

3.3.2. deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apollo-configservice
  namespace: pro
  labels: 
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-configservice
  template:
    metadata:
      labels: 
        app: apollo-configservice 
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10

3.3.3. service

kind: Service
apiVersion: v1
metadata: 
  name: apollo-configservice
  namespace: pro
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: apollo-configservice

3.3.4. ingress

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-configservice
  namespace: pro
spec:
  rules:
  - host: config.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-configservice
          servicePort: 80

3.3.5. 其它环境中的配置清单

以上是生产环境pro中的清单，其它环境参考修改。修改点:

• Namespace 属性修改
• configmap中 eureka.service.url修改成对应环境地址、数据库名
• ingress中域名的修改

  3.4. 应用配置清单

  Pro、Dev和Fat环境应用资源清单方式一致，检查 info location 与 config 页面是否正常即可。

  [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/configmap.yaml
  [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/deployment.yaml
  [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/service.yaml
  [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/ingress.yaml
  

[root@hdss7-22 ~]# curl -s 172.7.22.5:8080/info # 访问 configservice pod返回信息
{"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}

4. 部署Admin service

4.1. 制作adminservice 镜像

# 下载页面: https://github.com/ctripcorp/apollo/releases/tag/v1.5.1
[root@hdss7-200 src]# mkdir -p docker_files/apollo-adminservice
[root@hdss7-200 src]# unzip -q apollo-adminservice-1.5.1-github.zip -d docker_files/apollo-adminservice
[root@hdss7-200 src]# cd docker_files/apollo-adminservice

[root@hdss7-200 apollo-adminservice]# vim scripts/startup.sh 
#!/bin/bash
# 此处修改了 SERVER_PORT APOLLO_ADMIN_SERVICE_NAME
# 参考地址：https://github.com/ctripcorp/apollo/tree/master/scripts/apollo-on-kubernetes/apollo-admin-server/scripts
SERVICE_NAME=apollo-adminservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)

# SERVER_URL="http://localhost:${SERVER_PORT}"
SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

[root@hdss7-200 apollo-adminservice]# vim Dockerfile 
# jre:8u112 制作地址: https://www.yuque.com/duduniao/ww8pmw/gp8n04#JJ6SE
FROM harbor.od.com/public/jre:8u112
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone && mkdir -p /apollo-adminservice/scripts

ADD apollo-adminservice-1.5.1.jar /apollo-adminservice/apollo-adminservice.jar
ADD scripts/startup.sh /apollo-adminservice/scripts
CMD ["/apollo-adminservice/scripts/startup.sh"]
[root@hdss7-200 apollo-adminservice]# docker image build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
[root@hdss7-200 apollo-adminservice]# docker image push harbor.od.com/infra/apollo-adminservice:v1.5.1

4.2. 资源配置清单

4.2.1. configmap

# appId 与 configservice、portal 不一致
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: pro
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = Centos.1992!@#
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003172

4.2.2. deployment

# admimservice 不对外提供统一接入口，其它模块通过 Eureka 查询到具体admin service地址
kind: Deployment
apiVersion: apps/v1
metadata:
  name: apollo-adminservice
  namespace: pro
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        app: apollo-adminservice 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

4.2.3. 其它环境中的配置清单

以上是生产环境pro中的清单，其它环境参考修改。修改点:

• Namespace 属性修改
• configmap中 eureka.service.url修改成对应环境地址

  4.3. 交付adminservice

  [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/adminservice/configmap.yaml
  [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/adminservice/deployment.yaml
  

[root@hdss7-22 ~]# kubectl get pod -n pro -o wide -l name=apollo-adminservice
NAME                                  READY   STATUS    RESTARTS   AGE    IP           NODE                NOMINATED NODE   READINESS GATES
apollo-adminservice-bffbf595c-hhcbq   1/1     Running   0          6m4s   172.7.22.5   hdss7-22.host.com   <none>           <none>
[root@hdss7-22 ~]# curl -s 172.7.22.5:8080/info
{"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}

5. 部署Portal

5.1. 初始化数据库

[root@hdss7-150 src]# wget -O portal.sql https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql

mysql> source /opt/src/portal.sql ;
grant SELECT,INSERT,UPDATE,DELETE on ApolloPortalDB.* to apolloportal@'10.4.7.%' identified by 'Centos.1992!@#' ;

5.2. 制作portal镜像

# 下载页面: https://github.com/ctripcorp/apollo/releases/tag/v1.5.1
[root@hdss7-200 src]# mkdir -p docker_files/apollo-portal
[root@hdss7-200 src]# unzip -q apollo-portal-1.5.1-github.zip -d docker_files/apollo-portal
[root@hdss7-200 src]# cd docker_files/apollo-portal
[root@hdss7-200 apollo-portal]# ls config/ -l
total 12
-rw-r--r-- 1 root root 234 Nov  9 23:21 apollo-env.properties
-rw-r--r-- 1 root root 218 Nov  9 23:21 application-github.properties
-rw-r--r-- 1 root root  30 Apr 20  2017 app.properties
[root@hdss7-200 apollo-portal]# cat config/apollo-env.properties # 不同环境配置
local.meta=http://localhost:8080               # 本地环境
dev.meta=http://fill-in-dev-meta-server:8080   # 开发环境:Development environment
fat.meta=http://fill-in-fat-meta-server:8080   # 测试环境:Feature Acceptance Test environment
uat.meta=http://fill-in-uat-meta-server:8080   # 预生产环境:User Acceptance Test environment
lpt.meta=${lpt_meta}
pro.meta=http://fill-in-pro-meta-server:8080   # 生产环境:Production environment
# 目前支持的变量有: LOCAL, DEV, FWS, FAT, UAT, LPT, PRO, TOOLS, UNKNOWN
# reference: https://github.com/ctripcorp/apollo/blob/master/apollo-core/src/main/java/com/ctrip/framework/apollo/core/enums/Env.java

[root@hdss7-200 apollo-portal]# vim scripts/startup.sh
#!/bin/bash
# 此处修改了 SERVER_PORT APOLLO_ADMIN_SERVICE_NAME
# 参考地址：https://github.com/ctripcorp/apollo/blob/1.5.1/scripts/apollo-on-kubernetes/apollo-portal-server/scripts/startup-kubernetes.sh
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)

# SERVER_URL="http://localhost:$SERVER_PORT"
SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

[root@hdss7-200 apollo-portal]# vim Dockerfile 
# jre:8u112 制作地址: https://www.yuque.com/duduniao/ww8pmw/gp8n04#JJ6SE
# config 通过configmap来挂载, *sources.jar为源码，scritps/shutdown.sh 不需要
FROM harbor.od.com/public/jre:8u112
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone && mkdir -p /apollo-portal/scripts
ADD apollo-portal-1.5.1.jar /apollo-portal/apollo-portal.jar
ADD scripts/startup.sh /apollo-portal/scripts
CMD ["/apollo-portal/scripts/startup.sh"]
[root@hdss7-200 apollo-portal]# docker image build . -t harbor.od.com/infra/apollo-portal:v1.5.1
[root@hdss7-200 apollo-portal]# docker image push harbor.od.com/infra/apollo-portal:v1.5.1

5.3. 资源配置清单

5.3.1. configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-portal-cm
  namespace: infra
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = Centos.1992!@#
  app.properties: |
    appId=100003173
  apollo-env.properties: |
    pro.meta=http://config.od.com
    fat.meta=http://config-fat.od.com
    dev.meta=http://config-dev.od.com

5.3.2. deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apollo-portal
  namespace: infra
  labels: 
    name: apollo-portal
spec:
  # 当前的负载均衡方式(IPVS-nq) 不支持多台 portal，否则会出现会话异常
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-portal
  template:
    metadata:
      labels: 
        app: apollo-portal 
        name: apollo-portal
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-portal-cm
      containers:
      - name: apollo-portal
        image: harbor.od.com/infra/apollo-portal:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-portal/config
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10

5.3.3. service

apiVersion: v1
kind: Service
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: apollo-portal

5.3.4. ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  rules:
  - host: portal.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-portal
          servicePort: 80

5.3.5. DNS配置

[root@hdss7-11 ~]# vim /var/named/od.com.zone
......
portal             A    10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# dig -t A portal.od.com @10.4.7.11 +short
10.4.7.10

5.4. 交付Portal

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/configmap.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/ingress.yaml

6. Apollo Portal使用

6.1. 创建多环境

6.2. 创建部门

mysql> select id,`key`,value from ApolloPortalDB.ServerConfig; # key 是保留字，需要用`key`才能查询
+----+-------------------------------------+-------------------------------------------------------------------------------------------+
| id | key                                 | value                                                                                     |
+----+-------------------------------------+-------------------------------------------------------------------------------------------+
|  1 | apollo.portal.envs                  | pro,fat,dev                                                                               |
|  2 | organizations                       | [{"orgId":"TEST1","orgName":"样例部门1"},{"orgId":"TEST2","orgName":"样例部门2"}]         |
|  3 | superAdmin                          | apollo                                                                                    |
|  4 | api.readTimeout                     | 10000                                                                                     |
|  5 | consumer.token.salt                 | someSalt                                                                                  |
|  6 | admin.createPrivateNamespace.switch | true                                                                                      |
|  7 | configView.memberOnly.envs          | pro                                                                                       |
+----+-------------------------------------+-------------------------------------------------------------------------------------------+

通过【系统参数】入口，可修改系统参数，参数表为 ApolloPortalDB.ServerConfig，常修改部门信息。

6.3. 创建项目

6.3.1. 创建管理用户

6.3.2. Dubbo微服务提供者

6.3.3. Dubbo微服务消费者

6.4. 多环境中使用Apollo

6.4.1. 编译和打包镜像

6.4.2. 应用提供者到K8S集群

# 注意：C_OPTS 环境变量为configservice地址，不同环境中namespace不一致
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dubbo-demo-service
  namespace: pro
  labels: 
    name: dubbo-demo-service
    tier: provider
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:apollo_20200205_1342
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://config.od.com
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-service/deployment.yaml

# 检查是否连接到apollo和配置是否正确，通过看日志来实现
[root@hdss7-22 ~]# kubectl get pod -n pro
NAME                                    READY   STATUS    RESTARTS   AGE
apollo-adminservice-bffbf595c-g88wz     1/1     Running   0          116m
apollo-configservice-5bf6df98df-2b458   1/1     Running   0          116m
dubbo-demo-service-7f6d79b894-fhgqb     1/1     Running   0          5m1s
[root@hdss7-22 ~]# kubectl logs dubbo-demo-service-7f6d79b894-fhgqb -n pro | grep -Eo 'config(-...)?\.od\.com|zk.\.od.com'|sort|uniq
config.od.com
zk1.od.com

6.4.3. 应用消费者到K8S集群

# 当前的deployment是在前一版本基础上改了镜像和增加环境变量，以及名称空间
# 前一版本参考 https://www.yuque.com/duduniao/ww8pmw/gp8n04#CYcLG
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dubbo-demo-consumer
  namespace: pro
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:apollo_20200203_1346
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://config.od.com
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

# 当前的service是在前一版本基础上改了名称空间
# 前一版本参考 https://www.yuque.com/duduniao/ww8pmw/gp8n04#CYcLG
apiVersion: v1
kind: Service
metadata: 
  name: dubbo-demo-consumer
  namespace: pro
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer

# 当前的ingress是在前一版本基础上改了名称空间
# 前一版本参考 https://www.yuque.com/duduniao/ww8pmw/gp8n04#CYcLG 
apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  name: dubbo-demo-consumer
  namespace: pro
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-demo-consumer
          servicePort: 80

# 三种环境中DNS配置
[root@hdss7-11 ~]# vim /var/named/od.com.zone 
......
config             A    10.4.7.10
config-fat         A    10.4.7.10
config-dev         A    10.4.7.10
[root@hdss7-11 ~]# systemctl restart named

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-consumer/deployment.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-consumer/service.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-consumer/ingress.yaml

[root@hdss7-22 ~]# kubectl get pod -n pro -l name=dubbo-demo-consumer
NAME                                 READY   STATUS    RESTARTS   AGE
dubbo-demo-consumer-6cb7dfd8-h8ldl   1/1     Running   0          112s
[root@hdss7-22 ~]# kubectl logs dubbo-demo-consumer-6cb7dfd8-h8ldl -n pro | grep -oE 'config(-...)?\.od\.com|zk.\.od.com'|sort|uniq
config.od.com
zk1.od.com

6.4.4. 其它环境部署方式

不同的环境使用相同的镜像，区别在于名称空间不同、Pod环境变量C_OPTS的configservice域名不同