1. 配置中心介绍

1.1. 配置管理方式

当前的微服务环境中，配置项都不是硬编码到代码中的，需要通过外部文件或者参数的方式来实现可动态调整的配置，目前主要有以下几种方式来管理：

  • 数据库：部分开发会将配置写在数据库中，定期读取和应用
  • 环境变量：这种是云原生环境中很常用的一种方式，通过在容器启动时对全局环境变量赋值的方式来完成
  • 命令行参数：通过Pod的args或者docker容器的CMD来实现启动参数调整
  • configmap：K8S集群中独有配置管理方案，参考：ConfigMap
  • 存储卷：将公共配置放在共享存储中，然后容器启动时挂载配置文件，不建议使用
  • 第三方配置中心：如 Apollo、XDiamond、Qconf等，这类配置中心操作根据方便

1.2. Apollo

Apollo（阿波罗）是携程框架部门研发的开源配置管理中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性。Apollo支持4个维度管理Key-Value格式的配置：Namespace(名称空间)、Cluster(集群)、Environment(环境)、Application(应用)。具体介绍参考：https://github.com/ctripcorp/apollo/wiki

1.2.1. Apollo 特性

  • 统一管理不同环境(支持有限的环境关键词)、不同集群的配置
  • 配置修改实时生效
  • 版本发布管理
  • 灰度发布
  • 权限管理、发布审核、操作审计
  • 客户端配置信息监控
  • 提供Java和.Net原生客户端，且支持HTTP接口

1.2.2. Apollo 基础模型

用户发布配置到 Apollo 配置中心 —> 通知客户端配置发生变化 —> 客户端拉取配置

04-2-配置中心 - 图1

1.2.3. 整体架构

  • Config Service提供配置的读取、推送等功能，服务对象是Apollo客户端
  • Admin Service提供配置的修改、发布等功能，服务对象是Apollo Portal（管理界面）
  • Config Service和Admin Service都是多实例、无状态部署，所以需要将自己注册到Eureka中并保持心跳
  • 在Eureka之上我们架了一层Meta Server用于封装Eureka的服务发现接口
  • Client通过域名访问Meta Server获取Config Service服务列表（IP+Port），而后直接通过IP+Port访问服务，同时在Client侧会做load balance、错误重试
  • Portal通过域名访问Meta Server获取Admin Service服务列表（IP+Port），而后直接通过IP+Port访问服务，同时在Portal侧会做load balance、错误重试
  • 为了简化部署，我们实际上会把Config Service、Eureka和Meta Server三个逻辑角色部署在同一个JVM进程中，因此必须要先部署 Config Service，否则其它服务交付会出错

04-2-配置中心 - 图2

选择Eureka的原因：

  • 它提供了完整的Service Registry和Service Discovery实现
    • 首先是提供了完整的实现，并且也经受住了Netflix自己的生产环境考验，相对使用起来会比较省心。
  • 和Spring Cloud无缝集成
    • 我们的项目本身就使用了Spring Cloud和Spring Boot，同时Spring Cloud还有一套非常完善的开源代码来整合Eureka，所以使用起来非常方便。
    • 另外，Eureka还支持在我们应用自身的容器中启动，也就是说我们的应用启动完之后，既充当了Eureka的角色，同时也是服务的提供者。这样就极大的提高了服务的可用性。
    • 这一点是我们选择Eureka而不是zk、etcd等的主要原因，为了提高配置中心的可用性和降低部署复杂度，我们需要尽可能地减少外部依赖。
  • Open Source
    • 最后一点是开源，由于代码是开源的，所以非常便于我们了解它的实现原理和排查问题。

1.2.4. 客户端实现方式

04-2-配置中心 - 图3

  • 客户端和服务端保持了一个长连接，从而能第一时间获得配置更新的推送。
  • 客户端还会定时从Apollo配置中心服务端拉取应用的最新配置。
    • 这是一个fallback机制，为了防止推送机制失效导致配置不更新
    • 客户端定时拉取会上报本地版本，所以一般情况下，对于定时拉取的操作，服务端都会返回304 - Not Modified
    • 定时频率默认为每5分钟拉取一次，客户端也可以通过在运行时指定System Property: apollo.refreshInterval来覆盖，单位为分钟。
  • 客户端从Apollo配置中心服务端获取到应用的最新配置后，会保存在内存中
  • 客户端会把从服务端获取到的配置在本地文件系统缓存一份
    • 在遇到服务不可用，或网络不通的时候，依然能从本地恢复配置
  • 应用程序从Apollo客户端获取最新的配置、订阅配置更新通知

1.2.5. 高可用场景

场景 影响 降级 原因
某台config service下线 无影响 Config service无状态，客户端重连其它config service
所有config service下线 客户端无法读取最新配置，Portal无影响 客户端重启时,可以读取本地缓存配置文件
某台admin service下线 无影响 Admin service无状态，Portal重连其它admin service
所有admin service下线 客户端无影响，portal无法更新配置
某台portal下线 无影响 Portal域名通过slb绑定多台服务器，重试后指向可用的服务器
全部portal下线 客户端无影响，portal无法更新配置
某个数据中心下线 无影响 多数据中心部署，数据完全同步，Meta Server/Portal域名通过slb自动切换到其它存活的数据中心

1.3. 部署架构图

生产实践中，开发环境、测试环境、生产环境放到三个不同的K8S集群中，三个环境中使用三套不同的 admin service、config service和数据库，共用同一个Protal前端。从而实现了分环境的配置中心。本次实验环境资源有限，采用不同的K8S名称空间区分不同的环境。

04-2-配置中心 - 图4

04-2-配置中心 - 图5

2. 准备环境

2.1. 停止dubbo微服务集群

持续集成章节中的Provider、Consumer、Monitor全部停止掉。

2.2. 拆分zookeeper集群

为了模拟生产、测试、开发三种环境，将zookeeper集群拆分为独立的三个节点。

  1. # hdss7-11,hdss7-12,hdss7-21 操作一致
  2. [root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkServer.sh stop
  3. [root@hdss7-11 ~]# rm -fr /data/zookeeper/data/* /data/zookeeper/logs/* # 删除历史数据
  4. [root@hdss7-11 ~]# sed -i '/^server/d' /opt/apps/zookeeper/conf/zoo.cfg # 删除集群配置
  6. [root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkServer.sh start
  7. [root@hdss7-11 ~]# /opt/apps/zookeeper/bin/zkServer.sh status  # 启动后，所有节点都为 standalone 模式
  8. ZooKeeper JMX enabled by default
  9. Using config: /opt/apps/zookeeper/bin/../conf/zoo.cfg
  10. Mode: standalone

2.3. 创建名称空间

[root@hdss7-22 ~]# kubectl create namespace pro
[root@hdss7-22 ~]# kubectl create namespace fat
[root@hdss7-22 ~]# kubectl create namespace dev
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username=admin --docker-password='Harbor12345' --docker-server=http://harbor.od.com -n pro
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username=admin --docker-password='Harbor12345' --docker-server=http://harbor.od.com -n fat
[root@hdss7-22 ~]# kubectl create secret docker-registry harbor --docker-username=admin --docker-password='Harbor12345' --docker-server=http://harbor.od.com -n dev

3. 部署Config Service

3.1. 初始化数据库

3.1.1. 刷SQL脚本

# 数据库服务器: hdss7-150.host.com  MySQL 5.7
[root@hdss7-150 src]# wget -O config.sql https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql
# SQL脚本中的调整: 
# 1. 不同环境分库；
# 2. 不同环境中 ApolloConfigDB.ServerConfig 表中Eureka地址调整
[root@hdss7-150 src]# sed -i 's#http://localhost:8080/eureka/#http://config.od.com/eureka/#' config.sql
[root@hdss7-150 src]# sed 's/ApolloConfigDB/ApolloConfigFatDB/;s#http://config.od.com/eureka/#http://config-fat.od.com/eureka/#' config.sql > config-fat.sql
[root@hdss7-150 src]# sed 's/ApolloConfigDB/ApolloConfigDevDB/;s#http://config.od.com/eureka/#http://config-dev.od.com/eureka/#' config.sql > config-dev.sql

[root@hdss7-150 src]# mysql -uroot -p < config.sql 
[root@hdss7-150 src]# mysql -uroot -p < config-fat.sql 
[root@hdss7-150 src]# mysql -uroot -p < config-dev.sql

# 授权，实践中不同环境的库可以使用不同的用户名和密码
# 因为Pod连接MySQL是做了SNAT，所以需要使用Node IP授权，如果需要显示Pod地址，可以添加路由表
mysql> grant SELECT,INSERT,UPDATE,DELETE on ApolloConfigDB.* to apolloconfig@'10.4.7.%' identified by 'Centos.1992!@#' ;
mysql> grant SELECT,INSERT,UPDATE,DELETE on ApolloConfigFatDB.* to apolloconfig@'10.4.7.%' identified by 'Centos.1992!@#' ;
mysql> grant SELECT,INSERT,UPDATE,DELETE on ApolloConfigDevDB.* to apolloconfig@'10.4.7.%' identified by 'Centos.1992!@#' ;
mysql> show grants for apolloconfig@'10.4.7.%';
+--------------------------------------------------------------------------------------------+
| Grants for apolloconfig@10.4.7.%                                                           |
+--------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'apolloconfig'@'10.4.7.%'                                            |
| GRANT SELECT, INSERT, UPDATE, DELETE ON `ApolloConfigDB`.* TO 'apolloconfig'@'10.4.7.%'    |
| GRANT SELECT, INSERT, UPDATE, DELETE ON `ApolloConfigFatDB`.* TO 'apolloconfig'@'10.4.7.%' |
| GRANT SELECT, INSERT, UPDATE, DELETE ON `ApolloConfigDevDB`.* TO 'apolloconfig'@'10.4.7.%' |
+--------------------------------------------------------------------------------------------+

3.1.2. 配置DNS解析

[root@hdss7-11 ~]# vim /var/named/od.com.zone 
......
config             A    10.4.7.10
config-fat         A    10.4.7.10
config-dev         A    10.4.7.10
mysql              A    10.4.7.150
[root@hdss7-11 ~]# systemctl restart named

3.2. 制作Config Service镜像

# 下载页面: https://github.com/ctripcorp/apollo/releases/tag/v1.5.1
[root@hdss7-200 src]# unzip apollo-configservice-1.5.1-github.zip -d docker_files/apollo-configservice
[root@hdss7-200 src]# cd docker_files/apollo-configservice
[root@hdss7-200 apollo-configservice]# ls config/ -l # 以下配置采用 configmap 挂载
total 8
-rw-r--r-- 1 root root 289 Nov  9 23:21 application-github.properties # DB信息
-rw-r--r-- 1 root root  30 Apr 20  2017 app.properties # appid，jdkVersion信息

#!/bin/bash
# 修改 scripts/startup.sh; 其中内存参数需要根据情况调整，此处增加了一个APOLLO_CONFIG_SERVICE_NAME，没有通过环境变量传递
# https://github.com/ctripcorp/apollo/tree/1.5.1/scripts/apollo-on-kubernetes/apollo-config-server/scripts
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)

SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms6144m -Xmx6144m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

[root@hdss7-200 apollo-configservice]# vim Dockerfile
# jre:8u112 制作地址: https://www.yuque.com/duduniao/ww8pmw/gp8n04#JJ6SE
# config 通过configmap来挂载, *sources.jar为源码，scritps/shutdown.sh 不需要
FROM harbor.od.com/public/jre:8u112
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone && mkdir -p /apollo-configservice/scripts
ADD apollo-configservice-1.5.1.jar /apollo-configservice/apollo-configservice.jar
ADD scripts/ /apollo-configservice/scripts
CMD ["/apollo-configservice/scripts/startup.sh"]
[root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
[root@hdss7-200 apollo-configservice]# docker image push harbor.od.com/infra/apollo-configservice:v1.5.1

3.3. 资源配置清单

3.3.1. configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: pro
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = Centos.1992!@#
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003171

3.3.2. deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apollo-configservice
  namespace: pro
  labels: 
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-configservice
  template:
    metadata:
      labels: 
        app: apollo-configservice 
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10

3.3.3. service

kind: Service
apiVersion: v1
metadata: 
  name: apollo-configservice
  namespace: pro
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: apollo-configservice

3.3.4. ingress

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-configservice
  namespace: pro
spec:
  rules:
  - host: config.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-configservice
          servicePort: 80

3.3.5. 其它环境中的配置清单

以上是生产环境pro中的清单，其它环境参考修改。修改点:

  • Namespace 属性修改
  • configmap中 eureka.service.url修改成对应环境地址、数据库名
  • ingress中域名的修改

3.4. 应用配置清单

Pro、Dev和Fat环境应用资源清单方式一致，检查 info location 与 config 页面是否正常即可。

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/configmap.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/deployment.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/service.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/configservice/ingress.yaml

[root@hdss7-22 ~]# curl -s 172.7.22.5:8080/info # 访问 configservice pod返回信息
{"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}

04-2-配置中心 - 图6

4. 部署Admin service

4.1. 制作adminservice 镜像

# 下载页面: https://github.com/ctripcorp/apollo/releases/tag/v1.5.1
[root@hdss7-200 src]# mkdir -p docker_files/apollo-adminservice
[root@hdss7-200 src]# unzip -q apollo-adminservice-1.5.1-github.zip -d docker_files/apollo-adminservice
[root@hdss7-200 src]# cd docker_files/apollo-adminservice

[root@hdss7-200 apollo-adminservice]# vim scripts/startup.sh 
#!/bin/bash
# 此处修改了 SERVER_PORT APOLLO_ADMIN_SERVICE_NAME
# 参考地址：https://github.com/ctripcorp/apollo/tree/master/scripts/apollo-on-kubernetes/apollo-admin-server/scripts
SERVICE_NAME=apollo-adminservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)

# SERVER_URL="http://localhost:${SERVER_PORT}"
SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

[root@hdss7-200 apollo-adminservice]# vim Dockerfile 
# jre:8u112 制作地址: https://www.yuque.com/duduniao/ww8pmw/gp8n04#JJ6SE
FROM harbor.od.com/public/jre:8u112
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone && mkdir -p /apollo-adminservice/scripts

ADD apollo-adminservice-1.5.1.jar /apollo-adminservice/apollo-adminservice.jar
ADD scripts/startup.sh /apollo-adminservice/scripts
CMD ["/apollo-adminservice/scripts/startup.sh"]
[root@hdss7-200 apollo-adminservice]# docker image build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
[root@hdss7-200 apollo-adminservice]# docker image push harbor.od.com/infra/apollo-adminservice:v1.5.1

4.2. 资源配置清单

4.2.1. configmap

# appId 与 configservice、portal 不一致
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: pro
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = Centos.1992!@#
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003172

4.2.2. deployment

# admimservice 不对外提供统一接入口，其它模块通过 Eureka 查询到具体admin service地址
kind: Deployment
apiVersion: apps/v1
metadata:
  name: apollo-adminservice
  namespace: pro
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        app: apollo-adminservice 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

4.2.3. 其它环境中的配置清单

以上是生产环境pro中的清单，其它环境参考修改。修改点:

  • Namespace 属性修改
  • configmap中 eureka.service.url修改成对应环境地址

4.3. 交付adminservice

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/adminservice/configmap.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/adminservice/deployment.yaml

[root@hdss7-22 ~]# kubectl get pod -n pro -o wide -l name=apollo-adminservice
NAME                                  READY   STATUS    RESTARTS   AGE    IP           NODE                NOMINATED NODE   READINESS GATES
apollo-adminservice-bffbf595c-hhcbq   1/1     Running   0          6m4s   172.7.22.5   hdss7-22.host.com   <none>           <none>
[root@hdss7-22 ~]# curl -s 172.7.22.5:8080/info
{"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}

04-2-配置中心 - 图7

04-2-配置中心 - 图8

5. 部署Portal

5.1. 初始化数据库

[root@hdss7-150 src]# wget -O portal.sql https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/portaldb/V1.0.0__initialization.sql

mysql> source /opt/src/portal.sql ;
grant SELECT,INSERT,UPDATE,DELETE on ApolloPortalDB.* to apolloportal@'10.4.7.%' identified by 'Centos.1992!@#' ;

5.2. 制作portal镜像

# 下载页面: https://github.com/ctripcorp/apollo/releases/tag/v1.5.1
[root@hdss7-200 src]# mkdir -p docker_files/apollo-portal
[root@hdss7-200 src]# unzip -q apollo-portal-1.5.1-github.zip -d docker_files/apollo-portal
[root@hdss7-200 src]# cd docker_files/apollo-portal
[root@hdss7-200 apollo-portal]# ls config/ -l
total 12
-rw-r--r-- 1 root root 234 Nov  9 23:21 apollo-env.properties
-rw-r--r-- 1 root root 218 Nov  9 23:21 application-github.properties
-rw-r--r-- 1 root root  30 Apr 20  2017 app.properties
[root@hdss7-200 apollo-portal]# cat config/apollo-env.properties # 不同环境配置
local.meta=http://localhost:8080               # 本地环境
dev.meta=http://fill-in-dev-meta-server:8080   # 开发环境:Development environment
fat.meta=http://fill-in-fat-meta-server:8080   # 测试环境:Feature Acceptance Test environment
uat.meta=http://fill-in-uat-meta-server:8080   # 预生产环境:User Acceptance Test environment
lpt.meta=${lpt_meta}
pro.meta=http://fill-in-pro-meta-server:8080   # 生产环境:Production environment
# 目前支持的变量有: LOCAL, DEV, FWS, FAT, UAT, LPT, PRO, TOOLS, UNKNOWN
# reference: https://github.com/ctripcorp/apollo/blob/master/apollo-core/src/main/java/com/ctrip/framework/apollo/core/enums/Env.java

[root@hdss7-200 apollo-portal]# vim scripts/startup.sh
#!/bin/bash
# 此处修改了 SERVER_PORT APOLLO_ADMIN_SERVICE_NAME
# 参考地址：https://github.com/ctripcorp/apollo/blob/1.5.1/scripts/apollo-on-kubernetes/apollo-portal-server/scripts/startup-kubernetes.sh
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)

# SERVER_URL="http://localhost:$SERVER_PORT"
SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

[root@hdss7-200 apollo-portal]# vim Dockerfile 
# jre:8u112 制作地址: https://www.yuque.com/duduniao/ww8pmw/gp8n04#JJ6SE
# config 通过configmap来挂载, *sources.jar为源码，scritps/shutdown.sh 不需要
FROM harbor.od.com/public/jre:8u112
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone && mkdir -p /apollo-portal/scripts
ADD apollo-portal-1.5.1.jar /apollo-portal/apollo-portal.jar
ADD scripts/startup.sh /apollo-portal/scripts
CMD ["/apollo-portal/scripts/startup.sh"]
[root@hdss7-200 apollo-portal]# docker image build . -t harbor.od.com/infra/apollo-portal:v1.5.1
[root@hdss7-200 apollo-portal]# docker image push harbor.od.com/infra/apollo-portal:v1.5.1

5.3. 资源配置清单

5.3.1. configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-portal-cm
  namespace: infra
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = Centos.1992!@#
  app.properties: |
    appId=100003173
  apollo-env.properties: |
    pro.meta=http://config.od.com
    fat.meta=http://config-fat.od.com
    dev.meta=http://config-dev.od.com

5.3.2. deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apollo-portal
  namespace: infra
  labels: 
    name: apollo-portal
spec:
  # 当前的负载均衡方式(IPVS-nq) 不支持多台 portal，否则会出现会话异常
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-portal
  template:
    metadata:
      labels: 
        app: apollo-portal 
        name: apollo-portal
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-portal-cm
      containers:
      - name: apollo-portal
        image: harbor.od.com/infra/apollo-portal:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-portal/config
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10

5.3.3. service

apiVersion: v1
kind: Service
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: apollo-portal

5.3.4. ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  rules:
  - host: portal.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-portal
          servicePort: 80

5.3.5. DNS配置

[root@hdss7-11 ~]# vim /var/named/od.com.zone
......
portal             A    10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# dig -t A portal.od.com @10.4.7.11 +short
10.4.7.10

5.4. 交付Portal

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/configmap.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/apollo/portal/ingress.yaml

04-2-配置中心 - 图9

04-2-配置中心 - 图10

6. Apollo Portal使用

6.1. 创建多环境

04-2-配置中心 - 图11

04-2-配置中心 - 图12

04-2-配置中心 - 图13

6.2. 创建部门

mysql> select id,`key`,value from ApolloPortalDB.ServerConfig; # key 是保留字，需要用`key`才能查询
+----+-------------------------------------+-------------------------------------------------------------------------------------------+
| id | key                                 | value                                                                                     |
+----+-------------------------------------+-------------------------------------------------------------------------------------------+
|  1 | apollo.portal.envs                  | pro,fat,dev                                                                               |
|  2 | organizations                       | [{"orgId":"TEST1","orgName":"样例部门1"},{"orgId":"TEST2","orgName":"样例部门2"}]         |
|  3 | superAdmin                          | apollo                                                                                    |
|  4 | api.readTimeout                     | 10000                                                                                     |
|  5 | consumer.token.salt                 | someSalt                                                                                  |
|  6 | admin.createPrivateNamespace.switch | true                                                                                      |
|  7 | configView.memberOnly.envs          | pro                                                                                       |
+----+-------------------------------------+-------------------------------------------------------------------------------------------+

通过【系统参数】入口，可修改系统参数，参数表为 ApolloPortalDB.ServerConfig，常修改部门信息。

04-2-配置中心 - 图14

6.3. 创建项目

6.3.1. 创建管理用户

04-2-配置中心 - 图15

6.3.2. Dubbo微服务提供者

04-2-配置中心 - 图16

04-2-配置中心 - 图17

04-2-配置中心 - 图18

6.3.3. Dubbo微服务消费者

04-2-配置中心 - 图19

04-2-配置中心 - 图20

6.4. 多环境中使用Apollo

6.4.1. 编译和打包镜像

04-2-配置中心 - 图21

04-2-配置中心 - 图22

6.4.2. 应用提供者到K8S集群

# 注意：C_OPTS 环境变量为configservice地址，不同环境中namespace不一致
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dubbo-demo-service
  namespace: pro
  labels: 
    name: dubbo-demo-service
    tier: provider
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:apollo_20200205_1342
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://config.od.com
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-service/deployment.yaml

# 检查是否连接到apollo和配置是否正确，通过看日志来实现
[root@hdss7-22 ~]# kubectl get pod -n pro
NAME                                    READY   STATUS    RESTARTS   AGE
apollo-adminservice-bffbf595c-g88wz     1/1     Running   0          116m
apollo-configservice-5bf6df98df-2b458   1/1     Running   0          116m
dubbo-demo-service-7f6d79b894-fhgqb     1/1     Running   0          5m1s
[root@hdss7-22 ~]# kubectl logs dubbo-demo-service-7f6d79b894-fhgqb -n pro | grep -Eo 'config(-...)?\.od\.com|zk.\.od.com'|sort|uniq
config.od.com
zk1.od.com

04-2-配置中心 - 图23

6.4.3. 应用消费者到K8S集群

# 当前的deployment是在前一版本基础上改了镜像和增加环境变量，以及名称空间
# 前一版本参考 https://www.yuque.com/duduniao/ww8pmw/gp8n04#CYcLG
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dubbo-demo-consumer
  namespace: pro
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:apollo_20200203_1346
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://config.od.com
      imagePullSecrets:
      - name: harbor
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

# 当前的service是在前一版本基础上改了名称空间
# 前一版本参考 https://www.yuque.com/duduniao/ww8pmw/gp8n04#CYcLG
apiVersion: v1
kind: Service
metadata: 
  name: dubbo-demo-consumer
  namespace: pro
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer

# 当前的ingress是在前一版本基础上改了名称空间
# 前一版本参考 https://www.yuque.com/duduniao/ww8pmw/gp8n04#CYcLG 
apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  name: dubbo-demo-consumer
  namespace: pro
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-demo-consumer
          servicePort: 80

# 三种环境中DNS配置
[root@hdss7-11 ~]# vim /var/named/od.com.zone 
......
config             A    10.4.7.10
config-fat         A    10.4.7.10
config-dev         A    10.4.7.10
[root@hdss7-11 ~]# systemctl restart named

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-consumer/deployment.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-consumer/service.yaml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/pro/dubbo-demo-consumer/ingress.yaml

[root@hdss7-22 ~]# kubectl get pod -n pro -l name=dubbo-demo-consumer
NAME                                 READY   STATUS    RESTARTS   AGE
dubbo-demo-consumer-6cb7dfd8-h8ldl   1/1     Running   0          112s
[root@hdss7-22 ~]# kubectl logs dubbo-demo-consumer-6cb7dfd8-h8ldl -n pro | grep -oE 'config(-...)?\.od\.com|zk.\.od.com'|sort|uniq
config.od.com
zk1.od.com

04-2-配置中心 - 图24

04-2-配置中心 - 图25

6.4.4. 其它环境部署方式

不同的环境使用相同的镜像，区别在于名称空间不同、Pod环境变量C_OPTS的configservice域名不同