项目git地址：https://github.com/callicoder/spring-security-react-ant-design-polls-app
难点在于jenkins的jenkinsfile编写，如何自动的打包编译，构建镜像，最后发布到k8s上，都希望提交代码后能做到自动发布，这就需要将那些变量提前设置好，尽量做到，下次提交代码后还能自动打包编译。

架构图

image.png

image.png

环境准备

mysql，gitlab，jenkins不运行在容器中，单独运行在机器上，容器仓库用的是官方dockerhub库

IP Hostname 应用 CPU Memory
10.140.0.3 master k8s，mysql 2 4G
10.140.0.4 node01 k8s，jenkins 2 4G
10.140.0.5 node02 k8s，gitlab 2 4G
10.140.0.6 node03 k8s 2 4G

准备k8s集群

创建一个k8s集群
image.png

k8s中建立2个namespce

生产环境常用的test，dev环境

  1. kubectl  create ns  test
  2. kubectl  create ns  dev

准备mysql

在master上部署数据库mysql5.7

  1. wget -c https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm
  2. rpm -ivh mysql80-community-release-el7-1.noarch.rpm
  3. yum -y install yum-utils
  4. yum-config-manager --disable mysql80-community
  5. yum-config-manager --enable mysql57-community
  6. yum install mysql-community-server -y
  7. # 启动mysql
  8. systemctl start mysqld
  9. # 开机启动
  10. systemctl enable mysqld
  11. # 查看root临时密码
  12. grep 'temporary password' /var/log/mysqld.log
  13. #登陆数据库
  14. mysql -uroot -p'xxxxxx'
  15. # 使用mysql临时登录，修改root密码
  16. ALTER USER 'root'@'localhost' IDENTIFIED BY '123456@Abc';

配置mysql 参数

在/etc/my.cnf添加以下参数

  1. sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
  2. skip-name-resolve
  3. lower_case_table_names = 1
  4. max_connections = 4000
  5. character-set-server = utf8mb4  
  6. collation-server = utf8mb4_unicode_ci  
  7. init_connect='SET NAMES utf8mb4'  
  8. skip-character-set-client-handshake = true  
  9. [mysql]  
  10. default-character-set = utf8mb4

image.png

重启mysql

  1. systemctl restart mysqld

登陆数据库

  1. mysql -uroot -p'123456@Abc'

授权

  1. GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456@Abc';

建库给后端app用

  1. create database polling_app;

准备jenkins

在node01上安装jenkins。
jenkins机器需要git，java，maven，nodejs，docker，kubelet这些在代码编译打包都得用到

基础yum源

  1. yum install -y epel-release 
  2. yum install -y wget  bash-com*  git
  3. yum update -y

docker yum源

  1. wget -P /etc/yum.repos.d/  https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

kubeadm yum源

  1. cat <<EOF >/etc/yum.repos.d/kubernetes.repo
  2. [kubernetes]
  3. name=Kubernetes
  4. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
  5. enabled=1
  6. gpgcheck=0
  7. EOF

安装java

  1. yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel

安装maven

  1. wget -P /home https://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
  3. cd /home
  5. tar xvf apache-maven-3.6.3-bin.tar.gz
  1. cat > /etc/profile.d/maven.sh << EOF
  2. export MAVEN_HOME=/home/apache-maven-3.6.3
  3. export PATH=$PATH:${MAVEN_HOME}/bin
  4. EOF
  1. source /etc/profile

安装nodejs

  1. wget -P /home https://nodejs.org/dist/v14.8.0/node-v14.8.0-linux-x64.tar.xz
  3. cd /home
  5. tar xvf node-v14.8.0-linux-x64.tar.xz
  1. cat > /etc/profile.d/nodejs.sh << EOF
  2. export NODE_HOME=/home/node-v14.8.0-linux-x64
  3. export PATH=$NODE_HOME/bin:$PATH
  4. EOF
  1. source /etc/profile

安装docker

  1. yum -y install docker-ce
  3. systemctl enable --now docker

安装kubelet

  1. yum -y install kubelet

复制master机器上的.kube目录

  1. scp  -r master:/root/.kube  /root

下载tomcat和jenkins war包

tomcat包下载地址

  1. https://mirrors.bfsu.edu.cn/apache/tomcat/tomcat-8/

这里用8.5.57，可能会被更新掉

  1. wget -P /opt   https://mirrors.bfsu.edu.cn/apache/tomcat/tomcat-8/v8.5.57/bin/apache-tomcat-8.5.57.tar.gz
  3. cd /opt
  5. tar xvf apache-tomcat-8.5.57.tar.gz
  7. rm -rf  /opt/apache-tomcat-8.5.57/webapps/*
  9. wget -P /opt/apache-tomcat-8.5.57/webapps/ https://get.jenkins.io/war-stable/2.222.4/jenkins.war
  11. cd /opt/apache-tomcat-8.5.57/webapps/
  13. mv jenkins.war  ROOT.war

image.png

启动jenkins

  1. sh /opt/apache-tomcat-8.5.57/bin/startup.sh

访问jenkins，ip:8080

  1. cat /root/.jenkins/secrets/initialAdminPassword

image.png

安装插件

image.png
image.png
image.png

image.png

  1. Git Parameter

安装插件重启即可

image.png

安装gitlab

node02上部署gitlab

准备yum

  1. cat > /etc/yum.repos.d/gitlab-ce.repo << EOF
  2. [gitlab-ce]
  3. name=Gitlab CE Repository
  4. baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/
  5. gpgcheck=0
  6. enabled=1
  7. EOF
  1. yum install gitlab-ce git

编辑配置文件

  1. vim /etc/gitlab/gitlab.rb
  1. ##配置访问url
  2. external_url 'xxxxx'
  4. ##配置邮箱（可选）
  5. gitlab_rails['smtp_enable'] = true
  6. gitlab_rails['smtp_address'] = "smtp.exmail.qq.com"
  7. gitlab_rails['smtp_port'] = 465
  8. gitlab_rails['smtp_user_name'] = "tangwei@tk8s.com"
  9. gitlab_rails['smtp_password'] = "邮箱密码"
  10. gitlab_rails['smtp_domain'] = "tk8s.com"
  11. gitlab_rails['smtp_authentication'] = "login"
  12. gitlab_rails['smtp_tls'] = true
  13. gitlab_rails['smtp_enable_starttls_auto'] = true
  14. gitlab_rails['gitlab_email_from'] = "tangwei@tk8s.com"
  15. user['git_user_email'] = "tangwei@tk8s.com"
  16. gitlab_rails['gitlab_email_reply_to'] = 'tangwei@tk8s.com'

image.png
image.png

加载配置文件

  1. gitlab-ctl reconfigure

启动gitlab

  1. gitlab-ctl restart

访问gitlab

第一次登陆修改root密码

image.png

新建一个后端项目polling-app-server

visibility level这块选择internal 只允许登陆的用户访问
image.png

新建一个前端项目polling-app-client

visibility level这块选择internal 只允许登陆的用户访问
image.png

准备项目

在node01上操作

设置git参数

  1. git config --global user.name "Administrator"
  2. git config --global user.email "admin@example.com"

下载源项目

  1. git clone https://github.com/callicoder/spring-security-react-ant-design-polls-app

image.png

从gitlab下载空的后端项目,输入gitlab的root账号密码

  1. git clone http://35.221.252.82/root/polling-app-server.git

从gitlab下载空的前端项目，输入gitlab的root账号密码

用http的url拉取

  1. git clone http://35.221.252.82/root/polling-app-client.git

image.png

拷贝源目录中的前端和后端项目到gitlab项目中

用http的url拉取

  1. mv spring-security-react-ant-design-polls-app/polling-app-server/*  polling-app-server/
  2. mv spring-security-react-ant-design-polls-app/polling-app-client/*  polling-app-client/

image.png

修改后端数据库信息

  1. vim /root/polling-app-server/src/main/resources/application.properties

image.png

上传后端到gitlab

  1. cd /root/polling-app-server/
  2. git add .
  3. git commit -m "upload server"
  4. git push -u origin master

image.png

上传前端到gitlab

  1. cd /root/polling-app-client/
  2. git add .
  3. git commit -m "upload client"
  4. git push -u origin master

image.png

总结：前期准备完成，和实际生产环境一样，gitlab中项目先准备好，jenkins准备好，k8s环境准备好。

部署后端到k8s

jenkins添加gitlab和dockerhub的登陆账号密码

image.png
image.png

添加gitlab root账号密码，id为gitlab，后面需要用到

image.png

添加dockerhub的账号密码，id为jenkins-harbor-creds，后面需要用到

image.png

建立流水线

image.png
image.png
image.png

设置参数

需要设置4个string parameters，1个git parameters 1个choice parametes，这些变量需要用到
image.png

image.png

设置git

image.png
image.png

完整pipeline配置

FireShot Capture 003 - polling-app-server Config [Jenkins] - 35.194.187.183.png

进入gitlab中web ide。添加3个文件

image.png
image.png

编写Dockerfile

  1. FROM tanmgweiwow/jdkenv:v1.0
  3. ARG JAR_FILE
  4. ARG WORK_PATH="/opt/"
  5. #   环境变量
  6. ENV JAVA_OPTS="-server -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Xdebug -Xnoagent -Djava.compiler=NONE -XX:MaxNewSize=256m -Dsun.net.inetaddr.ttl=3 -Dsun.net.inetaddr.negative.ttl=1 -Dsun.misc.URLClassPath.disableJarChecking=true -XX:+UseParNewGC -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:+ScavengeBeforeFullGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSPermGenSweepingEnabled -XX:CMSInitiatingPermOccupancyFraction=70 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8" \
  7.     JAR_FILE=$JAR_FILE \
  8.     LANG=en_US.UTF-8 \
  9.     LANGUAGE=en_US:en \
  10.     LC_ALL=en_US.UTF-8 \
  11.     TZ=Asia/Shanghai
  13. COPY $JAR_FILE $WORK_PATH/
  15. WORKDIR $WORK_PATH
  17. ENTRYPOINT exec java $JAVA_OPTS -jar $JAR_FILE

编写Jenkinsfile

这里流程大致为
image.png

  1. pipeline {
  2.     agent any
  3.     environment {
  4.         HARBOR_CREDS = credentials('jenkins-harbor-creds')
  5.         GIT_TAG = sh(returnStdout: true,script: 'git describe --tags --always').trim()
  6.     }
  8.     stages {
  9.         stage('Maven Build') {
  10.             when { expression { env.GIT_TAG != null } }
  11.             steps {
  12.                sh 'mvn clean' 
  13.                sh 'mvn clean compile -U package -Dmaven.test.skip=true'
  14.             }
  16.         }
  17.         stage('Docker Build') {
  18.             when { 
  19.                 allOf {
  20.                     expression { env.GIT_TAG != null }
  21.                 }
  22.             }
  24.              steps {
  25.                 sh "docker login -u ${HARBOR_CREDS_USR} -p ${HARBOR_CREDS_PSW}"
  26.                 sh "cp -f /root/.jenkins/workspace/${ENV_NAME}/target/${JAR}  /root/.jenkins/workspace/${ENV_NAME}/"
  27.                 sh "cd /root/.jenkins/workspace/${ENV_NAME}/"
  28.                 sh "docker build  --build-arg JAR_FILE=${JAR} -t ${DOCKER_IMAGE}:${APP_NAME}_${GIT_TAG} . "
  29.             }    
  30.         }
  32.         stage('Docker Push') {
  33.             when { 
  34.                 allOf {
  35.                     expression { env.GIT_TAG != null }
  36.                 }
  37.             }
  39.             steps {
  40.                 sh "docker push ${DOCKER_IMAGE}:${APP_NAME}_${GIT_TAG}"
  41.                 sh "docker rmi  ${DOCKER_IMAGE}:${APP_NAME}_${GIT_TAG}"
  42.             }
  44.         }
  46.         stage('Deploy On K8S') {
  47.             when { 
  48.                 allOf {
  49.                     expression { env.GIT_TAG != null }
  50.                 }
  51.             }
  52.             steps {
  53.                 sh "cd /root/.jenkins/workspace/${ENV_NAME}/"
  54.                 sh """
  55.                 if [ -f "k8s-templete.yml" ];then
  56.                     kubectl delete -f k8s-templete.yml --namespace=${K8S_NAMESPACE} || sleep 1
  57.                 fi
  58.                 """
  59.                 sh "sed -e 's#{IMAGE_URL}#${DOCKER_IMAGE}#g;s#{IMAGE_TAG}#${APP_NAME}_${GIT_TAG}#g;s#{APP_NAME}#${APP_NAME}-${GIT_TAG}#g;s#{APP}#${APP_NAME}#g'  k8s-templete.tpl > k8s-templete.yml"
  60.                 sh "kubectl apply -f k8s-templete.yml --namespace=${K8S_NAMESPACE}"
  61.                 sh "rm -f ${JAR}"
  62.             }
  64.         }
  66.     }
  67. }

编写 k8s-templete.tpl

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: {APP_NAME}-deployment
  5.   labels:
  6.     app: {APP_NAME}
  7. spec:
  8.   replicas: 2
  9.   selector:
  10.     matchLabels:
  11.       app: {APP}
  12.       app-name: {APP_NAME}
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: {APP}
  17.         app-name: {APP_NAME}
  18.     spec:
  19.       containers:
  20.       - name: {APP_NAME}
  21.         image: {IMAGE_URL}:{IMAGE_TAG}
  22.         ports:
  23.         - name: http
  24.           containerPort: 8080
  25.         resources:
  26.           limits:
  27.             memory: 2Gi
  28. ---
  29. ---
  30. apiVersion: v1
  31. kind: Service
  32. metadata:
  33.   name: {APP}-svc
  34.   labels:
  35.     app: {APP}
  36. spec:
  37.   selector:
  38.     app: {APP}
  39.   type: NodePort
  40.   ports:
  41.   - port: 8080
  42.     targetPort: 8080
  43.     protocol: TCP
  44.     name: http
  45.     nodePort: 32682

commit提交，会自动建立一个新的分支

image.png

jenkins构建后端

image.png
image.png

后端构建成功

image.png
image.png
记住后端的svc，这个前端需要调用 http://nodeip:32682/api

部署前端到k8s中

建立流水线

image.png

image.png

设置参数

需要设置4个string parameters，1个git parameters 1个choice parametes，这些变量需要用到
image.png
image.png
image.png
image.png

设置git

image.png

完整pipeline配置

FireShot Capture 006 - polling-app-client Config [Jenkins] - 35.194.187.183.png

进入gitlab中web ide。添加4个文件

image.png

编写Dockerfile

  1. FROM node:12.4.0-alpine as build
  4. WORKDIR /app
  7. COPY package.json package-lock.json ./
  8. RUN npm install
  10. # Copy the main application
  11. COPY . ./
  13. # Arguments
  14. ARG REACT_APP_API_BASE_URL
  15. ENV REACT_APP_API_BASE_URL=$REACT_APP_API_BASE_URL
  17. # Build the application
  18. RUN npm run build
  20. #### Stage 2: Serve the React application from Nginx 
  21. FROM nginx:latest
  23. COPY --from=build /app/build /var/www
  25. # Copy our custom nginx config
  26. COPY nginx.conf /etc/nginx/nginx.conf
  28. # Expose port 3000 to the Docker host, so we can access it 
  29. # from the outside.
  30. EXPOSE 80
  32. ENTRYPOINT ["nginx","-g","daemon off;"]

编写Jenkinsfile

  1. pipeline {
  2.     agent any
  3.     environment {
  4.         HARBOR_CREDS = credentials('jenkins-harbor-creds')
  5.         GIT_TAG = sh(returnStdout: true,script: 'git describe --tags --always').trim()
  6.     }
  7.     stages {
  8.         stage('NPM build and Docker Build') {
  9.             when { 
  10.                 allOf {
  11.                     expression { env.GIT_TAG != null }
  12.                 }
  13.             }
  15.             steps {
  16.                 sh "docker login -u ${HARBOR_CREDS_USR} -p ${HARBOR_CREDS_PSW}"
  17.                 sh "docker build  --build-arg  REACT_APP_API_BASE_URL=${REACT_APP_API_BASE_URL}  -t ${DOCKER_IMAGE}:${APP_NAME}_${GIT_TAG} . "
  18.             }
  20.         }
  22.         stage('Docker Push') {
  23.             when { 
  24.                 allOf {
  25.                     expression { env.GIT_TAG != null }
  26.                 }
  27.             }
  29.             steps {
  30.                 sh "docker push ${DOCKER_IMAGE}:${APP_NAME}_${GIT_TAG}"
  31.                 sh "docker rmi  ${DOCKER_IMAGE}:${APP_NAME}_${GIT_TAG}"
  32.             }
  34.         }
  35.         stage('Deploy On K8S') {
  36.             when { 
  37.                 allOf {
  38.                     expression { env.GIT_TAG != null }
  39.                 }
  40.             }
  41.             steps {
  42.                 sh "cd /root/.jenkins/workspace/${ENV_NAME}/"
  43.                 sh """
  44.                 if [ -f "k8s-deployment.yml" ];then
  45.                     sudo kubectl delete -f k8s-deployment.yml --namespace=${K8S_NAMESPACE} || sleep 1
  46.                 fi
  47.                 """
  48.                 sh "sed -e 's#{IMAGE_URL}#${DOCKER_IMAGE}#g;s#{IMAGE_TAG}#${APP_NAME}_${GIT_TAG}#g;s#{APP_NAME}#${APP_NAME}-${GIT_TAG}#g;s#{APP}#${APP_NAME}#g'  k8s-deployment.tpl > k8s-deployment.yml"
  49.                 sh "sudo kubectl apply -f k8s-deployment.yml --namespace=${K8S_NAMESPACE}"
  50.             }
  52.         }   
  53.     }
  54. }

编写k8s-deployment.tpl

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: {APP_NAME}-deployment
  5.   labels:
  6.     app: {APP_NAME}
  7. spec:
  8.   replicas: 2
  9.   selector:
  10.     matchLabels:
  11.       app: {APP}
  12.       app-name: {APP_NAME}
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: {APP}
  17.         app-name: {APP_NAME}
  18.     spec:
  19.       containers:
  20.       - name: {APP_NAME}
  21.         image: {IMAGE_URL}:{IMAGE_TAG}
  22.         ports:
  23.         - name: http
  24.           containerPort: 80
  25.         resources:
  26.           limits:
  27.             memory: 1Gi
  28. ---
  29. apiVersion: v1
  30. kind: Service
  31. metadata:
  32.   name: {APP}-svc
  33.   labels:
  34.     app: {APP}
  35. spec:
  36.   selector:
  37.     app: {APP}
  38.   type: NodePort
  39.   ports:
  40.   - port: 80
  41.     targetPort: 80
  42.     protocol: TCP
  43.     name: http
  44.     nodePort: 32680

编写nginx.conf

  1. error_log stderr notice;
  3. worker_processes 2;
  4. worker_rlimit_nofile 130048;
  5. worker_shutdown_timeout 10s;
  7. events {
  8.   multi_accept on;
  9.   use epoll;
  10.   worker_connections 16384;
  11. }
  13. http {
  14.   default_type  application/octet-stream;
  15.   include       /etc/nginx/mime.types;
  16.   aio threads;
  17.   aio_write on;
  18.   tcp_nopush on;
  19.   tcp_nodelay on;
  21.   keepalive_timeout 5m;
  22.   keepalive_requests 100;
  23.   reset_timedout_connection on;
  24.   server_tokens off;
  25.   autoindex off;
  27.   server {
  28.     listen 8081;
  29.     location /healthz {
  30.       access_log off;
  31.       return 200;
  32.     }
  33.     location /stub_status {
  34.       stub_status on;
  35.       access_log off;
  36.     }
  37.   }
  40.     server {
  41.         listen 80;
  42.         root /var/www;
  43.         index index.html index.htm;
  45.         location / {
  46.             # First attempt to serve request as file, then
  47.             # as directory, then fall back to redirecting to index.html
  48.             try_files $uri $uri/ /index.html;
  49.         }
  51.         # Media: images, icons, video, audio, HTC
  52.         location ~* \.(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ {
  53.           expires 1M;
  54.           access_log off;
  55.           add_header Cache-Control "public";
  56.         }
  58.         # Javascript and CSS files
  59.         location ~* \.(?:css|js)$ {
  60.             try_files $uri =404;
  61.             expires 1y;
  62.             access_log off;
  63.             add_header Cache-Control "public";
  64.         }
  66.         # Any route containing a file extension (e.g. /devicesfile.js)
  67.         location ~ ^.+\..+$ {
  68.             try_files $uri =404;
  69.         }
  70.     }
  71. }

commit提交，会自动建立一个新的分支

image.png
image.png

jenkins构建前端

image.png
image.png

前端构建成功

image.png

image.png

访问

通过nodeport访问前端 nodeip:32080

image.png

注册一个账号测试

image.png

登陆

image.png

创建投票

image.png
image.png

如果有ingress的话，可以编写ingress访问前端

  1. cat > ingress.yaml <<EOF
  2. apiVersion: extensions/v1beta1
  3. kind: Ingress
  4. metadata:
  5.   name: poll-ingress
  6.   namespace: test
  7.   annotations: 
  8.     kubernetes.io/ingress.class: "nginx"
  9. spec:
  10.   rules:
  11.   - host: poll.tk8s.com
  12.     http:
  13.       paths:
  14.       - path: /
  15.         backend:
  16.           serviceName: polling-app-client-svc
  17.           servicePort: 80
  18. EOF

image.png
然后域名绑定公网ip即可
image.png