前置工作

安装docker

docker安装

安装minikube

参考官网：https://minikube.sigs.k8s.io/docs/start/

image.png

下载及安装

  1. curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
  2. sudo install minikube-linux-amd64 /usr/local/bin/minikube

配置当前会话支持 kubectl 命令

  1. alias kubectl="minikube kubectl --"

安装helm

参考文章：https://blog.51cto.com/u_12099683/4538065 官网：https://helm.sh/docs/intro/install/

下载及安装

  1. wget https://get.helm.sh/helm-v3.5.4-linux-amd64.tar.gz
  2. tar -zxvf helm-v3.5.4-linux-amd64.tar.gz 
  3. cd linux-amd64/
  4. cp helm /usr/bin/helm

查看版本

  1. helm version

添加仓库

  1. helm repo add stable http://mirror.azure.cn/kubernetes/charts
  2. helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
  3. helm repo add flink-operator-repo https://archive.apache.org/dist/flink/flink-kubernetes-operator-0.1.0/
  4. helm repo update

查看仓库

  1. helm repo list

image.png

指定k8s集群位置

  1. export KUBECONFIG=/root/.kube/config

简单测试

  1. #生成chart文件
  2. helm create one
  3. #打包
  4. helm package one
  5. #安装
  6. helm install one one-0.1.0.tgz

安装seatunnel

参考官网：https://seatunnel.apache.org/docs/2.1.1/start/kubernetes

seatunnel版本：2.1.1

启动minikube

指定k8s版本为1.23.3

  1. minikube start --kubernetes-version=v1.23.3 --force

下载镜像

编辑Dockerfile

  1. vi Dockerfile
  1. FROM flink:1.13
  3. ENV SEATUNNEL_VERSION="2.1.0"
  5. RUN wget https://archive.apache.org/dist/incubator/seatunnel/${SEATUNNEL_VERSION}/apache-seatunnel-incubating-${SEATUNNEL_VERSION}-bin.tar.gz
  6. RUN tar -xzvf apache-seatunnel-incubating-${SEATUNNEL_VERSION}-bin.tar.gz
  8. RUN mkdir -p $FLINK_HOME/usrlib
  9. RUN cp apache-seatunnel-incubating-${SEATUNNEL_VERSION}/lib/seatunnel-core-flink.jar $FLINK_HOME/usrlib/seatunnel-core-flink.jar
  11. RUN rm -fr apache-seatunnel-incubating-${SEATUNNEL_VERSION}*

构建

  1. docker build -t seatunnel:2.1.0-flink-1.13 -f Dockerfile .

加载镜像到minikube

  1. minikube image load seatunnel:2.1.0-flink-1.13

k8s安装cert文件

  1. #下载
  2. wget https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml
  3. #部署
  4. kubectl create -f cert-manager.yaml

helm安装Flink Kubernetes Operator

  1. helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-0.1.0/
  3. helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator

验证

验证pod进程

  1. #alias kubectl="minikube kubectl --"
  2. kubectl get pod -A

image.png

验证seatunnel应用

构造虚拟数据流

  1. vi flink.streaming.conf
  2. #新增如下内容
  3. env {
  4.  execution.parallelism = 1
  5. }
  7. source {
  8.    FakeSourceStream {
  9.      result_table_name = "fake"
  10.      field_name = "name,age"
  11.    }
  12. }
  14. transform {
  15.    sql {
  16.      sql = "select name,age from fake"
  17.    }
  18. }
  20. sink {
  21.  ConsoleSink {}
  22. }

挂载配置文件到pod中

pod中创建目录

  1. minikube ssh
  3. sudo mkdir /mnt/data

拷贝文件

  1. minikube cp flink.streaming.conf /mnt/data/flink.streaming.conf

编写seatunnel的yaml部署文件

  1. vi seatunnel-flink.yaml

新增如下内容

  1. apiVersion: flink.apache.org/v1alpha1
  2. kind: FlinkDeployment
  3. metadata:
  4.   namespace: default
  5.   name: seatunnel-flink-streaming-example
  6. spec:
  7.   image: seatunnel:2.1.0-flink-1.13
  8.   #image: ren:8088/keyboardone/seatunnel:2.1.0-flink-1.13
  9.   flinkVersion: v1_14
  10.   flinkConfiguration:
  11.     taskmanager.numberOfTaskSlots: "2"
  12.   serviceAccount: flink
  13.   jobManager:
  14.     replicas: 1
  15.     resource:
  16.       memory: "2048m"
  17.       cpu: 1
  18.   taskManager:
  19.     resource:
  20.       memory: "2048m"
  21.       cpu: 2
  22.   podTemplate:  
  23.     spec:
  24.       containers:
  25.         - name: flink-main-container
  26.           volumeMounts:
  27.             - mountPath: /data
  28.               name: config-volume
  29.       volumes:
  30.         - name: config-volume
  31.           hostPath:
  32.             path: "/mnt/data"
  33.             type: Directory
  35.   job:
  36.     jarURI: local:///opt/flink/usrlib/seatunnel-core-flink.jar
  37.     entryClass: org.apache.seatunnel.SeatunnelFlink
  38.     args: ["--config", "/data/flink.streaming.conf"]
  39.     parallelism: 2
  40.     upgradeMode: stateless

执行部署

  1. kubectl apply -f seatunnel-flink.yaml

验证

验证服务

  1. kubectl get svc

image.png

查看日志

  1. kubectl logs -f deploy/seatunnel-flink-streaming-example

暴露端口

  1. kubectl port-forward svc/seatunnel-flink-streaming-example-rest 8081

Now the Flink Dashboard is accessible at localhost:8081.

打印日志

  1. kubectl logs \
  2. -l 'app in (seatunnel-flink-streaming-example), component in (taskmanager)' \
  3. --tail=-1 \
  4. -f

image.png

删除部署

  1. kubectl delete -f seatunnel-flink.yaml

删除minikube集群

  1. minikube stop
  2. minikube delete --all