前置工作

安装docker

docker安装

安装k8s

脚本安装-跨云创建k8s集群(1.20.9)

安装helm

helm安装

安装seatunnel

参考官网：https://seatunnel.apache.org/docs/2.1.1/start/kubernetes

seatunnel版本：2.1.1

下载镜像

编辑Dockerfile

  1. vi Dockerfile
  1. FROM flink:1.13
  3. ENV SEATUNNEL_VERSION="2.1.0"
  5. RUN wget https://archive.apache.org/dist/incubator/seatunnel/${SEATUNNEL_VERSION}/apache-seatunnel-incubating-${SEATUNNEL_VERSION}-bin.tar.gz
  6. RUN tar -xzvf apache-seatunnel-incubating-${SEATUNNEL_VERSION}-bin.tar.gz
  8. RUN mkdir -p $FLINK_HOME/usrlib
  9. RUN cp apache-seatunnel-incubating-${SEATUNNEL_VERSION}/lib/seatunnel-core-flink.jar $FLINK_HOME/usrlib/seatunnel-core-flink.jar
  11. RUN rm -fr apache-seatunnel-incubating-${SEATUNNEL_VERSION}*

构建

  1. docker build -t seatunnel:2.1.0-flink-1.13 -f Dockerfile .

加载镜像到minikube

  1. minikube image load seatunnel:2.1.0-flink-1.13

k8s安装cert文件

  1. #下载
  2. wget https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml
  3. #部署
  4. kubectl create -f cert-manager.yaml

helm安装Flink Kubernetes Operator

  1. helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-0.1.0/
  3. helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator

验证

验证pod进程

  1. #alias kubectl="minikube kubectl --"
  2. kubectl get pod -A

image.png

验证seatunnel应用

构造虚拟数据流

  1. vi flink.streaming.conf
  2. #新增如下内容
  3. env {
  4.  execution.parallelism = 1
  5. }
  7. source {
  8.    FakeSourceStream {
  9.      result_table_name = "fake"
  10.      field_name = "name,age"
  11.    }
  12. }
  14. transform {
  15.    sql {
  16.      sql = "select name,age from fake"
  17.    }
  18. }
  20. sink {
  21.  ConsoleSink {}
  22. }

拷贝配置文件到nfs目录下

image.png

编写seatunnel的yaml部署文件

  1. vi seatunnel-flink.yaml

新增如下内容

  1. apiVersion: flink.apache.org/v1alpha1
  2. kind: FlinkDeployment
  3. metadata:
  4.   namespace: default
  5.   name: seatunnel-flink-streaming-example
  6. spec:
  7.   #image: seatunnel:2.1.0-flink-1.13
  8.   image: ren:8088/keyboardone/seatunnel:2.1.0-flink-1.13
  9.   flinkVersion: v1_14
  10.   flinkConfiguration:
  11.     taskmanager.numberOfTaskSlots: "2"
  12.   serviceAccount: flink
  13.   jobManager:
  14.     replicas: 1
  15.     resource:
  16.       memory: "2048m"
  17.       cpu: 1
  18.   taskManager:
  19.     resource:
  20.       memory: "2048m"
  21.       cpu: 2
  22.   podTemplate:  
  23.     spec:
  24.       containers:
  25.         - name: flink-main-container
  26.           volumeMounts:
  27.             - mountPath: /data
  28.               name: config-volume
  29.       volumes:
  30.         - name: config-volume
  31.           hostPath:
  32.             path: "/nfs/data"
  33.             type: Directory
  35.   job:
  36.     jarURI: local:///opt/flink/usrlib/seatunnel-core-flink.jar
  37.     entryClass: org.apache.seatunnel.SeatunnelFlink
  38.     args: ["--config", "/data/flink.streaming.conf"]
  39.     parallelism: 2
  40.     upgradeMode: stateless

执行部署

  1. kubectl apply -f seatunnel-flink.yaml

验证

验证服务

  1. kubectl get svc

image.png

查看日志

  1. kubectl logs -f deploy/seatunnel-flink-streaming-example

暴露端口

  1. kubectl port-forward svc/seatunnel-flink-streaming-example-rest 8081

Now the Flink Dashboard is accessible at localhost:8081.

打印日志

  1. kubectl logs \
  2. -l 'app in (seatunnel-flink-streaming-example), component in (taskmanager)' \
  3. --tail=-1 \
  4. -f

image.png

删除部署

  1. kubectl delete -f seatunnel-flink.yaml