k3sup是一个支持在PC、虚拟机、ARM设备上安装k3s的工具，官方网站：https://k3sup.dev/
img

配置SSH

k3sup是基于ssh，所以必须先生成ssh私钥并且配置无密码登陆。
修改sshd配置：

  1. sed -i '/PasswordAuthentication/s/^/#/'  /etc/ssh/sshd_config
  2. sed -i 's/^[ ]*StrictHostKeyChecking.*/StrictHostKeyChecking no/g' /etc/ssh/ssh_config
  3. #禁用sshd服务的UseDNS、GSSAPIAuthentication两项特性
  4. sed -i -e 's/^#UseDNS.*$/UseDNS no/' /etc/ssh/sshd_config
  5. sed -i -e 's/^GSSAPIAuthentication.*$/GSSAPIAuthentication no/' /etc/ssh/sshd_config
  6. systemctl restart sshd

生成ssh私钥：

  1. [ ! -d ~/.ssh ] && ( mkdir ~/.ssh )
  2. [ ! -f ~/.ssh/id_rsa.pub ] && (yes|ssh-keygen -f ~/.ssh/id_rsa -t rsa -N "")
  3. ( chmod 600 ~/.ssh/id_rsa.pub ) && cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

配置无密码登陆：

  1. ssh-copy-id 192.168.56.141 #192.168.56.141是我虚拟机的ip

如果ssh-agent没启动，则启动：

  1. eval `ssh-agent`
  2. ssh-add ~/.ssh/id_rsa

安装k3sup

在线安装：

  1. curl -sLS https://get.k3sup.dev | sh

离线安装，下载地址：https://github.com/alexellis/k3sup/releases

  1. wget https://github.com/alexellis/k3sup/releases/download/0.9.2/k3sup
  2. mv k3sup /usr/local/bin/
  3. chmod +x /usr/local/bin/k3sup

创建k3s集群

启动一个k3s：

  1. export SERVER_IP=144.34.194.100
  2. export USER=chenzj
  3. k3sup install --ip $SERVER_IP --user $USER --ssh-port 29219 --k3s-version v1.17.3+k3s1 --k3s-extra-args '--docker --no-deploy=traefik'

安装日志：

  1. Running: k3sup install
  2. Public IP: 144.34.194.100
  3. ssh -i /home/chenzj/.ssh/id_rsa -p 29219 chenzj@144.34.194.100
  4. ssh: curl -sLS https://get.k3s.io | INSTALL_K3S_EXEC='server  --tls-san 144.34.194.100 --docker' INSTALL_K3S_VERSION='v1.17.3+k3s1' sh -
  6. [INFO]  Using v1.17.3+k3s1 as release
  7. [INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.17.3+k3s1/sha256sum-amd64.txt
  8. [INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.17.3+k3s1/k3s
  9. [INFO]  Verifying binary download
  10. [INFO]  Installing k3s to /usr/local/bin/k3s
  11. chcon: can't apply partial context to unlabeled file ‘/usr/local/bin/k3s’
  12. [INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
  13. [INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
  14. [INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
  15. [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
  16. [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
  17. [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
  18. [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
  19. [INFO]  systemd: Enabling k3s unit
  20. [INFO]  systemd: Starting k3s

运行成功之后，会将kubeconfig保存到 ~/kubeconfig，可以将该文件拷贝到kubectl的配置文件目录下：

  1. mkdir ~/.kube
  2. cp kubeconfig ~/.kube/config

或者是设置KUBECONFIG变量：

  1. export KUBECONFIG=/home/chenzj/kubeconfig

参考更多参数说明：

  1. $ k3sup install --help
  2. Install k3s on a server via SSH.
  4. Usage:
  5.   k3sup install [flags]
  7. Examples:
  8.   k3sup install --ip 192.168.0.100 --user root
  10. Flags:
  11.       --cluster                 Form a dqlite cluster
  12.       --context string          Set the name of the kubeconfig context. (default "default")
  13.   -h, --help                    help for install
  14.       --ip ip                   Public IP of node (default 127.0.0.1)
  15.       --ipsec                   Enforces and/or activates optional extra argument for k3s: flannel-backend option: ipsec
  16.       --k3s-extra-args string   Optional extra arguments to pass to k3s installer, wrapped in quotes (e.g. --k3s-extra-args '--no-deploy servicelb')
  17.       --k3s-version string      Optional version to install, pinned at a default (default "v1.17.2+k3s1")
  18.       --local                   Perform a local install without using ssh
  19.       --local-path string       Local path to save the kubeconfig file (default "kubeconfig")
  20.       --merge                   Merge the config with existing kubeconfig if it already exists.
  21.                                 Provide the --local-path flag with --merge if a kubeconfig already exists in some other directory
  22.       --no-extras               Disable "servicelb" and "traefik"
  23.       --skip-install            Skip the k3s installer
  24.       --ssh-key string          The ssh key to use for remote login (default "~/.ssh/id_rsa")
  25.       --ssh-port int            The port on which to connect for ssh (default 22)
  26.       --sudo                    Use sudo for installation. e.g. set to false when using the root user and no sudo is available. (default true)
  27.       --user string             Username for SSH login (default "root")

安装日志：

  1. Running: k3sup install
  2. Public IP: 192.168.56.141
  3. ssh -i /root/.ssh/id_rsa -p 22 root@192.168.56.141
  4. ssh: curl -sLS https://get.k3s.io | INSTALL_K3S_EXEC='server --tls-san 192.168.56.141 ' INSTALL_K3S_VERSION='v1.17.2+k3s1' sh -
  6. [INFO]  Using v1.17.2+k3s1 as release
  7. [INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.17.2+k3s1/sha256sum-amd64.txt
  8. [INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.17.2+k3s1/k3s

从安装日志，可以看出来k3sup其实就是对k3s进行了一层封装。启动K3s的命令如下：

  1. curl -sLS https://get.k3s.io | INSTALL_K3S_EXEC='server --tls-san 192.168.56.141 ' sh -

可以看到:

  • 使用的是container，没有使用docker
  • 这里指定了—tls-san参数，给TLS SAN添加了一个IP。查看节点信息进行验证：
  1. NAME          STATUS   ROLES    AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIMEk3s-node001   Ready    master   2m38s   v1.17.3+k3s1   10.0.2.15     <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.3.el7.x86_64   containerd://1.3.3-k3s1

查看集群状态：

  1. $ kubectl get svc -n kube-systemNAME                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGEkube-dns             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       14hmetrics-server       ClusterIP      10.43.194.172   <none>           443/TCP                      14htraefik-prometheus   ClusterIP      10.43.31.237    <none>           9100/TCP                     14htraefik              LoadBalancer   10.43.53.194    144.34.194.100   80:32572/TCP,443:31258/TCP   14h
  • 可以看到traefik的EXTERNAL-IP是外网IP地址。

添加一个work节点到集群：

  1. export SERVER_IP=192.168.56.141export USER=rootexport AGENT_IP=192.168.56.142k3sup join --ip $AGENT_IP --server-ip $SERVER_IP --user $USER

创建一个HA集群

创建一个集群：

  1. export SERVER_IP=192.168.56.141export USER=rootk3sup install \  --ip $SERVER_IP \  --user $USER \  --cluster

再创建一个master节点：

  1. export USER=rootexport SERVER_IP=192.168.56.141export NEXT_SERVER_IP=192.168.56.143k3sup join \  --ip $NEXT_SERVER_IP \  --user $USER \  --server-user $USER \  --server-ip $SERVER_IP \  --server

查看集群节点：

  1. kubectl get node

arkade安装应用

下载

  1. k3sup app install nginx-ingress

提示下载arkade：

  1. curl -sSL https://dl.get-arkade.dev/ | sudo sh
  3. #离线安装
  4. wget https://github.com/alexellis/arkade/releases/download/0.2.0/arkade
  5. cp arkade /usr/local/bin/
  6. chmod +x /usr/local/bin/arkade

使用说明

查看帮助：

  1. arkade install --help

安装应用：

  1. arkade install cert-manager
  3. # 制定命名空间
  4. arkade install postgresql --helm3 -n postgresql
  6. arkade install nginx-ingress --help

更新应用：

  1. arkade update

查看应用：

  1. arkade info postgresql

查看能够安装的应用有哪些：

  1. $ arkade install
  2. You can install:
  3.  - openfaas
  4.  - nginx-ingress
  5.  - cert-manager
  6.  - openfaas-ingress
  7.  - inlets-operator
  8.  - metrics-server
  9.  - chart
  10.  - linkerd
  11.  - cron-connector
  12.  - kafka-connector
  13.  - minio
  14.  - postgresql
  15.  - kubernetes-dashboard
  16.  - istio
  17.  - crossplane
  18.  - mongodb
  19.  - docker-registry
  20.  - docker-registry-ingress
  21.  - traefik2
  22.  - grafana
  23. Run arkade install NAME --help to see configuration options.

安装应用

安装nginx-ingress

  1. arkade install nginx-ingress

查看安装日志：

  1. Using kubeconfig: /home/chenzj/kubeconfig
  2. Using helm3
  3. Client: x86_64, Linux
  4. 2020/03/20 09:20:41 User dir established as: /home/chenzj/.arkade/
  5. https://get.helm.sh/helm-v3.1.1-linux-amd64.tar.gz
  6. /home/chenzj/.arkade/bin/helm3/linux-amd64 linux-amd64/
  7. /home/chenzj/.arkade/bin/helm3/README.md linux-amd64/README.md
  8. /home/chenzj/.arkade/bin/helm3/LICENSE linux-amd64/LICENSE
  9. /home/chenzj/.arkade/bin/helm3/helm linux-amd64/helm
  10. 2020/03/20 09:20:43 extracted tarball into /home/chenzj/.arkade/bin/helm3: 3 files, 0 dirs (1.059805161s)
  11. "stable" has been added to your repositories
  12. Hang tight while we grab the latest from your chart repositories...
  13. ...Successfully got an update from the "stable" chart repository
  14. Update Complete. ⎈ Happy Helming!⎈
  15. Node architecture: "amd64"
  16. Chart path:  /tmp/charts
  17. VALUES values.yaml
  18. Command: /root/.arkade/bin/helm3/helm [upgrade --install nginx-ingress stable/nginx-ingress --namespace default --values /tmp/charts/nginx-ingress/values.yaml --set defaultBackend.enabled=false]
  19. Release "nginx-ingress" has been upgraded. Happy Helming!

可以看到arkade自动做了一下几件事：

  • 下载helm3
  • 查找合适的chart仓库，并添加
  • 使用helm3安装应用，安装脚本：
  1. Command: /root/.arkade/bin/helm3/helm [upgrade --install nginx-ingress stable/nginx-ingress --namespace default --values /tmp/charts/nginx-ingress/values.yaml --set defaultBackend.enabled=false]

稍等几分钟，查看服务：

  1. $ kubectl get svc nginx-ingress-controllerNAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGEnginx-ingress-controller   LoadBalancer   10.43.155.95   <pending>     80:31992/TCP,443:31153/TCP   48m

如果使用的是共有云，则EXTERNAL-IP会是一个公网IP，如果一直显示pending状态，可以pod信息：

  1. $ kubectl describe pod/svclb-nginx-ingress-controller-h96vg  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

提示没有找到可用的端口，这是因为安装k3s时候，默认安装了traefix，而traefik已经占用了80和443端口：

  1. get svc -n kube-system
  2. NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
  3. kube-dns             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       15h
  4. metrics-server       ClusterIP      10.43.194.172   <none>           443/TCP                      15h
  5. traefik-prometheus   ClusterIP      10.43.31.237    <none>           9100/TCP                     14h
  6. traefik              LoadBalancer   10.43.53.194    144.34.194.100   80:32572/TCP,443:31258/TCP   14h

所以，需要禁用traefik，修改k3s启动脚本 /etc/systemd/system/k3s.service：添加 —no-deploy=traefik

  1. ExecStart=/usr/local/bin/k3s \
  2.     server \
  3.         '--tls-san' \
  4.         '144.34.194.100' \
  5.         '--docker' \
  6.         '--no-deploy=traefik' \

然后，重启k3s：

  1. systemctl daemon-reload
  2. systemctl restart k3s

如果上面没有生效，则卸载treafik，或者如果可能的话，重装k3s。
最好，再次查看svc：

  1. $ kubectl get svc nginx-ingress-controller
  2. NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
  3. nginx-ingress-controller   LoadBalancer   10.43.188.86   144.34.194.100   80:30079/TCP,443:30806/TCP   7s

安装TLS的docker-registry

详细步骤可以参考：https://blog.alexellis.io/get-a-tls-enabled-docker-registry-in-5-minutes/
registry
先设置kubeconfig：

  1. cat /etc/rancher/k3s/k3s.yaml > ~/.kube/config

安装：

  1. arkade install nginx-ingress
  2. arkade install cert-manager
  3. arkade install docker-registry

查看cert-manager是否安装成功：

  1. kubectl get all -n cert-manager

下面是将svc/docker-registry端口转发进行访问：

  1. kubectl port-forward svc/docker-registry --address 0.0.0.0 5000 & 
  3. export PASSWORD=h1p580SuX14352N9ZLje
  4. export IP="144.34.194.100"
  6. docker login $IP:5000 --username admin --password $PASSWORD
  7. docker tag alpine:3.11 $IP:5000/alpine:3.11
  8. docker push $IP:5000/alpine:3.11

安装docker-registry-ingress配置TLS：

  1. arkade install docker-registry-ingress \
  2.   --email junecloud@163.com \
  3.   --domain vps.javachen.space
  • 注意：vps.javachen.space是做了DNS解析到了144.34.194.100

查看证书是否生成：

  1. $ kubectl get cert
  2. NAME              READY   SECRET            AGE
  3. docker-registry   True    docker-registry   12m

这时候可以通过域名访问：

  1. docker login vps.javachen.space
  2. docker pull alpine:3.11
  3. docker tag alpine:3.11 vps.javachen.space/alpine:3.11
  4. docker push vps.javachen.space/alpine:3.11

浏览器访问：https://vps.javachen.space/v2/，输入用户名和秘密，可以查看生成的证书
image-20200320105740833