前提:有可用的kubernetes集群和skywalking监控。

软件版本信息:

软件 版本
kubernetes 1.17.2
nginx-ingress-controller 0.34.1
skywalking 8.1.0
skywalking-nginx-lua 0.2.0

下面直接进入正题……

(1)下载skywalking-nginx-lua

  1. git clone https://github.com/apache/skywalking-nginx-lua.git

(2)修改util.lua文件名,因为其和nginx-ingress默认的一个lua脚本名字冲突,这里改一下。

  1. $ skywalking-nginx-lua/lib/skywalking
  2. # 修改文件名
  3. $ mv util.lua swutil.lua
  4. # 修改文件调用
  5. ## 可以使用grep查看一下哪些文件有调用这个lua
  6. $ grep util `find ./ -type f`
  7. ./correlation_context.lua:local Util = require('util')
  8. ./segment_ref.lua:local Util = require('util')
  9. ./span.lua:local Util = require('util')
  10. ./tracing_context.lua:local Util = require('util')
  11. ./swutil_test.lua:local Util = require('util')
  12. # 将里面调用的util都改为swuitl

(3)修改Nginx-ingress的nginx.tmpl模板文件,增加Skywalking的配置。

  • 添加Skywalking Lua脚本扫描路径
  • 增加环境变量读取,如:SW_SERVICE_NAME、SW_SERVICE_INSTANCE_NAME、SW_BACKEND_SERVERS
  • 添加Tracing使用的缓存tracing_buffer
  • 设置Skywalking Lua Agent的初始化方法,并将相关配置从环境变量中提取。
  • 设置http节点的追踪配置。 ```shell

    Skywalking ENV

    env SW_SERVICE_NAME; env SW_SERVICE_INSTANCE_NAME; env SW_BACKEND_SERVERS;

events { multi_accept {{ if $cfg.EnableMultiAccept }}on{{ else }}off{{ end }}; worker_connections {{ $cfg.MaxWorkerConnections }}; use epoll; }

http {

  1. # 引入lua脚本
  2. lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/skywalking/?.lua;;";
  3. # 使用缓存
  4. lua_shared_dict tracing_buffer 100m;
  5. {{ buildLuaSharedDictionaries $cfg $servers }}
  6. init_by_lua_block {
  7. collectgarbage("collect")
  8. -- init modules
  9. local ok, res
  10. ok, res = pcall(require, "lua_ingress")
  11. if not ok then
  12. error("require failed: " .. tostring(res))
  13. else
  14. lua_ingress = res
  15. lua_ingress.set_config({{ configForLua $all }})
  16. end
  17. ok, res = pcall(require, "configuration")
  18. if not ok then
  19. error("require failed: " .. tostring(res))
  20. else
  21. configuration = res
  22. end
  23. ok, res = pcall(require, "balancer")
  24. if not ok then
  25. error("require failed: " .. tostring(res))
  26. else
  27. balancer = res
  28. end
  29. {{ if $all.EnableMetrics }}
  30. ok, res = pcall(require, "monitor")
  31. if not ok then
  32. error("require failed: " .. tostring(res))
  33. else
  34. monitor = res
  35. end
  36. {{ end }}
  37. ok, res = pcall(require, "certificate")
  38. if not ok then
  39. error("require failed: " .. tostring(res))
  40. else
  41. certificate = res
  42. certificate.is_ocsp_stapling_enabled = {{ $cfg.EnableOCSP }}
  43. end
  44. ok, res = pcall(require, "plugins")
  45. if not ok then
  46. error("require failed: " .. tostring(res))
  47. else
  48. plugins = res
  49. end
  50. -- load all plugins that'll be used here
  51. plugins.init({ {{ range $idx, $plugin := $cfg.Plugins }}{{ if $idx }},{{ end }}{{ $plugin | quote }}{{ end }} })
  52. }
  53. init_worker_by_lua_block {

……

  1. -- Skywalking
  2. local metadata_buffer = ngx.shared.tracing_buffer
  3. metadata_buffer:set('serviceName', os.getenv("SW_SERVICE_NAME"))
  4. metadata_buffer:set('serviceInstanceName', os.getenv("SW_SERVICE_INSTANCE_NAME"))
  5. -- set random seed
  6. require("swutil").set_randomseed()
  7. require("client"):startBackendTimer(os.getenv("SW_BACKEND_SERVERS"))
  8. }

……

  1. rewrite_by_lua_block {
  2. lua_ingress.rewrite({{ locationConfigForLua $location $all }})
  3. balancer.rewrite()
  4. plugins.run()
  5. -- Skywalking
  6. require("tracer"):start({{ buildUpstreamName $location | quote }})
  7. }

……

  1. body_filter_by_lua_block {
  2. -- Skywalking
  3. if ngx.arg[2] then
  4. require("tracer"):finish()
  5. end
  6. }

……

  1. log_by_lua_block {
  2. balancer.log()
  3. {{ if $all.EnableMetrics }}
  4. monitor.call()
  5. {{ end }}
  6. plugins.run()
  7. -- Skywalking
  8. require("tracer"):prepareForReport()
  9. }

…….

  1. 4)将这些脚本添加到nginx-ingress镜像中,Dockerfile如下:
  2. ```dockerfile
  3. FROM quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0
  4. ADD --chown=www-data nginx.tmpl /etc/nginx/template
  5. ADD --chown=www-data skywalking /etc/nginx/lua/skywalking

(5)构建镜像并上传仓库

  1. $ docker build -t registry.cn-hangzhou.aliyuncs.com/rookieops/ingress-nginx-controller-skywalking:0.34.1 .
  2. $ docker push registry.cn-hangzhou.aliyuncs.com/rookieops/ingress-nginx-controller-skywalking:0.34.1

(6)修改nginx-ingress的deployment文件,主要增加以下环境变量

  1. ......
  2. containers:
  3. - name: controller
  4. image: registry.cn-hangzhou.aliyuncs.com/rookieops/nginx-ingress-controller:0.32.0
  5. imagePullPolicy: IfNotPresent
  6. ......
  7. - name: SW_SERVICE_NAME
  8. value: Kubernetes Ingress
  9. - name: SW_BACKEND_SERVERS
  10. value: http://skywalking-oap.skywalking.svc.cluster.local:12800
  11. - name: SW_SERVICE_INSTANCE_NAME
  12. valueFrom:
  13. fieldRef:
  14. fieldPath: metadata.uid
  15. ......

(7)然后重新部署ingress-controller应用。

然后可以在skywalking的面板上看到了。
image.png

已将所需的代码都放在github了,仓库地址:https://github.com/sunsharing-note/skywalking-ingress.git

参考: