前提:有可用的kubernetes集群和skywalking监控。
软件版本信息:
软件 | 版本 |
---|---|
kubernetes | 1.17.2 |
nginx-ingress-controller | 0.34.1 |
skywalking | 8.1.0 |
skywalking-nginx-lua | 0.2.0 |
下面直接进入正题……
(1)下载skywalking-nginx-lua
git clone https://github.com/apache/skywalking-nginx-lua.git
(2)修改util.lua
文件名,因为其和nginx-ingress默认的一个lua脚本名字冲突,这里改一下。
$ skywalking-nginx-lua/lib/skywalking
# 修改文件名
$ mv util.lua swutil.lua
# 修改文件调用
## 可以使用grep查看一下哪些文件有调用这个lua
$ grep util `find ./ -type f`
./correlation_context.lua:local Util = require('util')
./segment_ref.lua:local Util = require('util')
./span.lua:local Util = require('util')
./tracing_context.lua:local Util = require('util')
./swutil_test.lua:local Util = require('util')
# 将里面调用的util都改为swuitl
(3)修改Nginx-ingress的nginx.tmpl模板文件,增加Skywalking的配置。
- 添加Skywalking Lua脚本扫描路径
- 增加环境变量读取,如:SW_SERVICE_NAME、SW_SERVICE_INSTANCE_NAME、SW_BACKEND_SERVERS
- 添加Tracing使用的缓存
tracing_buffer
- 设置Skywalking Lua Agent的初始化方法,并将相关配置从环境变量中提取。
- 设置http节点的追踪配置。
```shell
Skywalking ENV
env SW_SERVICE_NAME; env SW_SERVICE_INSTANCE_NAME; env SW_BACKEND_SERVERS;
events { multi_accept {{ if $cfg.EnableMultiAccept }}on{{ else }}off{{ end }}; worker_connections {{ $cfg.MaxWorkerConnections }}; use epoll; }
http {
# 引入lua脚本
lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/skywalking/?.lua;;";
# 使用缓存
lua_shared_dict tracing_buffer 100m;
{{ buildLuaSharedDictionaries $cfg $servers }}
init_by_lua_block {
collectgarbage("collect")
-- init modules
local ok, res
ok, res = pcall(require, "lua_ingress")
if not ok then
error("require failed: " .. tostring(res))
else
lua_ingress = res
lua_ingress.set_config({{ configForLua $all }})
end
ok, res = pcall(require, "configuration")
if not ok then
error("require failed: " .. tostring(res))
else
configuration = res
end
ok, res = pcall(require, "balancer")
if not ok then
error("require failed: " .. tostring(res))
else
balancer = res
end
{{ if $all.EnableMetrics }}
ok, res = pcall(require, "monitor")
if not ok then
error("require failed: " .. tostring(res))
else
monitor = res
end
{{ end }}
ok, res = pcall(require, "certificate")
if not ok then
error("require failed: " .. tostring(res))
else
certificate = res
certificate.is_ocsp_stapling_enabled = {{ $cfg.EnableOCSP }}
end
ok, res = pcall(require, "plugins")
if not ok then
error("require failed: " .. tostring(res))
else
plugins = res
end
-- load all plugins that'll be used here
plugins.init({ {{ range $idx, $plugin := $cfg.Plugins }}{{ if $idx }},{{ end }}{{ $plugin | quote }}{{ end }} })
}
init_worker_by_lua_block {
……
-- Skywalking
local metadata_buffer = ngx.shared.tracing_buffer
metadata_buffer:set('serviceName', os.getenv("SW_SERVICE_NAME"))
metadata_buffer:set('serviceInstanceName', os.getenv("SW_SERVICE_INSTANCE_NAME"))
-- set random seed
require("swutil").set_randomseed()
require("client"):startBackendTimer(os.getenv("SW_BACKEND_SERVERS"))
}
……
rewrite_by_lua_block {
lua_ingress.rewrite({{ locationConfigForLua $location $all }})
balancer.rewrite()
plugins.run()
-- Skywalking
require("tracer"):start({{ buildUpstreamName $location | quote }})
}
……
body_filter_by_lua_block {
-- Skywalking
if ngx.arg[2] then
require("tracer"):finish()
end
}
……
log_by_lua_block {
balancer.log()
{{ if $all.EnableMetrics }}
monitor.call()
{{ end }}
plugins.run()
-- Skywalking
require("tracer"):prepareForReport()
}
…….
(4)将这些脚本添加到nginx-ingress镜像中,Dockerfile如下:
```dockerfile
FROM quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0
ADD --chown=www-data nginx.tmpl /etc/nginx/template
ADD --chown=www-data skywalking /etc/nginx/lua/skywalking
(5)构建镜像并上传仓库
$ docker build -t registry.cn-hangzhou.aliyuncs.com/rookieops/ingress-nginx-controller-skywalking:0.34.1 .
$ docker push registry.cn-hangzhou.aliyuncs.com/rookieops/ingress-nginx-controller-skywalking:0.34.1
(6)修改nginx-ingress的deployment文件,主要增加以下环境变量
......
containers:
- name: controller
image: registry.cn-hangzhou.aliyuncs.com/rookieops/nginx-ingress-controller:0.32.0
imagePullPolicy: IfNotPresent
......
- name: SW_SERVICE_NAME
value: Kubernetes Ingress
- name: SW_BACKEND_SERVERS
value: http://skywalking-oap.skywalking.svc.cluster.local:12800
- name: SW_SERVICE_INSTANCE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.uid
......
(7)然后重新部署ingress-controller应用。
然后可以在skywalking的面板上看到了。
已将所需的代码都放在github了,仓库地址:https://github.com/sunsharing-note/skywalking-ingress.git
参考: