1. LTTng
1.1 安装LTTng
LTTng: (Linux Trace Toolkit Next Generation),它是用于跟踪 Linux 内核、应用程序以及库的系统软件包。
Session守护进程和用户空间跟踪库使用Unix域套接字进行通信 ,
Consumer守护进程总是在创建事件规则后由会话守护进程生成 。
yum install libuuid-devel popt-devel libxml2-devel \
numactl-devel userspace-rcu-devel glib2-devel -y
ubuntu安装lttng-ust、lttng-tools、babeltrace可以尝试apt-get;
centos8的yum或者不可用,或者没有,已尝试过。
1.1.1 lttng-ust
用于检测和跟踪用户应用程序的库和Java/Python包。
# yum packages (NG)
yum install lttng-ust-devel
yum install libbabeltrace-devel
yum安装成功,但是
在/usr/include/lttng/ust-config.h中定义了 LTTNG_UST_HAVE_SDT_INTEGRATION,
期望是undef的,所以centos8 yum install后,在msquic cmake时仍然未启用lttng,
直接yum安装不符合msquic要求,接下来从源码中安装,ubuntu等系统未尝试。
# build from source (2.11.5~OK)
mkdir software &&
cd software &&
wget http://lttng.org/files/lttng-ust/lttng-ust-2.11.5.tar.bz2 &&
tar -xf lttng-ust-2.11.5.tar.bz2 &&
cd lttng-ust-2.11.5 &&
./configure --prefix=/usr &&
make -j4 &&
sudo make install &&
sudo ldconfig
1.1.2 lttng-tools
用于控制跟踪的库和命令行接口。
# build from source (2.11.5~OK)
mkdir software &&
cd software &&
wget http://lttng.org/files/lttng-tools/lttng-tools-2.11.5.tar.bz2 &&
tar -xf lttng-tools-2.11.5.tar.bz2 &&
cd lttng-tools-2.11.5 &&
./configure --prefix=/usr &&
make -j4 &&
sudo make install &&
sudo ldconfig
1.1.3 babeltrace
Babeltrace是一个命令行工具,它可以转换跟踪格式; 它支持lttng产生的格式,CTF,以及一个基本的文本输出,可以被grep。 babeltrace命令是babeltrace项目的一部分。
mkdir software &&
cd software &&
wget https://www.efficios.com/files/babeltrace/babeltrace2-2.0.3.tar.bz2 &&
tar -xf babeltrace2-2.0.3.tar.bz2 &&
cd babeltrace2-2.0.3 &&
./configure --prefix=/usr --disable-debug-info &&
make -j4 &&
sudo make install &&
sudo ldconfig
1.2 抓取日志
# 开启lttng会话守护进程
lttng-sessiond --daemonize
# 展示所有session
lttng list
# 展示所有追踪列表
lttng list --userspace
# 展示指定会话列表
lttng list --userspace my-user-space-session
结果如下:
UST events:
-------------
PID: 222054 - Name: ./quicsample
CLOG_WORKER_C:AllocFailure (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
CLOG_WORKER_C:WorkerStop (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
......
# 创建自己的会话 (自定义my-user-space-session)
lttng create my-user-space-session
# 创建一个事件
lttng enable-event --userspace --session=my-user-space-session CLOG_STREAM_RECV_C:*
# 开启所有事件
lttng enable-event --userspace --loglevel=TRACE_DEBUG --session=my-user-space-session --all
# 暂停所有事件
lttng disable-event --userspace --session=my-user-space-session --all-events
# 暂停一个事件
lttng disable-event --userspace --session=my-user-space-session CLOG_STREAM_RECV_C:*
# 开启追踪
lttng start
# 关闭追踪
lttng stop
# 删除会话(不会删除数据)
lttng destroy
# 设置当前lttng跟踪会话
lttng set-session my-user-space-session
# 查看当前会话
lttng status
# 存储路径在 $LTTNG_HOME/lttng-traces/name-date-time
# name是会话名字,LTTNG_HOME是环境变量,如果没设置,默认到$HOME
1.3 分析日志
# 无选项(打印到屏幕,可以重定向 > msquic.log)
babeltrace2 ~/lttng-traces/my-user-space-session*
# 增加grep
babeltrace2 /tmp/my-kernel-trace | grep _switch
# 增加wc
babeltrace2 /tmp/my-kernel-trace | grep _open | wc --lines
2. msquic开启日志
2.1 编译
cmake升级到3.6.0及以上版本(centos8是3.18.2符合要求)
cmake -version查看
根目录的cmakelist开启QUIC_ENABLE_LOGGING
增加选项 -DQUIC_ENABLE_LOGGING=ON
即:
mkdir build &&
cd build &&
cmake -G 'Unix Makefiles' -DCMAKE_INSTALL_PREFIX=/usr \
-DQUIC_TLS_SECRETS_SUPPORT=ON -DCMAKE_BUILD_TYPE=Debug \
-DQUIC_ENABLE_LOGGING=ON ..
make -j4
cmake执行后,确认是否开启日志
- 失败例子
- 成功例子
include
<a name="lzzfc"></a>
## 2.2 抓取/分析日志
msquic与vpp建立连接
./quicsample -client -target:172.16.1.1 -unsecure -user:bob
开启守护进程
lttng-sessiond —daemonize
创建会话 (自定义msquic-session)
lttng create msquic-session
开启所有日志
lttng enable-event —userspace —loglevel=TRACE_DEBUG —session=msquic-session —all
开启追踪
lttng start
发送数据(打流等)
ping 172.16.2.100 -i 2
关闭追踪
lttng stop
无选项
babeltrace2 ~/lttng-traces/msquic-session* > ~/lttng-traces/msquic.log
[msquic_trace_func.txt](https://www.yuque.com/attachments/yuque/0/2022/txt/12487488/1650876711775-fb5bf4b2-8037-401c-90cd-a7bf969ea087.txt?_lake_card=%7B%22src%22%3A%22https%3A%2F%2Fwww.yuque.com%2Fattachments%2Fyuque%2F0%2F2022%2Ftxt%2F12487488%2F1650876711775-fb5bf4b2-8037-401c-90cd-a7bf969ea087.txt%22%2C%22name%22%3A%22msquic_trace_func.txt%22%2C%22size%22%3A77919%2C%22type%22%3A%22text%2Fplain%22%2C%22ext%22%3A%22txt%22%2C%22source%22%3A%22%22%2C%22status%22%3A%22done%22%2C%22mode%22%3A%22title%22%2C%22download%22%3Atrue%2C%22taskId%22%3A%22u42cf8f2d-be04-4ef8-a2c5-d3f31bda784%22%2C%22taskType%22%3A%22upload%22%2C%22id%22%3A%22u5992050b%22%2C%22card%22%3A%22file%22%7D)<br />[start_lttng.sh](https://www.yuque.com/attachments/yuque/0/2022/sh/12487488/1654680921686-dfa0ef5e-1816-42a6-b071-0bf7e909df00.sh?_lake_card=%7B%22src%22%3A%22https%3A%2F%2Fwww.yuque.com%2Fattachments%2Fyuque%2F0%2F2022%2Fsh%2F12487488%2F1654680921686-dfa0ef5e-1816-42a6-b071-0bf7e909df00.sh%22%2C%22name%22%3A%22start_lttng.sh%22%2C%22size%22%3A196%2C%22type%22%3A%22text%2Fx-sh%22%2C%22ext%22%3A%22sh%22%2C%22source%22%3A%22%22%2C%22status%22%3A%22done%22%2C%22mode%22%3A%22title%22%2C%22download%22%3Atrue%2C%22taskId%22%3A%22uc1eeb70c-3c97-432a-abf3-b6e684a3a69%22%2C%22taskType%22%3A%22upload%22%2C%22__spacing%22%3A%22both%22%2C%22id%22%3A%22u334104be%22%2C%22margin%22%3A%7B%22top%22%3Atrue%2C%22bottom%22%3Atrue%7D%2C%22card%22%3A%22file%22%7D)
<a name="INIee"></a>
## 2.3 查询
<a name="bJ7t7"></a>
### 2.3.1 lttng list
lttng list
<a name="Qc6VN"></a>
### 
<a name="jgufo"></a>
### <br />2.3.2 lttng list --userspace
lttng list —userspace

<a name="zistT"></a>
### 2.3.3 lttng list --userspace msquic-session
lttng list —userspace msquic-session

<a name="fimeu"></a>
### 2.3.4 确认需要追踪的日志
<br />以stream_recv.c中的一个事件为例,想要打印Receive的这个日志,那么需要找到stream_recv.c.clog.h.lttng.h文件,确认其在lttng中的名字(即使用lttng list --userspace展示的结果)。
lttng list --userspace中展示的结果如下:<br /><br />那么执行如下命令后即可添加Receive日志,再抓取、解析后可获得
lttng enable-event —userspace CLOG_STREAM_RECV_C:Receive

<a name="ibFy6"></a>
# 3. ETW
<a name="8f65ddd8"></a>
## 3.1 抓取日志
1. 管理员运行cmd,开启日志;
netsh.exe trace start overwrite=yes report=dis correlation=dis traceFile=quic.etl provider={ff15e657-4f26-570e-88ab-0796b258d11c} level=0x5 keywords=0xffffffff
netsh.exe trace start overwrite=yes report=dis correlation=dis traceFile=quic.etl provider={ff15e657-4f26-570e-88ab-0796b258d11c} level=0x5 keywords=0x00000040
<br />
2. 开启日志后,运行quicsample程序;
2. 日志抓取完成后,关闭抓取;
netsh.exe trace stop
<br />产生quic.etl文件在quicsample.exe路径,即C:\Program Files (x86)\eversec\quic\quic-client路径,quic.etl是解析前的日志文件,相当于加密的文件。
<a name="497f0bfa"></a>
## 3.2 解析日志
使用msquic项目src/manifest/MsQuicEtw.man作为解析参考,可拷贝到exe路径。
wevtutil.exe um “C:\Program Files (x86)\eversec\quic\quic-client\MsQuicEtw.man” wevtutil.exe im “C:\Program Files (x86)\eversec\quic\quic-client\MsQuicEtw.man” /rf:”C:\Program Files (x86)\eversec\quic\quic-client\msquic.dll” /mf:”C:\Program Files (x86)\eversec\quic\quic-client\msquic.dll”
netsh.exe trace convert quic.etl overwrite=yes
<br />产生quic.txt文件在quicsample.exe路径,quic.txt文件就是解析后的日志文件。<br />
<a name="Vo2S8"></a>
# 4. msquic实例分析
```c
# 收发包
ConnPacketRecv|ConnPacketSent
# 操作情况
ConnExecOper|ConnExecApiOper|ConnExecTimerOper
# windows收发包
FLUSH_RECV|FLUSH_SEND|Sending batch
4.1 收包
- 大小 :::info CLOG_DATAPATH_EPOLL_C: DatapathRecv: { cpu_id = 0 }, { arg2 = 0x24D2EE0, arg3 = 1412, arg4 = 1412, arg5_len = 28。。。。。。
// arg2 = arg2 = SocketContext->Binding
// arg3 = arg3 = (uint32_t)RecvPacket->BufferLength
// arg4 = arg4 = (uint32_t)RecvPacket->BufferLength
// arg5 = arg5 = CASTED_CLOG_BYTEARRAY(sizeof(LocalAddr), LocalAddr)
// arg6 = arg6 = CASTED_CLOG_BYTEARRAY(sizeof(RemoteAddr), RemoteAddr)
:::
socket收包信息
- 数量 :::info CLOG_CONNECTION_C: QueueDatagrams: { cpu_id = 0 }, { arg1 = 0x7F98380010C0, arg3 = 3 }
// arg1 = arg1 = Connection
// arg3 = arg3 = DatagramChainLength
:::
记录recv的个数,与上面日志一起
- 大小 :::info CLOG_CONNECTION_C: ConnPacketRecv: { cpu_id = 2 }, { arg2 = 0x7EFFD4001DE0, arg3 = 12, arg4 = 5, arg5 = 47 }
// arg2 = arg2 = Connection
// arg3 = arg3 = Packet->PacketNumber
// arg4 = arg4 = Packet->IsShortHeader ? QUIC_TRACE_PACKET_ONE_RTT : (Packet->LH->Type + 1)
// arg5 = arg5 = Packet->HeaderLength + Packet->PayloadLength
:::
每个conn收到的包
4.2 发包
- 大小 :::info CLOG_PACKET_BUILDER_C: ConnPacketSent: { cpu_id = 3 }, { arg2 = 0x7FFB30001130, arg3 = 3, arg4 = 5, arg5 = 1220 }
// arg2 = arg2 = Connection
// arg3 = arg3 = Builder->Metadata->PacketNumber
// arg4 = arg4 = QuicPacketTraceType(Builder->Metadata)
// arg5 = arg5 = Builder->Metadata->PacketLength
:::
每个conn发送的包
- 数量 :::info CLOG_PACKET_BUILDER_C: PacketBuilderSendBatch: { cpu_id = 3 }, { arg1 = 0x7F98380010C0, arg3 = 1 }
// arg1 = arg1 = Builder->Connection
// arg3 = arg3 = (uint16_t)Builder->TotalCountDatagrams
:::
记录socket发送个数
4.3 操作
- 操作 :::info CLOG_OPERATION_H:ConnExecOper: { cpu_id = 2 }, { arg2 = 0x7F7A5C001DE0, arg3 = 1 }
// arg2 = arg2 = Connection
// arg3 = arg3 = Oper->Type
:::
- api操作 :::info CLOG_OPERATION_H:ConnExecApiOper: { cpu_id = 2 }, { arg2 = 0x7F7A5C001DE0, arg3 = 3 }
// arg2 = arg2 = Connection
// arg3 = arg3 = Oper->API_CALL.Context->Type
:::
- timer操作 :::info CLOG_OPERATION_H:ConnExecTimerOper: { cpu_id = 2 }, { arg2 = 0x7F7A5C001DE0, arg3 = 1 }
// arg2 = arg2 = Connection
// arg3 = arg3 = Oper->TIMER_EXPIRED.Type
:::