title: linux历史问题排查神器-sar命令 #标题tags: #标签
date: 2022-01-28
categories: linux大杂烩 # 分类

工作中总会碰到几次要分析机器历史问题的需求，之前没有系统化的整理过，也没有深入的去理解那些排查系统性能工具的指标输出，导致在遇上这种需求的时候，总是心虚，能搪塞过去就搪塞过去，不过随着工作年限的增加，遇到这种需求，再说不出个123来，就配不上工牌上那“高级运维工程师”几个字了。这篇文章，就来拆解下这个linux历史问题排查神器-sar命令 。

参考资料：

如果你的系统没有安装sysstat这个包，那么是没有sar命令的，也不会有可分析的文件，需要执行命令yum -y install sysstat去安装，安装完成后，就有sar命令了。同时也会自动生成相关目录及计划任务/etc/cron.d/sysstat。

sar常用命令

假设你要全面的分析系统指标，又没有足够的耐心将这篇文章看完，想着现学现卖，那么直接执行下面的命令，然后去看系统输出的各项指标吧，看不懂的，就来这个页面搜索下指标名称，就可以找到对应的解释，如果文章中没有解释，那就需要去文章开头附的参考资料中寻找答案了。

万能的sar命令如下：

  1. # 输出26日的20:00:01到23:00:01之间的所有报告
  2. $ sar -f sa26 -A -s 20:00:01 -e 23:00:01 | less
  4. # 每隔3s打印下当前系统的所有性能指标并输出到/tmp/sar_3.log文件中
  5. $ sar -A 3 >> /tmp/sar_3.log

各选项如下：

  • -f：指定要分析的文件sa文件，默认保留近三十天的，文件名格式：sa[日期]，如1号的就是sa1，15号的就是sa15，不过建议使用stat命令查一下文件最近改动时间，比如sa15，查出来的可能是上个月15号的日志，等到这个月16号，再看sa15这个文件，那么文件最近改动时间，就是当月的15号了，上月的已经被删除了。在分析之前，一定要先确认自己分析的文件没错哦，别分析了半天，看的不是你想看的那天的，那就尴尬了。
  • -A：输出全部的性能指标，包括磁盘、系统平均负载、内存、网卡流量等等，如果想单独输出某方面的指标，需要加其他选项，这个后面细说。
  • -s：指定要输出的报告起始时间，上面的命令是输出20:00:01之后的。
  • -e：指定要输出的报告截止时间，上面的命令是输出23:00:01之前的。
  • -u：输出CPU使用情况的统计信息
  • -v：输出inode、文件和其他内核表的统计信息
  • -d：输出每一个块设备的活动信息
  • -r：输出内存和交换空间的统计信息
  • -b：显示I/O和传送速率的统计信息
  • -a：文件读写情况
  • -c：输出进程统计信息，每秒创建的进程数
  • -R：输出内存页面的统计信息
  • -y：终端设备活动情况
  • -w：输出系统交换活动信息

CPU相关指标

  1. # 输出26号，20:00:00到22:00:00，CPU相关的指标(-P ALL：输出每个核心的详细指标信息)
  2. $ sar -f sa26 -u -P ALL -u ALL -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • CPU：表示当前是哪个CPU，all表示所有cpu的指标平均值。
  • %usr：CPU在用户模式下，执行进程的时间百分比 。
  • %nice：CPU处在带NICE值的用户模式下的时间百分比。
  • %system： CPU处在系统模式(内核态)下，执行进程的时间百分比。
  • %iowait：CPU用于等待I/O操作完成(等待输入输出完成),占用CPU总时间的百分比，如果此值过高，表示硬盘存在I/O瓶颈 。
  • %steal：管理程序为另一个虚拟进程提供服务而等待虚拟CPU的百分比。
  • %irq：CPU服务硬件中断（简称硬中断）所花费的时间百分比。
  • %soft：CPU服务软件中断（简称软中断）所花费的时间百分比。
  • %guest：CPU运行虚拟处理器所花费的时间百分比。
  • %gnice：Percentage of time spent by the CPU or CPUs to run a niced guest.
  • %idle：CPU空闲时间百分比，如果此值过高，表示CPU较空闲。如果此值高但系统响应慢时，有可能是 CPU 等待分配内存，此时应查看内存使用，必要时需要加大内存容量 ，如果此值持续低于10，则系统的 CPU 处理能力相对较低，表明系统中最需要解决的资源是 CPU。

任务创建和系统切换活动相关

  1. # 输出26号，20:00:00到22:00:00，任务创建和系统切换活动相关指标
  2. $ sar -f sa26 -w -s 20:00:00 -e 22:00:00

各项指标解释如下：

  • proc/s：每秒创建的总任务数
  • cswch/s：每秒上下文切换的总数。

系统负载相关

  1. # 输出26号，20:00:00到22:00:00，进程队列长度和平均负载状态的指标
  2. $ sar -f sa26 -q -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • runq-sz ：运行队列的长度，等待运行的进程数量
  • plist-sz：进程列表中进程和线程的数量
  • ldavg-1 ：最后1分钟的系统平均负载
  • ldavg-5 ：过去5分钟的系统平均负载
  • ldavg-15：过去15分钟的系统平均负载
  • blocked：当前阻塞的等待I/O完成的任务数。

内存统计信息相关

  1. # 输出26号，20:00:00到22:00:00，内存统计信息相关的指标
  2. $ sar -f sa26 -R -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • frmpg/s：系统每秒释放的内存页数。负数表示系统分配的页面数。注意，根据机器的体系结构，页面的大小为4 kB或8 kB。
  • bufpg/s：每秒系统用作缓冲区的额外内存页数。负值意味着系统使用更少的页作为缓冲区。
  • campg/s：系统每秒缓存的额外内存页数。负值意味着缓存中的页面更少。

内存和交换空间相关

  1. # 输出26号，20:00:00到22:00:00，内存和交换空间相关的指标
  2. $ sar -f sa26 -r -u ALL -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • kbmemfree：空闲的内存数量（单位为KB)，这个值和free命令中的free值基本一致,不包括buffer和cache的空间。
  • kbmemused：已使用的内存数量,不包含内核使用的内存(单位为KB)。这个值和free命令中的used值基本一致，所以它包括buffer和cache的空间。
  • %memused：已使用内存的百分数。物理内存使用率，这个值是kbmemused和内存总量(不包括swap)的一个百分比。
  • kbbuffers：内核缓冲区buffer,使用的内存数量(单位为KB)。这个值就是free命令中的buffer。
  • kbcached：内核高速缓存cache数据使用的内存数量(单位为KB)。这个值就是free命令中的cache。
  • kbcommit：保证当前系统所需要的内存,即为了确保不溢出而需要的内存(RAM+swap)。
  • %commit：这个值是kbcommit与内存总量(包括swap)的一个百分比。
  • kbactive：以KB为单位的活动内存数量(最近使用的内存，除非绝对必要，通常不会回收)。
  • kbinact：非活动内存的数量(以KB为单位)(最近使用较少的内存)。
  • kbdirty：等待“写回磁盘”的以KB为单位的内存量。

系统交换活动相关信息

  1. # 输出26号，20:00:00到22:00:00，系统交换活动相关的指标
  2. $ sar -f sa26 -W -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • pswpin/s：每秒系统换入的交换页面（swap page）数量
  • pswpout/s：每秒系统换出的交换页面（swap page）数量

内存分页相关

  1. # 输出26号，20:00:00到22:00:00，内存和交换空间相关的指标
  2. $ sar -f sa26 -B -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • pgpgin/s：每秒从磁盘空间或交换空间置换到内存的字节数(单位为KB)。
  • pgpgout/s：每秒从内存置换到磁盘空间或交换空间的字节数(单位为KB)。
  • fault/s：每秒钟系统产生的缺页数(主缺页加次缺页)。
  • majflt/s：每秒钟产生的主缺页数。
  • pgfree/s：每秒被放入空闲队列中的页个数。
  • pgscank/s：每秒被kswapd扫描的页个数。
  • pgscand/s：每秒直接被扫描的页个数。
  • pgsteal/s：每秒钟从cache中被清除来满足内存需要的页个数。
  • %vmeff：每秒清除的页占总扫描页的百分比。

网络设备统计相关

  1. # 输出26号，20:00:00到22:00:00，eth0这块网卡的网络设备统计相关的指标
  2. # 省略egrep的话，输出所有网卡，也可以egrep中填入多块网卡名称，| 分隔即可
  3. $ sar -f sa26 -n DEV -s 20:00:00 -e 22:00:00 | egrep 'eth0|IFACE'

输出的各项指标含义如下：

  • IFACE：网络设备名。
  • rxpck/s：每秒接收的包数量。
  • txpck/s：每秒传输的包数量。
  • rxbyt/s：每秒接收的字节数（单位为byte）。
  • txbyt/s：每秒传输的字节数（单位为byte）。
  • rxkB/s：每秒收的数据量(单位为kB)。
  • txkB/s：每秒发的数据量(单位为kB)。
  • rxcmp/s：每秒接收压缩包的数量。
  • txcmp/s：每秒传输压缩包的数量。
  • rxmcst/s：每秒接收的多播（multicast）包的总数排查网络设备故障。

网络设备故障相关

  1. # 输出26号，20:00:00到22:00:00，eth0这块网卡的故障相关的指标（省略egrep的话，输出所有网卡，也可以egrep中填入多块网卡名称，| 分隔即可）
  2. $ sar -f sa26 -n EDEV -s 20:00:00 -e 22:00:00 | egrep 'eth0|IFACE'

输出的各项指标含义如下：

  • IFACE：网络设备名。
  • rxerr/s：每秒接收的坏包数量。
  • txerr/s：传输包时每秒发生错误的数量。
  • coll/s：传输包时每秒发生冲突的数量。
  • rxdrop/s：接收包时，每秒丢弃的包的数量(缺乏缓存导致)。
  • txdrop/s：传输包时，每秒丢弃的包的数量(缺乏缓存导致)。
  • txcarr/s：传输包时，每秒发生的传输错误的数量。
  • rxfram/s：接收包时，每秒发生帧校验错误的数量。
  • rxfifo/s：接收包时，每秒钟缓冲区溢出错误的数量。
  • txfifo/s：传输包时，每秒钟缓冲区溢出错误的数量。

I/O和传送速率

  1. # 输出26号，20:00:00到22:00:00，机器上所有I/O和传送速率相关指标
  2. $ sar -f sa26 -b -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • tps：每每秒向磁盘设备请求数据的次数，包括读、写请求，其为rtps与wtps的和。每一次IO下发后会先将多个请求合并为一个I/O磁盘请求，这里tps指请求合并后的请求计数
  • rtps：每秒向磁盘设备的读请求次数
  • wtps：每秒向磁盘设备的写请求次数
  • bread/s：每秒钟从物理设备读入的数据量，单位为 块/s
  • bwrtn/s：每秒钟向物理设备写入的数据量，单位为 块/s

磁盘相关

  1. # 输出26号，20:00:00到22:00:00，机器上所有磁盘的使用相关指标
  2. $ sar -f sa26 -d -p -s 20:00:00 -e 22:00:00
  4. # -p：打印设备的名字。此选项与选项-d一起使用。默认不加-p的话情况下，名称打印为dev m-n，
  5. # 其中m和n为设备的主号码和副号码。使用-p选项将显示设备名，易读性更高。
  6. # 设备的主号码和副号码可以通过lsblk命令（输出的MAJ:MIN列，就是主号码和副号码）查看。

输出的各项指标含义如下：

  • DEV：磁盘设备，加上用参数-p可以打印出sda等磁盘设备名称；如不加参数-p,设备则显示为dev253-0等。
  • tps：每秒向磁盘设备请求数据的次数，包括读、写请求，其为rtps与wtps的和。每一次IO下发后会先将多个请求合并为一个I/O磁盘请求，这里tps指请求合并后的请求计数。
  • rd_sec/s：每秒读扇区的次数。
  • wr_sec/s：每秒写扇区的次数。
  • avgrq-sz：平均每次设备I/O操作的数据大小(扇区)。
  • avgqu-sz：磁盘请求队列的平均长度。
  • await：从请求磁盘到系统处理完,每次请求的平均消耗时间,包括请求队列等待时间（单位是毫秒）。
  • svctm：系统处理每次请求的平均时间,不包括在请求队列中消耗的时间。man手册中提示该字段将在将来的sysstat版本中删除。不要相信这个指标了。
  • %util：I/O请求占CPU的百分比。man手册解释：Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

交换空间swap利用率相关

  1. # 输出26号，20:00:00到22:00:00，机器上交换空间swap利用率相关指标
  2. $ sar -f sa26 -b -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • kbswpfree：可用的空闲交换空间大小。
  • kbswpused：已使用的交换空间大小。
  • %swpused：已使用交换空间的百分数。
  • kbswpcad：交换空间的高速缓存使用的内存大小。man手册解释：Amount of cached swap memory in kilobytes. This is memory that once was swapped out, is swapped back in but still also is in the swap area (if memory is needed it doesn’t need to be swapped out again because it is already in the swap area. This saves I/O)。
  • %swpcad：缓存的交换内存与已用交换空间的比例。

大量页面的利用率统计数据

  1. # 输出26号，20:00:00到22:00:00， 大量页面的利用率统计数据相关指标
  2. $ sar -f sa26 -H -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • kbhugfree：未分配的大页面内存数量(以KB为单位)。
  • kbhugused：已分配的大页面内存数量(以KB为单位)。
  • %hugused：已分配的大页内存占总内存的百分比。

inode、file和其他内核相关

  1. # 输出26号，20:00:00到22:00:00，inode、file和其他内核相关指标
  2. $ sar -f sa26 -v -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • dentunusd：目录缓存中未使用的缓存条目数。
  • file-nr：系统使用的块句柄数。
  • inode-nr：系统使用的inode句柄数
  • pty-nr：系统使用的伪终端数目，如果此时没人登录着系统，则此值应该为0。

TTY设备活动相关

  1. # 输出26号，20:00:00到22:00:00，TTY设备活动相关指标
  2. $ sar -f sa26 -y -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • TTY：当前终端
  • rcvin/s：当前串行线每秒接收中断数。序列号在TTY列中给出。
  • xmtin/s：当前串行线每秒传输中断数。
  • framerr/s：当前串行线每秒的帧错误数。
  • prtyerr/s：当前串行线每秒的奇偶校验错误数。
  • brk/s：当前串行线每秒的断行数。
  • ovrun/s：当前串行线每秒溢出错误数。

NFS客户端相关

  1. # 输出26号，20:00:00到22:00:00，关于NFS客户端活动的统计信息。
  2. $ sar -f sa26 -n NFS -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • call/s：每秒发出的RPC请求数。
  • retrans/s：每秒RPC请求的数量，需要重传的请求(例如由于服务器超时)。
  • read/s：每秒的“读”RPC调用数。
  • write/s：每秒的“写”RPC调用数。
  • access/s：每秒进行的“访问”RPC调用数
  • getatt/s：每秒RPC调用’ getattr’的次数。

NFSD 客户端相关

  1. # 输出26号，20:00:00到22:00:00，关于NFS服务器活动的统计信息
  2. $ sar -f sa26 -n NFSD -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • scall/s：每秒收到的RPC请求数。
  • badcall/s：每秒收到的不良RPC请求数，这些请求的处理产生了错误。
  • packet/s：每秒接收的网络数据包数。
  • udp/s：每秒接收到的UDP报文数。
  • tcp/s：每秒接收到的TCP报文数
  • hit/s：每秒命中应答缓存的次数
  • miss/s：每秒回复缓存未命中的次数。
  • sread/s：每秒接收到的“读”RPC调用数
  • swrite/s：每秒收到写RPC请求的次数。
  • saccess/s：每秒接收到的“访问”RPC调用数。
  • sgetatt/s：每秒接收到的’ getattr’ RPC调用数。

套接字的使用(IPv4)相关

  1. # 输出26号，20:00:00到22:00:00，关于套接字的使用(IPv4)相关的统计信息
  2. $ sar -f sa26 -n SOCK -s 20:00:00 -e 22:00:00

输出的各项指标含义如下：

  • totsck：系统使用的socket总数。
  • tcpsck：当前正在使用的TCP套接字数量。
  • udpsck：当前使用的UDP套接字的数量。
  • rawsck：当前正在使用的RAW套接字数量。
  • ip-frag：当前排队的IP分片数
  • tcp-tw：处于TIME_WAIT状态的TCP socket数量。

-n指定其他指标信息

在上面，我们使用-n查过nfs和套接字相关的信息，其实还有其他关键字可以指定。如DEV, EDEV, FC, ICMP, EICMP, ICMP6,EICMP6, IP, EIP, IP6, EIP6, NFS, NFSD, SOCK, SOCK6, SOFT,TCP, ETCP, UDP and UDP6.

这里只将官方提供的英文贴到这，有需要的话，自行翻译即可。

  1.  -n { keyword[,...] | ALL }
  2.               Report network statistics.
  4.               Possible keywords are DEV, EDEV, FC, ICMP, EICMP, ICMP6,
  5.               EICMP6, IP, EIP, IP6, EIP6, NFS, NFSD, SOCK, SOCK6, SOFT,
  6.               TCP, ETCP, UDP and UDP6.
  8.               With the DEV keyword, statistics from the network devices
  9.               are reported.  Statistics for all network interfaces are
  10.               displayed unless a restricted list is specified using
  11.               option --iface= (see corresponding option entry).  The
  12.               following values are displayed:
  14.               IFACE  Name of the network interface for which statistics
  15.                      are reported.
  17.               rxpck/s
  18.                      Total number of packets received per second.
  20.               txpck/s
  21.                      Total number of packets transmitted per second.
  23.               rxkB/s Total number of kilobytes received per second.
  25.               txkB/s Total number of kilobytes transmitted per second.
  27.               rxcmp/s
  28.                      Number of compressed packets received per second
  29.                      (for cslip etc.).
  31.               txcmp/s
  32.                      Number of compressed packets transmitted per
  33.                      second.
  35.               rxmcst/s
  36.                      Number of multicast packets received per second.
  38.               %ifutil
  39.                      Utilization percentage of the network interface.
  40.                      For half-duplex interfaces, utilization is
  41.                      calculated using the sum of rxkB/s and txkB/s as a
  42.                      percentage of the interface speed. For full-duplex,
  43.                      this is the greater of rxkB/S or txkB/s.
  45.               With the EDEV keyword, statistics on failures (errors)
  46.               from the network devices are reported.  Statistics for all
  47.               network interfaces are displayed unless a restricted list
  48.               is specified using option --iface= (see corresponding
  49.               option entry).  The following values are displayed:
  51.               IFACE  Name of the network interface for which statistics
  52.                      are reported.
  54.               rxerr/s
  55.                      Total number of bad packets received per second.
  57.               txerr/s
  58.                      Total number of errors that happened per second
  59.                      while transmitting packets.
  61.               coll/s Number of collisions that happened per second while
  62.                      transmitting packets.
  64.               rxdrop/s
  65.                      Number of received packets dropped per second
  66.                      because of a lack of space in linux buffers.
  68.               txdrop/s
  69.                      Number of transmitted packets dropped per second
  70.                      because of a lack of space in linux buffers.
  72.               txcarr/s
  73.                      Number of carrier-errors that happened per second
  74.                      while transmitting packets.
  76.               rxfram/s
  77.                      Number of frame alignment errors that happened per
  78.                      second on received packets.
  80.               rxfifo/s
  81.                      Number of FIFO overrun errors that happened per
  82.                      second on received packets.
  84.               txfifo/s
  85.                      Number of FIFO overrun errors that happened per
  86.                      second on transmitted packets.
  88.               With the FC keyword, statistics about fibre channel
  89.               traffic are reported.  Note that fibre channel statistics
  90.               depend on sadc's option -S DISK to be collected.  The
  91.               following values are displayed:
  93.               FCHOST Name of the fibre channel host bus adapter (HBA)
  94.                      interface for which statistics are reported.
  96.               fch_rxf/s
  97.                      The total number of frames received per second.
  99.               fch_txf/s
  100.                      The total number of frames transmitted per second.
  102.               fch_rxw/s
  103.                      The total number of transmission words received per
  104.                      second.
  106.               fch_txw/s
  107.                      The total number of transmission words transmitted
  108.                      per second.
  110.               With the ICMP keyword, statistics about ICMPv4 network
  111.               traffic are reported.  Note that ICMPv4 statistics depend
  112.               on sadc's option -S SNMP to be collected.  The following
  113.               values are displayed (formal SNMP names between square
  114.               brackets):
  116.               imsg/s The total number of ICMP messages which the entity
  117.                      received per second [icmpInMsgs].  Note that this
  118.                      counter includes all those counted by ierr/s.
  120.               omsg/s The total number of ICMP messages which this entity
  121.                      attempted to send per second [icmpOutMsgs].  Note
  122.                      that this counter includes all those counted by
  123.                      oerr/s.
  125.               iech/s The number of ICMP Echo (request) messages received
  126.                      per second [icmpInEchos].
  128.               iechr/s
  129.                      The number of ICMP Echo Reply messages received per
  130.                      second [icmpInEchoReps].
  132.               oech/s The number of ICMP Echo (request) messages sent per
  133.                      second [icmpOutEchos].
  135.               oechr/s
  136.                      The number of ICMP Echo Reply messages sent per
  137.                      second [icmpOutEchoReps].
  139.               itm/s  The number of ICMP Timestamp (request) messages
  140.                      received per second [icmpInTimestamps].
  142.               itmr/s The number of ICMP Timestamp Reply messages
  143.                      received per second [icmpInTimestampReps].
  145.               otm/s  The number of ICMP Timestamp (request) messages
  146.                      sent per second [icmpOutTimestamps].
  148.               otmr/s The number of ICMP Timestamp Reply messages sent
  149.                      per second [icmpOutTimestampReps].
  151.               iadrmk/s
  152.                      The number of ICMP Address Mask Request messages
  153.                      received per second [icmpInAddrMasks].
  155.               iadrmkr/s
  156.                      The number of ICMP Address Mask Reply messages
  157.                      received per second [icmpInAddrMaskReps].
  159.               oadrmk/s
  160.                      The number of ICMP Address Mask Request messages
  161.                      sent per second [icmpOutAddrMasks].
  163.               oadrmkr/s
  164.                      The number of ICMP Address Mask Reply messages sent
  165.                      per second [icmpOutAddrMaskReps].
  167.               With the EICMP keyword, statistics about ICMPv4 error
  168.               messages are reported.  Note that ICMPv4 statistics depend
  169.               on sadc's option -S SNMP to be collected.  The following
  170.               values are displayed (formal SNMP names between square
  171.               brackets):
  173.               ierr/s The number of ICMP messages per second which the
  174.                      entity received but determined as having ICMP-
  175.                      specific errors (bad ICMP checksums, bad length,
  176.                      etc.) [icmpInErrors].
  178.               oerr/s The number of ICMP messages per second which this
  179.                      entity did not send due to problems discovered
  180.                      within ICMP such as a lack of buffers
  181.                      [icmpOutErrors].
  183.               idstunr/s
  184.                      The number of ICMP Destination Unreachable messages
  185.                      received per second [icmpInDestUnreachs].
  187.               odstunr/s
  188.                      The number of ICMP Destination Unreachable messages
  189.                      sent per second [icmpOutDestUnreachs].
  191.               itmex/s
  192.                      The number of ICMP Time Exceeded messages received
  193.                      per second [icmpInTimeExcds].
  195.               otmex/s
  196.                      The number of ICMP Time Exceeded messages sent per
  197.                      second [icmpOutTimeExcds].
  199.               iparmpb/s
  200.                      The number of ICMP Parameter Problem messages
  201.                      received per second [icmpInParmProbs].
  203.               oparmpb/s
  204.                      The number of ICMP Parameter Problem messages sent
  205.                      per second [icmpOutParmProbs].
  207.               isrcq/s
  208.                      The number of ICMP Source Quench messages received
  209.                      per second [icmpInSrcQuenchs].
  211.               osrcq/s
  212.                      The number of ICMP Source Quench messages sent per
  213.                      second [icmpOutSrcQuenchs].
  215.               iredir/s
  216.                      The number of ICMP Redirect messages received per
  217.                      second [icmpInRedirects].
  219.               oredir/s
  220.                      The number of ICMP Redirect messages sent per
  221.                      second [icmpOutRedirects].
  223.               With the ICMP6 keyword, statistics about ICMPv6 network
  224.               traffic are reported.  Note that ICMPv6 statistics depend
  225.               on sadc's option -S IPV6 to be collected.  The following
  226.               values are displayed (formal SNMP names between square
  227.               brackets):
  229.               imsg6/s
  230.                      The total number of ICMP messages received by the
  231.                      interface per second which includes all those
  232.                      counted by ierr6/s [ipv6IfIcmpInMsgs].
  234.               omsg6/s
  235.                      The total number of ICMP messages which this
  236.                      interface attempted to send per second
  237.                      [ipv6IfIcmpOutMsgs].
  239.               iech6/s
  240.                      The number of ICMP Echo (request) messages received
  241.                      by the interface per second [ipv6IfIcmpInEchos].
  243.               iechr6/s
  244.                      The number of ICMP Echo Reply messages received by
  245.                      the interface per second [ipv6IfIcmpInEchoReplies].
  247.               oechr6/s
  248.                      The number of ICMP Echo Reply messages sent by the
  249.                      interface per second [ipv6IfIcmpOutEchoReplies].
  251.               igmbq6/s
  252.                      The number of ICMPv6 Group Membership Query
  253.                      messages received by the interface per second
  254.                      [ipv6IfIcmpInGroupMembQueries].
  256.               igmbr6/s
  257.                      The number of ICMPv6 Group Membership Response
  258.                      messages received by the interface per second
  259.                      [ipv6IfIcmpInGroupMembResponses].
  261.               ogmbr6/s
  262.                      The number of ICMPv6 Group Membership Response
  263.                      messages sent per second
  264.                      [ipv6IfIcmpOutGroupMembResponses].
  266.               igmbrd6/s
  267.                      The number of ICMPv6 Group Membership Reduction
  268.                      messages received by the interface per second
  269.                      [ipv6IfIcmpInGroupMembReductions].
  271.               ogmbrd6/s
  272.                      The number of ICMPv6 Group Membership Reduction
  273.                      messages sent per second
  274.                      [ipv6IfIcmpOutGroupMembReductions].
  276.               irtsol6/s
  277.                      The number of ICMP Router Solicit messages received
  278.                      by the interface per second
  279.                      [ipv6IfIcmpInRouterSolicits].
  281.               ortsol6/s
  282.                      The number of ICMP Router Solicitation messages
  283.                      sent by the interface per second
  284.                      [ipv6IfIcmpOutRouterSolicits].
  286.               irtad6/s
  287.                      The number of ICMP Router Advertisement messages
  288.                      received by the interface per second
  289.                      [ipv6IfIcmpInRouterAdvertisements].
  291.               inbsol6/s
  292.                      The number of ICMP Neighbor Solicit messages
  293.                      received by the interface per second
  294.                      [ipv6IfIcmpInNeighborSolicits].
  296.               onbsol6/s
  297.                      The number of ICMP Neighbor Solicitation messages
  298.                      sent by the interface per second
  299.                      [ipv6IfIcmpOutNeighborSolicits].
  301.               inbad6/s
  302.                      The number of ICMP Neighbor Advertisement messages
  303.                      received by the interface per second
  304.                      [ipv6IfIcmpInNeighborAdvertisements].
  306.               onbad6/s
  307.                      The number of ICMP Neighbor Advertisement messages
  308.                      sent by the interface per second
  309.                      [ipv6IfIcmpOutNeighborAdvertisements].
  311.               With the EICMP6 keyword, statistics about ICMPv6 error
  312.               messages are reported.  Note that ICMPv6 statistics depend
  313.               on sadc's option -S IPV6 to be collected.  The following
  314.               values are displayed (formal SNMP names between square
  315.               brackets):
  317.               ierr6/s
  318.                      The number of ICMP messages per second which the
  319.                      interface received but determined as having ICMP-
  320.                      specific errors (bad ICMP checksums, bad length,
  321.                      etc.)  [ipv6IfIcmpInErrors]
  323.               idtunr6/s
  324.                      The number of ICMP Destination Unreachable messages
  325.                      received by the interface per second
  326.                      [ipv6IfIcmpInDestUnreachs].
  328.               odtunr6/s
  329.                      The number of ICMP Destination Unreachable messages
  330.                      sent by the interface per second
  331.                      [ipv6IfIcmpOutDestUnreachs].
  333.               itmex6/s
  334.                      The number of ICMP Time Exceeded messages received
  335.                      by the interface per second
  336.                      [ipv6IfIcmpInTimeExcds].
  338.               otmex6/s
  339.                      The number of ICMP Time Exceeded messages sent by
  340.                      the interface per second [ipv6IfIcmpOutTimeExcds].
  342.               iprmpb6/s
  343.                      The number of ICMP Parameter Problem messages
  344.                      received by the interface per second
  345.                      [ipv6IfIcmpInParmProblems].
  347.               oprmpb6/s
  348.                      The number of ICMP Parameter Problem messages sent
  349.                      by the interface per second
  350.                      [ipv6IfIcmpOutParmProblems].
  352.               iredir6/s
  353.                      The number of Redirect messages received by the
  354.                      interface per second [ipv6IfIcmpInRedirects].
  356.               oredir6/s
  357.                      The number of Redirect messages sent by the
  358.                      interface by second [ipv6IfIcmpOutRedirects].
  360.               ipck2b6/s
  361.                      The number of ICMP Packet Too Big messages received
  362.                      by the interface per second
  363.                      [ipv6IfIcmpInPktTooBigs].
  365.               opck2b6/s
  366.                      The number of ICMP Packet Too Big messages sent by
  367.                      the interface per second [ipv6IfIcmpOutPktTooBigs].
  369.               With the IP keyword, statistics about IPv4 network traffic
  370.               are reported.  Note that IPv4 statistics depend on sadc's
  371.               option -S SNMP to be collected.  The following values are
  372.               displayed (formal SNMP names between square brackets):
  374.               irec/s The total number of input datagrams received from
  375.                      interfaces per second, including those received in
  376.                      error [ipInReceives].
  378.               fwddgm/s
  379.                      The number of input datagrams per second, for which
  380.                      this entity was not their final IP destination, as
  381.                      a result of which an attempt was made to find a
  382.                      route to forward them to that final destination
  383.                      [ipForwDatagrams].
  385.               idel/s The total number of input datagrams successfully
  386.                      delivered per second to IP user-protocols
  387.                      (including ICMP) [ipInDelivers].
  389.               orq/s  The total number of IP datagrams which local IP
  390.                      user-protocols (including ICMP) supplied per second
  391.                      to IP in requests for transmission [ipOutRequests].
  392.                      Note that this counter does not include any
  393.                      datagrams counted in fwddgm/s.
  395.               asmrq/s
  396.                      The number of IP fragments received per second
  397.                      which needed to be reassembled at this entity
  398.                      [ipReasmReqds].
  400.               asmok/s
  401.                      The number of IP datagrams successfully re-
  402.                      assembled per second [ipReasmOKs].
  404.               fragok/s
  405.                      The number of IP datagrams that have been
  406.                      successfully fragmented at this entity per second
  407.                      [ipFragOKs].
  409.               fragcrt/s
  410.                      The number of IP datagram fragments that have been
  411.                      generated per second as a result of fragmentation
  412.                      at this entity [ipFragCreates].
  414.               With the EIP keyword, statistics about IPv4 network errors
  415.               are reported.  Note that IPv4 statistics depend on sadc's
  416.               option -S SNMP to be collected.  The following values are
  417.               displayed (formal SNMP names between square brackets):
  419.               ihdrerr/s
  420.                      The number of input datagrams discarded per second
  421.                      due to errors in their IP headers, including bad
  422.                      checksums, version number mismatch, other format
  423.                      errors, time-to-live exceeded, errors discovered in
  424.                      processing their IP options, etc. [ipInHdrErrors]
  426.               iadrerr/s
  427.                      The number of input datagrams discarded per second
  428.                      because the IP address in their IP header's
  429.                      destination field was not a valid address to be
  430.                      received at this entity. This count includes
  431.                      invalid addresses (e.g., 0.0.0.0) and addresses of
  432.                      unsupported Classes (e.g., Class E). For entities
  433.                      which are not IP routers and therefore do not
  434.                      forward datagrams, this counter includes datagrams
  435.                      discarded because the destination address was not a
  436.                      local address [ipInAddrErrors].
  438.               iukwnpr/s
  439.                      The number of locally-addressed datagrams received
  440.                      successfully but discarded per second because of an
  441.                      unknown or unsupported protocol
  442.                      [ipInUnknownProtos].
  444.               idisc/s
  445.                      The number of input IP datagrams per second for
  446.                      which no problems were encountered to prevent their
  447.                      continued processing, but which were discarded
  448.                      (e.g., for lack of buffer space) [ipInDiscards].
  449.                      Note that this counter does not include any
  450.                      datagrams discarded while awaiting re-assembly.
  452.               odisc/s
  453.                      The number of output IP datagrams per second for
  454.                      which no problem was encountered to prevent their
  455.                      transmission to their destination, but which were
  456.                      discarded (e.g., for lack of buffer space)
  457.                      [ipOutDiscards].  Note that this counter would
  458.                      include datagrams counted in fwddgm/s if any such
  459.                      packets met this (discretionary) discard criterion.
  461.               onort/s
  462.                      The number of IP datagrams discarded per second
  463.                      because no route could be found to transmit them to
  464.                      their destination [ipOutNoRoutes].  Note that this
  465.                      counter includes any packets counted in fwddgm/s
  466.                      which meet this 'no-route' criterion.  Note that
  467.                      this includes any datagrams which a host cannot
  468.                      route because all of its default routers are down.
  470.               asmf/s The number of failures detected per second by the
  471.                      IP re-assembly algorithm (for whatever reason:
  472.                      timed out, errors, etc) [ipReasmFails].  Note that
  473.                      this is not necessarily a count of discarded IP
  474.                      fragments since some algorithms can lose track of
  475.                      the number of fragments by combining them as they
  476.                      are received.
  478.               fragf/s
  479.                      The number of IP datagrams that have been discarded
  480.                      per second because they needed to be fragmented at
  481.                      this entity but could not be, e.g., because their
  482.                      Don't Fragment flag was set [ipFragFails].
  484.               With the IP6 keyword, statistics about IPv6 network
  485.               traffic are reported.  Note that IPv6 statistics depend on
  486.               sadc's option -S IPV6 to be collected.  The following
  487.               values are displayed (formal SNMP names between square
  488.               brackets):
  490.               irec6/s
  491.                      The total number of input datagrams received from
  492.                      interfaces per second, including those received in
  493.                      error [ipv6IfStatsInReceives].
  495.               fwddgm6/s
  496.                      The number of output datagrams per second which
  497.                      this entity received and forwarded to their final
  498.                      destinations [ipv6IfStatsOutForwDatagrams].
  500.               idel6/s
  501.                      The total number of datagrams successfully
  502.                      delivered per second to IPv6 user-protocols
  503.                      (including ICMP) [ipv6IfStatsInDelivers].
  505.               orq6/s The total number of IPv6 datagrams which local IPv6
  506.                      user-protocols (including ICMP) supplied per second
  507.                      to IPv6 in requests for transmission
  508.                      [ipv6IfStatsOutRequests].  Note that this counter
  509.                      does not include any datagrams counted in
  510.                      fwddgm6/s.
  512.               asmrq6/s
  513.                      The number of IPv6 fragments received per second
  514.                      which needed to be reassembled at this interface
  515.                      [ipv6IfStatsReasmReqds].
  517.               asmok6/s
  518.                      The number of IPv6 datagrams successfully
  519.                      reassembled per second [ipv6IfStatsReasmOKs].
  521.               imcpck6/s
  522.                      The number of multicast packets received per second
  523.                      by the interface [ipv6IfStatsInMcastPkts].
  525.               omcpck6/s
  526.                      The number of multicast packets transmitted per
  527.                      second by the interface [ipv6IfStatsOutMcastPkts].
  529.               fragok6/s
  530.                      The number of IPv6 datagrams that have been
  531.                      successfully fragmented at this output interface
  532.                      per second [ipv6IfStatsOutFragOKs].
  534.               fragcr6/s
  535.                      The number of output datagram fragments that have
  536.                      been generated per second as a result of
  537.                      fragmentation at this output interface
  538.                      [ipv6IfStatsOutFragCreates].
  540.               With the EIP6 keyword, statistics about IPv6 network
  541.               errors are reported.  Note that IPv6 statistics depend on
  542.               sadc's option -S IPV6 to be collected.  The following
  543.               values are displayed (formal SNMP names between square
  544.               brackets):
  546.               ihdrer6/s
  547.                      The number of input datagrams discarded per second
  548.                      due to errors in their IPv6 headers, including
  549.                      version number mismatch, other format errors, hop
  550.                      count exceeded, errors discovered in processing
  551.                      their IPv6 options, etc. [ipv6IfStatsInHdrErrors]
  553.               iadrer6/s
  554.                      The number of input datagrams discarded per second
  555.                      because the IPv6 address in their IPv6 header's
  556.                      destination field was not a valid address to be
  557.                      received at this entity. This count includes
  558.                      invalid addresses (e.g., ::0) and unsupported
  559.                      addresses (e.g., addresses with unallocated
  560.                      prefixes). For entities which are not IPv6 routers
  561.                      and therefore do not forward datagrams, this
  562.                      counter includes datagrams discarded because the
  563.                      destination address was not a local address
  564.                      [ipv6IfStatsInAddrErrors].
  566.               iukwnp6/s
  567.                      The number of locally-addressed datagrams received
  568.                      successfully but discarded per second because of an
  569.                      unknown or unsupported protocol
  570.                      [ipv6IfStatsInUnknownProtos].
  572.               i2big6/s
  573.                      The number of input datagrams that could not be
  574.                      forwarded per second because their size exceeded
  575.                      the link MTU of outgoing interface
  576.                      [ipv6IfStatsInTooBigErrors].
  578.               idisc6/s
  579.                      The number of input IPv6 datagrams per second for
  580.                      which no problems were encountered to prevent their
  581.                      continued processing, but which were discarded
  582.                      (e.g., for lack of buffer space)
  583.                      [ipv6IfStatsInDiscards]. Note that this counter
  584.                      does not include any datagrams discarded while
  585.                      awaiting re-assembly.
  587.               odisc6/s
  588.                      The number of output IPv6 datagrams per second for
  589.                      which no problem was encountered to prevent their
  590.                      transmission to their destination, but which were
  591.                      discarded (e.g., for lack of buffer space)
  592.                      [ipv6IfStatsOutDiscards]. Note that this counter
  593.                      would include datagrams counted in fwddgm6/s if any
  594.                      such packets met this (discretionary) discard
  595.                      criterion.
  597.               inort6/s
  598.                      The number of input datagrams discarded per second
  599.                      because no route could be found to transmit them to
  600.                      their destination [ipv6IfStatsInNoRoutes].
  602.               onort6/s
  603.                      The number of locally generated IP datagrams
  604.                      discarded per second because no route could be
  605.                      found to transmit them to their destination
  606.                      [unknown formal SNMP name].
  608.               asmf6/s
  609.                      The number of failures detected per second by the
  610.                      IPv6 re-assembly algorithm (for whatever reason:
  611.                      timed out, errors, etc.) [ipv6IfStatsReasmFails].
  612.                      Note that this is not necessarily a count of
  613.                      discarded IPv6 fragments since some algorithms can
  614.                      lose track of the number of fragments by combining
  615.                      them as they are received.
  617.               fragf6/s
  618.                      The number of IPv6 datagrams that have been
  619.                      discarded per second because they needed to be
  620.                      fragmented at this output interface but could not
  621.                      be [ipv6IfStatsOutFragFails].
  623.               itrpck6/s
  624.                      The number of input datagrams discarded per second
  625.                      because datagram frame didn't carry enough data
  626.                      [ipv6IfStatsInTruncatedPkts].
  628.               With the NFS keyword, statistics about NFS client activity
  629.               are reported.  The following values are displayed:
  631.               call/s Number of RPC requests made per second.
  633.               retrans/s
  634.                      Number of RPC requests per second, those which
  635.                      needed to be retransmitted (for example because of
  636.                      a server timeout).
  638.               read/s Number of 'read' RPC calls made per second.
  640.               write/s
  641.                      Number of 'write' RPC calls made per second.
  643.               access/s
  644.                      Number of 'access' RPC calls made per second.
  646.               getatt/s
  647.                      Number of 'getattr' RPC calls made per second.
  649.               With the NFSD keyword, statistics about NFS server
  650.               activity are reported.  The following values are
  651.               displayed:
  653.               scall/s
  654.                      Number of RPC requests received per second.
  656.               badcall/s
  657.                      Number of bad RPC requests received per second,
  658.                      those whose processing generated an error.
  660.               packet/s
  661.                      Number of network packets received per second.
  663.               udp/s  Number of UDP packets received per second.
  665.               tcp/s  Number of TCP packets received per second.
  667.               hit/s  Number of reply cache hits per second.
  669.               miss/s Number of reply cache misses per second.
  671.               sread/s
  672.                      Number of 'read' RPC calls received per second.
  674.               swrite/s
  675.                      Number of 'write' RPC calls received per second.
  677.               saccess/s
  678.                      Number of 'access' RPC calls received per second.
  680.               sgetatt/s
  681.                      Number of 'getattr' RPC calls received per second.
  683.               With the SOCK keyword, statistics on sockets in use are
  684.               reported (IPv4).  The following values are displayed:
  686.               totsck Total number of sockets used by the system.
  688.               tcpsck Number of TCP sockets currently in use.
  690.               udpsck Number of UDP sockets currently in use.
  692.               rawsck Number of RAW sockets currently in use.
  694.               ip-frag
  695.                      Number of IP fragments currently in queue.
  697.               tcp-tw Number of TCP sockets in TIME_WAIT state.
  699.               With the SOCK6 keyword, statistics on sockets in use are
  700.               reported (IPv6).  Note that IPv6 statistics depend on
  701.               sadc's option -S IPV6 to be collected.  The following
  702.               values are displayed:
  704.               tcp6sck
  705.                      Number of TCPv6 sockets currently in use.
  707.               udp6sck
  708.                      Number of UDPv6 sockets currently in use.
  710.               raw6sck
  711.                      Number of RAWv6 sockets currently in use.
  713.               ip6-frag
  714.                      Number of IPv6 fragments currently in use.
  716.               With the SOFT keyword, statistics about software-based
  717.               network processing are reported.  The following values are
  718.               displayed:
  720.               total/s
  721.                      The total number of network frames processed per
  722.                      second.
  724.               dropd/s
  725.                      The total number of network frames dropped per
  726.                      second because there was no room on the processing
  727.                      queue.
  729.               squeezd/s
  730.                      The number of times the softirq handler function
  731.                      terminated per second because its budget was
  732.                      consumed or the time limit was reached, but more
  733.                      work could have been done.
  735.               rx_rps/s
  736.                      The number of times the CPU has been woken up per
  737.                      second to process packets via an inter-processor
  738.                      interrupt.
  740.               flw_lim/s
  741.                      The number of times the flow limit has been reached
  742.                      per second.  Flow limiting is an optional RPS
  743.                      feature that can be used to limit the number of
  744.                      packets queued to the backlog for each flow to a
  745.                      certain amount.  This can help ensure that smaller
  746.                      flows are processed even though much larger flows
  747.                      are pushing packets in.
  749.               With the TCP keyword, statistics about TCPv4 network
  750.               traffic are reported.  Note that TCPv4 statistics depend
  751.               on sadc's option -S SNMP to be collected.  The following
  752.               values are displayed (formal SNMP names between square
  753.               brackets):
  755.               active/s
  756.                      The number of times TCP connections have made a
  757.                      direct transition to the SYN-SENT state from the
  758.                      CLOSED state per second [tcpActiveOpens].
  760.               passive/s
  761.                      The number of times TCP connections have made a
  762.                      direct transition to the SYN-RCVD state from the
  763.                      LISTEN state per second [tcpPassiveOpens].
  765.               iseg/s The total number of segments received per second,
  766.                      including those received in error [tcpInSegs].
  767.                      This count includes segments received on currently
  768.                      established connections.
  770.               oseg/s The total number of segments sent per second,
  771.                      including those on current connections but
  772.                      excluding those containing only retransmitted
  773.                      octets [tcpOutSegs].
  775.               With the ETCP keyword, statistics about TCPv4 network
  776.               errors are reported.  Note that TCPv4 statistics depend on
  777.               sadc's option -S SNMP to be collected.  The following
  778.               values are displayed (formal SNMP names between square
  779.               brackets):
  781.               atmptf/s
  782.                      The number of times per second TCP connections have
  783.                      made a direct transition to the CLOSED state from
  784.                      either the SYN-SENT state or the SYN-RCVD state,
  785.                      plus the number of times per second TCP connections
  786.                      have made a direct transition to the LISTEN state
  787.                      from the SYN-RCVD state [tcpAttemptFails].
  789.               estres/s
  790.                      The number of times per second TCP connections have
  791.                      made a direct transition to the CLOSED state from
  792.                      either the ESTABLISHED state or the CLOSE-WAIT
  793.                      state [tcpEstabResets].
  795.               retrans/s
  796.                      The total number of segments retransmitted per
  797.                      second - that is, the number of TCP segments
  798.                      transmitted containing one or more previously
  799.                      transmitted octets [tcpRetransSegs].
  801.               isegerr/s
  802.                      The total number of segments received in error
  803.                      (e.g., bad TCP checksums) per second [tcpInErrs].
  805.               orsts/s
  806.                      The number of TCP segments sent per second
  807.                      containing the RST flag [tcpOutRsts].
  809.               With the UDP keyword, statistics about UDPv4 network
  810.               traffic are reported.  Note that UDPv4 statistics depend
  811.               on sadc's option -S SNMP to be collected.  The following
  812.               values are displayed (formal SNMP names between square
  813.               brackets):
  815.               idgm/s The total number of UDP datagrams delivered per
  816.                      second to UDP users [udpInDatagrams].
  818.               odgm/s The total number of UDP datagrams sent per second
  819.                      from this entity [udpOutDatagrams].
  821.               noport/s
  822.                      The total number of received UDP datagrams per
  823.                      second for which there was no application at the
  824.                      destination port [udpNoPorts].
  826.               idgmerr/s
  827.                      The number of received UDP datagrams per second
  828.                      that could not be delivered for reasons other than
  829.                      the lack of an application at the destination port
  830.                      [udpInErrors].
  832.               With the UDP6 keyword, statistics about UDPv6 network
  833.               traffic are reported.  Note that UDPv6 statistics depend
  834.               on sadc's option -S IPV6 to be collected.  The following
  835.               values are displayed (formal SNMP names between square
  836.               brackets):
  838.               idgm6/s
  839.                      The total number of UDP datagrams delivered per
  840.                      second to UDP users [udpInDatagrams].
  842.               odgm6/s
  843.                      The total number of UDP datagrams sent per second
  844.                      from this entity [udpOutDatagrams].
  846.               noport6/s
  847.                      The total number of received UDP datagrams per
  848.                      second for which there was no application at the
  849.                      destination port [udpNoPorts].
  851.               idgmer6/s
  852.                      The number of received UDP datagrams per second
  853.                      that could not be delivered for reasons other than
  854.                      the lack of an application at the destination port
  855.                      [udpInErrors].
  857.               The ALL keyword is equivalent to specifying all the
  858.               keywords above and therefore all the network activities
  859.               are reported.