参考及疑问

参考文献

疑问

在开始之前，有几个疑问：

  • PCIE是端到端的数据传输，那么 PCIE链路是如何扩展的？
  • 如何从Linux环境下来 分析自己的PCIE链路（实战）？

在分析完后，也存在了几个疑惑：

  • 软件代码是如何初始化的？
  • 我的主机里边，为啥总线没有bus2，直接从bus1 到了bus3（BUS号不连续？）
  • 在前边我们知道，非ARI设备，Device的编号间隔是8， 为什么我的设备怎么这么奇怪？
    1. 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1480
    2. 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 1481
    3. 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
    4. 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
    5. 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
    6. 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
    7. 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
    8. 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
    9. 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
    10. 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
    11. 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
    12. 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484

Pcie链路的扩展

Switch的作用

在《PCI+E 体系结构》4.1.4 链路扩展 章节中描述：
PCIE 的链路通信是 端到端的数据传输， 在一条PCIE链路中，分别是发送和接收设备，因此产生了PCIE Switch进行扩展
image.png

Switch设备就和USB Hub一样，支持 1个上游端口（连接主机或者RC）， 2-n个下游端口（扩展口）。
image.png

因此，现代计算机的结构类似如下：
image.png
image.png

总结：

  • CPU平时非常的忙，便把很多事交给（Root Complex 简称RC）来干，RC可以访问内存，通过内部PCIe总线以及若干个PCIe Bridge，扩展出若干个其他的PCIe端口， RC内部十分复杂。
  • PCIe终端设备称为 Endpoint，比如 PCIe SSD，PCIe 网卡等等，Endpoint可以挂载到 RC 上，也可以挂载到Switch上。
  • switch(交换器)扩展了PCIe端口，将一个PCIe端口扩展为若干个，靠近RC的端口称为上游端口，扩展出来的端口称为下游端口。下游端口可以挂载其他switch或者Endpoint，并且对他们进行管理。
  • 从上游过来的数据，它需要鉴定：

1.是否是传给自己的数据，如果是便接收。
2. 是否是自己下游端口的数据，如果是便转发
3.如果都不是，便拒绝。

  • 从下游端口挂载的Endpoint传给RC的数据，switch也会检查并转发。

PCI-e总线的扩展

sec - secondary bus，下一级bus
pri - primary bus, 上一级bus
sub - subordinate bus，最末级bus。
image.png

分析自己的PCIE链路

基础信息

X86下PCIE的结构如下：
image.png

系统信息

根据系统 可以分析到：

  1. (base) baiy@inno-MS-7B89:vf_testpcie$ lspci
  2. 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1480
  3. 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 1481
  5. 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  6. 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
  7. 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
  9. 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  11. 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  12. 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
  15. 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  16. 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  17. 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  18. 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
  20. 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  21. 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
  22. 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
  23. 00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
  25. 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
  26. 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
  27. 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1440
  28. 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1441
  29. 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1442
  30. 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1443
  31. 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1444
  32. 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1445
  33. 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1446
  34. 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1447
  37. 01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
  38. 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43d5 (rev 01)
  39. 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43c8 (rev 01)
  40. 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c6 (rev 01)
  41. 20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01)
  42. 20:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01)
  43. 20:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43c7 (rev 01)
  44. 22:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
  45. 26:00.0 VGA compatible controller: NVIDIA Corporation Device 1f82 (rev a1)
  46. 26:00.1 Audio device: NVIDIA Corporation Device 10fa (rev a1)
  47. 27:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 148a
  48. 28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1485
  49. 28:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1486
  50. 28:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 149c
  51. 28:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1487
  52. 30:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
  53. 31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
  55. (base) baiy@inno-MS-7B89:vf_testpcie$ ls /sys/bus/pci/devices/ -l
  56.  0000:00:00.0 -> ../../../devices/pci0000:00/0000:00:00.0
  57.  0000:00:00.2 -> ../../../devices/pci0000:00/0000:00:00.2
  58.  0000:00:01.0 -> ../../../devices/pci0000:00/0000:00:01.0
  59.  0000:00:01.1 -> ../../../devices/pci0000:00/0000:00:01.1
  60.  0000:00:01.3 -> ../../../devices/pci0000:00/0000:00:01.3
  61.  0000:00:02.0 -> ../../../devices/pci0000:00/0000:00:02.0
  62.  0000:00:03.0 -> ../../../devices/pci0000:00/0000:00:03.0
  63.  0000:00:03.1 -> ../../../devices/pci0000:00/0000:00:03.1
  64.  0000:00:04.0 -> ../../../devices/pci0000:00/0000:00:04.0
  65.  0000:00:05.0 -> ../../../devices/pci0000:00/0000:00:05.0
  66.  0000:00:07.0 -> ../../../devices/pci0000:00/0000:00:07.0
  67.  0000:00:07.1 -> ../../../devices/pci0000:00/0000:00:07.1
  68.  0000:00:08.0 -> ../../../devices/pci0000:00/0000:00:08.0
  69.  0000:00:08.1 -> ../../../devices/pci0000:00/0000:00:08.1
  70.  0000:00:08.2 -> ../../../devices/pci0000:00/0000:00:08.2
  71.  0000:00:08.3 -> ../../../devices/pci0000:00/0000:00:08.3
  72.  0000:00:14.0 -> ../../../devices/pci0000:00/0000:00:14.0
  73.  0000:00:14.3 -> ../../../devices/pci0000:00/0000:00:14.3
  74.  0000:00:18.0 -> ../../../devices/pci0000:00/0000:00:18.0
  75.  0000:00:18.1 -> ../../../devices/pci0000:00/0000:00:18.1
  76.  0000:00:18.2 -> ../../../devices/pci0000:00/0000:00:18.2
  77.  0000:00:18.3 -> ../../../devices/pci0000:00/0000:00:18.3
  78.  0000:00:18.4 -> ../../../devices/pci0000:00/0000:00:18.4
  79.  0000:00:18.5 -> ../../../devices/pci0000:00/0000:00:18.5
  80.  0000:00:18.6 -> ../../../devices/pci0000:00/0000:00:18.6
  81.  0000:00:18.7 -> ../../../devices/pci0000:00/0000:00:18.7
  83.  0000:01:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0
  84.  0000:03:00.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.0
  85.  0000:03:00.1 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.1
  86.  0000:03:00.2 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2
  87.  0000:20:00.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:00.0
  88.  0000:20:01.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:01.0
  89.  0000:20:04.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:04.0
  90.  0000:22:00.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:01.0/0000:22:00.0
  91.  0000:26:00.0 -> ../../../devices/pci0000:00/0000:00:03.1/0000:26:00.0
  92.  0000:26:00.1 -> ../../../devices/pci0000:00/0000:00:03.1/0000:26:00.1
  93.  0000:27:00.0 -> ../../../devices/pci0000:00/0000:00:07.1/0000:27:00.0
  94.  0000:28:00.0 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.0
  95.  0000:28:00.1 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.1
  96.  0000:28:00.3 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.3
  97.  0000:28:00.4 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.4
  98.  0000:30:00.0 -> ../../../devices/pci0000:00/0000:00:08.2/0000:30:00.0
  99.  0000:31:00.0 -> ../../../devices/pci0000:00/0000:00:08.3/0000:31:00.0
  101. (base) baiy@inno-MS-7B89:vf_testpcie$ lspci -t
  102. -[0000:00]-+-00.0
  103.            +-00.2
  104.            +-01.0
  105.            +-01.1-[01]----00.0
  106.            +-01.3-[03-25]--+-00.0
  107.            |               +-00.1
  108.            |               \-00.2-[20-25]--+-00.0-[21]--
  109.            |                               +-01.0-[22]----00.0
  110.            |                               \-04.0-[25]--
  111.            +-02.0
  112.            +-03.0
  113.            +-03.1-[26]--+-00.0
  114.            |            \-00.1
  115.            +-04.0
  116.            +-05.0
  117.            +-07.0
  118.            +-07.1-[27]----00.0
  119.            +-08.0
  120.            +-08.1-[28]--+-00.0
  121.            |            +-00.1
  122.            |            +-00.3
  123.            |            \-00.4
  124.            +-08.2-[30]----00.0
  125.            +-08.3-[31]----00.0
  126.            +-14.0
  127.            +-14.3
  128.            +-18.0
  129.            +-18.1
  130.            +-18.2
  131.            +-18.3
  132.            +-18.4
  133.            +-18.5
  134.            +-18.6
  135.            \-18.7

分析方式

确定主桥个数

首先，确定当前主机分配了几个主桥，以及每个主桥的资源信息
dmesg | grep “PCI Root Bridge” 查看系统启动信息，可以知道 当前RC下边挂了几个Swiitch（大部分机器都只有一个SEG，所以不管多少主桥，BUS号都在00-ff之间）

  1. (base) baiy@inno-MS-7B89:~$ dmesg | grep "PCI Root Bridge"
  2. [    0.500591] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
  5. PCI总线00的资源信息
  6. [    0.500591] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
  7. [    0.502397] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
  8. [    0.506523] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug PME LTR]
  9. [    0.510513] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug AER PCIeCapability]
  10. [    0.514399] acpi PNP0A08:00: [Firmware Info]: MMCONFIG for domain 0000 [bus 00-3f] only partially covers this bridge
  11. [    0.518652] PCI host bridge to bus 0000:00
  12. [    0.522393] pci_bus 0000:00: root bus resource [io  0x0000-0x03af window]
  13. [    0.526392] pci_bus 0000:00: root bus resource [io  0x03e0-0x0cf7 window]
  14. [    0.530392] pci_bus 0000:00: root bus resource [io  0x03b0-0x03df window]
  15. [    0.534392] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
  16. [    0.538394] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
  17. [    0.542393] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff window]
  18. [    0.546392] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfec2ffff window]
  19. [    0.550392] pci_bus 0000:00: root bus resource [mem 0xfee00000-0xffffffff window]
  20. [    0.554392] pci_bus 0000:00: root bus resource [bus 00-ff]

所以，我这个设备只有一个主桥，按照如下方式链接的。
image.png

分析桥连接情况

先看一个设备

  1. 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
  2. 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
  3. 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483

这里
00:01.0 的ClassCode 为 00 00 06 ，对应是 Host Bridge 设备
00:01.1 和 00:01.3 的Class Code为 00 04 06，对应是 PCI bridge 设备

  1. C 06  Bridge                                                                                                                                                                                                                                                                  
  2.     00  Host bridge
  3.     04  PCI bridge
  4.         00  Normal decode
  1. 其次，我们根据 /sys/bus/pci/devices/ 分析了设备的位置，
  1.  0000:01:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0
  2.  0000:03:00.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.0
  3.  0000:03:00.1 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.1
  4.  0000:03:00.2 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2
  5.  0000:20:00.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:00.0
  6.  0000:20:01.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:01.0
  7.  0000:20:04.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:04.0
  8.  0000:22:00.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:03:00.2/0000:20:01.0/0000:22:00.0
  9.  0000:26:00.0 -> ../../../devices/pci0000:00/0000:00:03.1/0000:26:00.0
  10.  0000:26:00.1 -> ../../../devices/pci0000:00/0000:00:03.1/0000:26:00.1
  11.  0000:27:00.0 -> ../../../devices/pci0000:00/0000:00:07.1/0000:27:00.0
  12.  0000:28:00.0 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.0
  13.  0000:28:00.1 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.1
  14.  0000:28:00.3 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.3
  15.  0000:28:00.4 -> ../../../devices/pci0000:00/0000:00:08.1/0000:28:00.4
  16.  0000:30:00.0 -> ../../../devices/pci0000:00/0000:00:08.2/0000:30:00.0
  17.  0000:31:00.0 -> ../../../devices/pci0000:00/0000:00:08.3/0000:31:00.0
  1. 最后，根据lspci 查看设备类型，确定设备(先不考虑多功能设备)结构如下： 1.PCIE链路拓展 - 图8

其实通过 /sys/bus/pci/devices/ 进行总线分析 是最好用的手段
可以用以下 参考代码 来遍历所有总线及设备：

  1. static void dump_pci_tree(struct pci_bus * bus)
  2. {
  3.     struct pci_dev *dev;
  4.     pr_info("PCI BUS is %d, primary is %d\n", bus->number, bus->primary);
  5.     list_for_each_entry(dev, &bus->devices, bus_list) {
  6.         struct pci_bus *b = dev->subordinate;
  7.         printk("%.2x:%.2x.%.2x: vendor=%x,device=%x\n",
  8.              dev->bus->number, dev->devfn >> 3,dev->devfn & 7 ,dev->vendor,dev->device);
  9.         if(b)
  10.             dump_pci_tree(b);
  11.     }
  12. }
  14. list_for_each_entry(bus, &pci_root_buses, node){
  15.     dump_pci_tree(bus);
  16. }