1. server (quicsample)

1.1 接口调用

1.1.1 begin

:::info

  1. MsQuicOpen(QUIC_API_TABLE MsQuic)
    2. MsQuicRegistrationOpen(QUIC_REGISTRATION_CONFIG     Config, HQUIC Registration)
    3. MsQuicConfigurationOpen(HQUIC Registration, QUIC_BUFFER     AlpnBuffers, uint32_t AlpnBufferCount, QUIC_SETTINGS Settings, uint32_t SettingsSize, void Context, HQUIC Configuration)
    4. MsQuicConfigurationLoadCredential(HQUIC Configuration, QUIC_CREDENTIAL_CONFIG     CredConfig) :::

    1.1.2 listen

    :::info
  2. MsQuicListenerOpen(HQUIC Registration, QUIC_LISTENER_CALLBACK_HANDLER Handler, void Context, HQUIC NewListener) —-> listen cb
    6. MsQuicListenerStart(HQUIC Handle, QUIC_BUFFER AlpnBuffers, uint32_t AlpnBufferCount, QUIC_ADDR LocalAddress)
    —-> MsQuicListenerClose(HQUIC Handle) :::

    1.1.3 listen cb

    :::info
  3. MsQuicSetCallbackHandler(HQUIC Handle, void Handler, void Context) —-> connection cb
    8. MsQuicConnectionSetConfiguration(HQUIC Handle, HQUIC ConfigHandle) :::

    1.1.4 connection cb

    :::info
  4. MsQuicConnectionSendResumptionTicket(HQUIC Handle, QUIC_SEND_RESUMPTION_FLAGS Flags, uint16_t DataLength, uint8_t ResumptionData)
    10. MsQuicSetCallbackHandler(HQUIC Handle, void     Handler, void *Context) —-> stream cb
    —-> MsQuicConnectionClose(HQUIC Handle) :::

    1.1.5 stream cb

    :::info
  5. MsQuicStreamSend(HQUIC Handle, QUIC_BUFFER Buffers, uint32_t BufferCount, QUIC_SEND_FLAGS Flags, void ClientSendContext)
    —-> StreamShutdown(HQUIC Handle, QUIC_STREAM_SHUTDOWN_FLAGS Flags, QUIC_UINT62 ErrorCode)
    14. MsQuicStreamClose(HQUIC Handle)
    16. MsQuicConnectionClose(HQUIC Handle) :::

    1.1.6 end

    :::info
  6. MsQuicConfigurationClose(HQUIC Handle)
    18. MsQuicRegistrationClose(HQUIC Handle)
    19. MsQuicClose(QUIC_API_TABLE* MsQuic) :::

    1.2 状态转换

    1.2.1 listen cb

    :::info 7.8. QUIC_LISTENER_EVENT_NEW_CONNECTION :::

    1.2.1 connection cb

    :::info
  7. QUIC_CONNECTION_EVENT_CONNECTED
    10. QUIC_CONNECTION_EVENT_PEER_STREAM_STARTED :::

    1.2.1 stream cb

    :::info
  8. QUIC_STREAM_EVENT_RECEIVE
    12. QUIC_STREAM_EVENT_PEER_SEND_SHUTDOWN
    13. QUIC_STREAM_EVENT_SEND_COMPLETE
    14. QUIC_STREAM_EVENT_SHUTDOWN_COMPLETE :::

    1.2.1 close

    :::info
  9. QUIC_CONNECTION_EVENT_SHUTDOWN_INITIATED_BY_TRANSPORT
    16. QUIC_CONNECTION_EVENT_SHUTDOWN_COMPLETE :::

    1.3 调用堆栈

    1.3.1 线程启动

  10. QuicConnDrainOperations是动作区分函数，包括各种动作：
    连接处理：QuicConnProcessApiOperation —-> CONN：close,shutdown,start,SetConf,SendResumptionTicket /
    Stream：close,shutdown,start,SendFlush,RecvComplete,RecvSetEnabledState
    连接接收：QuicConnFlushRecv
    流接收：QuicStreamRecvFlush
    超时：QuicConnProcessExpiredTimer

开启线程数：cpu (open + registration + listen) = cpu(4) 3
open线程：

  1. #0  CxPlatThreadCreate -> CxPlatDataPathWorkerThread (epoll_wait)
  2. #1  CxPlatProcessorContextInitialize (Datapath=Datapath@entry=0x605560, Index=Index@entry=0, ProcContext=ProcContext@entry=0x605600) at /root/source/msquic/src/platform/datapath_epoll.c:608
  3. #2  0x00007ffff7961b57 in CxPlatDataPathInitialize (ClientRecvContextLength=ClientRecvContextLength@entry=48, UdpCallbacks=UdpCallbacks@entry=0x7fffffffddd0, TcpCallbacks=TcpCallbacks@entry=0x0, NewDataPath=NewDataPath@entry=0x7ffff7dc97e0 <MsQuicLib+320>) at /root/source/msquic/src/platform/datapath_epoll.c:701
  4. #3  0x00007ffff7925c9a in MsQuicLibraryInitialize () at /root/source/msquic/src/core/library.c:393
  5. #4  0x00007ffff7926271 in MsQuicAddRef () at /root/source/msquic/src/core/library.c:598
  6. #5  0x00007ffff79262ca in MsQuicOpen (QuicApi=QuicApi@entry=0x603080 <MsQuic>) at /root/source/msquic/src/core/library.c:1325
  7. #6  0x00007ffff79264b9 in MsQuicOpenVersion (Version=Version@entry=1, QuicApi=QuicApi@entry=0x603080 <MsQuic>) at /root/source/msquic/src/core/library.c:1405
  8. #7  0x0000000000401717 in main (argc=4, argv=0x7fffffffdf68) at /root/source/msquic/src/tools/sample/sample.c:865

registration线程：

  1. #0  CxPlatThreadCreate -> QuicWorkerThread
  2. #1  QuicWorkerInitialize (Owner=Owner@entry=0x606190, ThreadFlags=ThreadFlags@entry=0, IdealProcessor=IdealProcessor@entry=1, Worker=Worker@entry=0x6069a0) at /root/source/msquic/src/core/worker.c:83
  3. #2  0x00007ffff7932041 in QuicWorkerPoolInitialize (Owner=Owner@entry=0x606190, ThreadFlags=0, WorkerCount=4, NewWorkerPool=0x6061c8) at /root/source/msquic/src/core/worker.c:783
  4. #3  0x00007ffff79285c6 in MsQuicRegistrationOpen (Config=0x602da0 <RegConfig>, NewRegistration=0x603070 <Registration>) at /root/source/msquic/src/core/registration.c:120
  5. #4  0x000000000040173a in main (argc=4, argv=0x7fffffffdf68) at /root/source/msquic/src/tools/sample/sample.c:873

listen线程：

  1. #0  CxPlatThreadCreate -> QuicWorkerThread
  2. #1  QuicWorkerInitialize (Owner=Owner@entry=0x62af20, ThreadFlags=ThreadFlags@entry=0, IdealProcessor=IdealProcessor@entry=2, Worker=Worker@entry=0x6edb70) at /root/source/msquic/src/core/worker.c:83
  3. #2  0x00007ffff7932041 in QuicWorkerPoolInitialize (Owner=Owner@entry=0x62af20, ThreadFlags=0, WorkerCount=4, NewWorkerPool=0x62af58) at /root/source/msquic/src/core/worker.c:783
  4. #3  0x00007ffff79285c6 in MsQuicRegistrationOpen (Config=Config@entry=0x7fffffffdd70, NewRegistration=NewRegistration@entry=0x7ffff7dc9808 <MsQuicLib+360>) at /root/source/msquic/src/core/registration.c:120
  5. #4  0x00007ffff7925152 in QuicLibraryOnListenerRegistered (Listener=Listener@entry=0x6eb380) at /root/source/msquic/src/core/library.c:1778  (StatelessRegistration)
  6. #5  0x00007ffff7926b2c in MsQuicListenerStart (Handle=0x6eb380, AlpnBuffers=0x602d90 <Alpn>, AlpnBufferCount=1, LocalAddress=0x7fffffffde30) at /root/source/msquic/src/core/listener.c:256
  7. #6  0x000000000040112b in RunServer (argc=argc@entry=4, argv=argv@entry=0x7fffffffdf68) at /root/source/msquic/src/tools/sample/sample.c:532
  8. #7  0x000000000040183e in main (argc=4, argv=0x7fffffffdf68) at /root/source/msquic/src/tools/sample/sample.c:883

1.3.2 listen cb

7.8. QUIC_LISTENER_EVENT_NEW_CONNECTION

  1. #0  ServerListenerCallback (Listener=<optimized out>, Context=<optimized out>, Event=0x7ffff1ee8940) at /root/source/msquic/src/tools/sample/sample.c:379
  2. #1  0x00007ffff7926e85 in QuicListenerIndicateEvent (Listener=Listener@entry=0x6eb380, Event=Event@entry=0x7ffff1ee8940) at /root/source/msquic/src/core/listener.c:410
  3. #2  0x00007ffff7927038 in QuicListenerClaimConnection (Listener=Listener@entry=0x6eb380, Connection=Connection@entry=0x7fffec000b60, Info=Info@entry=0x7ffff1ee89e0) at /root/source/msquic/src/core/listener.c:521
  4. #3  0x00007ffff7927164 in QuicListenerAcceptConnection (Listener=Listener@entry=0x6eb380, Connection=Connection@entry=0x7fffec000b60, Info=Info@entry=0x7ffff1ee89e0) at /root/source/msquic/src/core/listener.c:583
  5. #4  0x00007ffff7935d0c in QuicBindingAcceptConnection (Binding=<optimized out>, Connection=Connection@entry=0x7fffec000b60, Info=Info@entry=0x7ffff1ee89e0) at /root/source/msquic/src/core/binding.c:550
  6. #5  0x00007ffff79457da in QuicCryptoProcessData (Crypto=Crypto@entry=0x7fffec001380, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1759
  7. #6  0x00007ffff7945b22 in QuicCryptoProcessFrame (Crypto=Crypto@entry=0x7fffec001380, KeyType=<optimized out>, Frame=Frame@entry=0x7ffff1ee8af0) at /root/source/msquic/src/core/crypto.c:1250
  8. #7  0x00007ffff793f445 in QuicConnRecvFrames (Connection=Connection@entry=0x7fffec000b60, Path=<optimized out>, Packet=Packet@entry=0x71a940, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4203
  9. #8  0x00007ffff79409d5 in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x7fffec000b60, Path=<optimized out>, Path@entry=0x7fffec000c88, BatchCount=<optimized out>, Datagrams=Datagrams@entry=0x7ffff1ee8d10, Cipher=Cipher@entry=0x7ffff1ee8c90 "\200^\001i\313*\004\235\272\326\f\252\354+\366R\330\023", RecvState=RecvState@entry=0x7ffff1ee8d54) at /root/source/msquic/src/core/connection.c:5071
  10. #9  0x00007ffff79410e2 in QuicConnRecvDatagrams (Connection=Connection@entry=0x7fffec000b60, DatagramChain=<optimized out>, DatagramChain@entry=0x71a328, DatagramChainCount=DatagramChainCount@entry=1, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5276
  11. #10 0x00007ffff794151b in QuicConnFlushRecv (Connection=Connection@entry=0x7fffec000b60) at /root/source/msquic/src/core/connection.c:5420
  12. #11 0x00007ffff79434cd in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec000b60) at /root/source/msquic/src/core/connection.c:6831
  13. #12 0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x6edb70, Connection=Connection@entry=0x7fffec000b60, ThreadID=ThreadID@entry=188917) at /root/source/msquic/src/core/worker.c:489
  14. #13 0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x6edb70, ThreadID=ThreadID@entry=188917, Reason=<optimized out>, Delay=Delay@entry=0x7ffff1ee8ec8) at /root/source/msquic/src/core/worker.c:644
  15. #14 0x00007ffff7931e00 in QuicWorkerThread (Context=0x6edb70) at /root/source/msquic/src/core/worker.c:727
  16. #15 0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
  17. #16 0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6

1.3.3 connection cb

  1. QUIC_CONNECTION_EVENT_CONNECTED
    1. #0  ServerConnectionCallback (Connection=0x7fffec0037c0, Context=<optimized out>, Event=0x7ffff3eec930) at /root/source/msquic/src/tools/sample/sample.c:306
    2. #1  0x00007ffff7939aaf in QuicConnIndicateEvent (Connection=Connection@entry=0x7fffec0037c0, Event=Event@entry=0x7ffff3eec930) at /root/source/msquic/src/core/connection.c:692
    3. #2  0x00007ffff7944f04 in QuicCryptoProcessTlsCompletion (Crypto=Crypto@entry=0x7fffec003fe0) at /root/source/msquic/src/core/crypto.c:1582
    4. #3  0x00007ffff79455a3 in QuicCryptoProcessDataComplete (Crypto=Crypto@entry=0x7fffec003fe0, RecvBufferConsumed=<optimized out>) at /root/source/msquic/src/core/crypto.c:1630
    5. #4  0x00007ffff7945652 in QuicCryptoProcessData (Crypto=Crypto@entry=0x7fffec003fe0, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1786
    6. #5  0x00007ffff7945b22 in QuicCryptoProcessFrame (Crypto=Crypto@entry=0x7fffec003fe0, KeyType=<optimized out>, Frame=Frame@entry=0x7ffff3eecaf0) at /root/source/msquic/src/core/crypto.c:1250
    7. #6  0x00007ffff793f445 in QuicConnRecvFrames (Connection=Connection@entry=0x7fffec0037c0, Path=<optimized out>, Packet=Packet@entry=0x71b600, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4203
    8. #7  0x00007ffff79409d5 in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x7fffec0037c0, Path=<optimized out>, Path@entry=0x7fffec0038e8, BatchCount=<optimized out>, Datagrams=Datagrams@entry=0x7ffff3eecd10, Cipher=Cipher@entry=0x7ffff3eecc90 "\227\v\312;#\a\224\304\003p\336Pϫ,\267", RecvState=RecvState@entry=0x7ffff3eecd54) at /root/source/msquic/src/core/connection.c:5071
    9. #8  0x00007ffff79410e2 in QuicConnRecvDatagrams (Connection=Connection@entry=0x7fffec0037c0, DatagramChain=<optimized out>, DatagramChain@entry=0x71afe8, DatagramChainCount=DatagramChainCount@entry=3, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5276
    10. #9  0x00007ffff794151b in QuicConnFlushRecv (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:5420
    11. #10 0x00007ffff79434cd in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:6831
    12. #11 0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec0037c0, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    13. #12 0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    14. #13 0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    15. #14 0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    16. #15 0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6
  2. QUIC_CONNECTION_EVENT_PEER_STREAM_STARTED
    1. #0  ServerConnectionCallback (Connection=0x7fffec0037c0, Context=<optimized out>, Event=0x7ffff3eeca30) at /root/source/msquic/src/tools/sample/sample.c:340
    2. #1  0x00007ffff7939aaf in QuicConnIndicateEvent (Connection=Connection@entry=0x7fffec0037c0, Event=Event@entry=0x7ffff3eeca30) at /root/source/msquic/src/core/connection.c:692
    3. #2  0x00007ffff792f840 in QuicStreamSetGetStreamForPeer (StreamSet=StreamSet@entry=0x7fffec003e60, StreamId=StreamId@entry=0, FrameIn0Rtt=0 '\000', CreateIfMissing=<optimized out>, FatalError=FatalError@entry=0x7ffff3eecae0 "") at /root/source/msquic/src/core/stream_set.c:684
    4. #3  0x00007ffff793f76b in QuicConnRecvFrames (Connection=Connection@entry=0x7fffec0037c0, Path=<optimized out>, Packet=Packet@entry=0x7fffec003780, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4331
    5. #4  0x00007ffff79409d5 in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x7fffec0037c0, Path=<optimized out>, Path@entry=0x7fffec0038e8, BatchCount=BatchCount@entry=2 '\002', Datagrams=Datagrams@entry=0x7ffff3eecd10, Cipher=Cipher@entry=0x7ffff3eecc90 "+\222Yu\227\213wW\026\020诚\301o3\322\333O_:T\331\326$Pۘ\213", <incomplete sequence \367\275\260>, RecvState=RecvState@entry=0x7ffff3eecd54) at /root/source/msquic/src/core/connection.c:5071
    6. #5  0x00007ffff79412c0 in QuicConnRecvDatagrams (Connection=Connection@entry=0x7fffec0037c0, DatagramChain=0x0, DatagramChain@entry=0x71afe8, DatagramChainCount=DatagramChainCount@entry=3, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5336
    7. #6  0x00007ffff794151b in QuicConnFlushRecv (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:5420
    8. #7  0x00007ffff79434cd in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:6831
    9. #8  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec0037c0, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    10. #9  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    11. #10 0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    12. #11 0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    13. #12 0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6

    1.3.4 stream cb

  3. QUIC_STREAM_EVENT_RECEIVE
    1. #0  ServerStreamCallback (Stream=0x7fffe8008cf0, Context=0x0, Event=0x7ffff3eecd90) at /root/source/msquic/src/tools/sample/sample.c:256
    2. #1  0x00007ffff792b1f8 in QuicStreamIndicateEvent (Stream=Stream@entry=0x7fffe8008cf0, Event=Event@entry=0x7ffff3eecd90) at /root/source/msquic/src/core/stream.c:410
    3. #2  0x00007ffff792c4ab in QuicStreamRecvFlush (Stream=0x7fffe8008cf0) at /root/source/msquic/src/core/stream_recv.c:776
    4. #3  0x00007ffff79435b4 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:6841
    5. #4  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec0037c0, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    6. #5  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    7. #6  0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    8. #7  0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    9. #8  0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6
  4. QUIC_STREAM_EVENT_PEER_SEND_SHUTDOWN
    1. #0  ServerStreamCallback (Stream=0x7fffe8008cf0, Context=0x0, Event=0x7ffff3eecd20) at /root/source/msquic/src/tools/sample/sample.c:262
    2. #1  0x00007ffff792b1f8 in QuicStreamIndicateEvent (Stream=Stream@entry=0x7fffe8008cf0, Event=Event@entry=0x7ffff3eecd20) at /root/source/msquic/src/core/stream.c:410
    3. #2  0x00007ffff792c272 in QuicStreamReceiveComplete (Stream=Stream@entry=0x7fffe8008cf0, BufferLength=100) at /root/source/msquic/src/core/stream_recv.c:934
    4. #3  0x00007ffff792c3fe in QuicStreamRecvFlush (Stream=0x7fffe8008cf0) at /root/source/msquic/src/core/stream_recv.c:828
    5. #4  0x00007ffff79435b4 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:6841
    6. #5  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec0037c0, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    7. #6  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    8. #7  0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    9. #8  0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    10. #9  0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6
  5. QUIC_STREAM_EVENT_SEND_COMPLETE
    1. #0  ServerStreamCallback (Stream=0x7fffe8008cf0, Context=<optimized out>, Event=<optimized out>) at /root/source/msquic/src/tools/sample/sample.c:250
    2. #1  0x00007ffff792b1f8 in QuicStreamIndicateEvent (Stream=Stream@entry=0x7fffe8008cf0, Event=Event@entry=0x7ffff3eecce0) at /root/source/msquic/src/core/stream.c:410
    3. #2  0x00007ffff792cafa in QuicStreamSendBufferRequest (Stream=Stream@entry=0x7fffe8008cf0, Req=Req@entry=0x7fffe8019f30) at /root/source/msquic/src/core/stream_send.c:475
    4. #3  0x00007ffff7958591 in QuicSendBufferFill (Connection=0x7fffec0037c0) at /root/source/msquic/src/core/send_buffer.c:181
    5. #4  0x00007ffff792d64f in QuicStreamSendFlush (Stream=0x7fffe8008cf0) at /root/source/msquic/src/core/stream_send.c:590
    6. #5  0x00007ffff794329d in QuicConnProcessApiOperation (Connection=Connection@entry=0x7fffec0037c0, ApiCtx=0x7fffe4004be0) at /root/source/msquic/src/core/connection.c:6688
    7. #6  0x00007ffff79434a6 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:6825
    8. #7  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec0037c0, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    9. #8  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    10. #9  0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    11. #10 0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    12. #11 0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6
  6. QUIC_STREAM_EVENT_SHUTDOWN_COMPLETE
    1. #0  ServerStreamCallback (Stream=0x7fffe8008cf0, Context=<optimized out>, Event=<optimized out>) at /root/source/msquic/src/tools/sample/sample.c:278
    2. #1  0x00007ffff792b1f8 in QuicStreamIndicateEvent (Stream=Stream@entry=0x7fffe8008cf0, Event=Event@entry=0x7ffff3eecc40) at /root/source/msquic/src/core/stream.c:410
    3. #2  0x00007ffff792b2a3 in QuicStreamIndicateShutdownComplete (Stream=Stream@entry=0x7fffe8008cf0) at /root/source/msquic/src/core/stream.c:467
    4. #3  0x00007ffff792b756 in QuicStreamTryCompleteShutdown (Stream=Stream@entry=0x7fffe8008cf0) at /root/source/msquic/src/core/stream.c:543
    5. #4  0x00007ffff792d3f7 in QuicStreamSendShutdown (Stream=Stream@entry=0x7fffe8008cf0, Graceful=Graceful@entry=0 '\000', Silent=Silent@entry=0 '\000', DelaySend=DelaySend@entry=0 '\000', ErrorCode=ErrorCode@entry=0) at /root/source/msquic/src/core/stream_send.c:238
    6. #5  0x00007ffff792b38e in QuicStreamShutdown (Stream=0x7fffe8008cf0, Flags=Flags@entry=32774, ErrorCode=ErrorCode@entry=0) at /root/source/msquic/src/core/stream.c:492
    7. #6  0x00007ffff792ea3a in QuicStreamSetShutdown (StreamSet=StreamSet@entry=0x7fffec003e60) at /root/source/msquic/src/core/stream_set.c:163
    8. #7  0x00007ffff793a582 in QuicConnTryClose (Connection=0x7fffec0037c0, Flags=<optimized out>, ErrorCode=<optimized out>, RemoteReasonPhrase=<optimized out>, RemoteReasonPhrase@entry=0x0, RemoteReasonPhraseLength=<optimized out>) at /root/source/msquic/src/core/connection.c:1675
    9. #8  0x00007ffff793a765 in QuicConnCloseLocally (Connection=<optimized out>, Flags=Flags@entry=19, ErrorCode=ErrorCode@entry=62, ErrorMsg=<optimized out>, ErrorMsg@entry=0x0) at /root/source/msquic/src/core/connection.c:1715
    10. #9  0x00007ffff7941603 in QuicConnProcessIdleTimerOperation (Connection=<optimized out>) at /root/source/msquic/src/core/connection.c:5591
    11. #10 0x00007ffff79433cb in QuicConnProcessExpiredTimer (Connection=Connection@entry=0x7fffec0037c0, Type=<optimized out>) at /root/source/msquic/src/core/connection.c:6752
    12. #11 0x00007ffff7943603 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec0037c0) at /root/source/msquic/src/core/connection.c:6861
    13. #12 0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec0037c0, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    14. #13 0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    15. #14 0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    16. #15 0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    17. #16 0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6

    1.3.5 close

  7. QUIC_CONNECTION_EVENT_SHUTDOWN_INITIATED_BY_TRANSPORT
    1. #0  ServerConnectionCallback (Connection=0x7fffec000b60, Context=0x0, Event=0x7ffff3eecd20) at /root/source/msquic/src/tools/sample/sample.c:315
    2. #1  0x00007ffff7939aaf in QuicConnIndicateEvent (Connection=<optimized out>, Event=Event@entry=0x7ffff3eecd20) at /root/source/msquic/src/core/connection.c:692
    3. #2  0x00007ffff793a232 in QuicConnIndicateShutdownBegin (Connection=Connection@entry=0x7fffec000b60) at /root/source/msquic/src/core/connection.c:1367
    4. #3  0x00007ffff793a693 in QuicConnTryClose (Connection=0x7fffec000b60, Flags=19, ErrorCode=<optimized out>, RemoteReasonPhrase=<optimized out>, RemoteReasonPhrase@entry=0x0, RemoteReasonPhraseLength=<optimized out>) at /root/source/msquic/src/core/connection.c:1624
    5. #4  0x00007ffff793a765 in QuicConnCloseLocally (Connection=<optimized out>, Flags=Flags@entry=19, ErrorCode=ErrorCode@entry=62, ErrorMsg=<optimized out>, ErrorMsg@entry=0x0) at /root/source/msquic/src/core/connection.c:1715
    6. #5  0x00007ffff7941603 in QuicConnProcessIdleTimerOperation (Connection=<optimized out>) at /root/source/msquic/src/core/connection.c:5591
    7. #6  0x00007ffff79433cb in QuicConnProcessExpiredTimer (Connection=Connection@entry=0x7fffec000b60, Type=<optimized out>) at /root/source/msquic/src/core/connection.c:6752
    8. #7  0x00007ffff7943603 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec000b60) at /root/source/msquic/src/core/connection.c:6861
    9. #8  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec000b60, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    10. #9  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    11. #10 0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    12. #11 0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    13. #12 0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6
  8. QUIC_CONNECTION_EVENT_SHUTDOWN_COMPLETE
    1. #0  ServerConnectionCallback (Connection=0x7fffec000b60, Context=<optimized out>, Event=0x7ffff3eecda0) at /root/source/msquic/src/tools/sample/sample.c:332
    2. #1  0x00007ffff7939aaf in QuicConnIndicateEvent (Connection=Connection@entry=0x7fffec000b60, Event=Event@entry=0x7ffff3eecda0) at /root/source/msquic/src/core/connection.c:692
    3. #2  0x00007ffff793abd1 in QuicConnOnShutdownComplete (Connection=Connection@entry=0x7fffec000b60) at /root/source/msquic/src/core/connection.c:1414
    4. #3  0x00007ffff7943721 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffec000b60) at /root/source/msquic/src/core/connection.c:6907
    5. #4  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x607040, Connection=Connection@entry=0x7fffec000b60, ThreadID=ThreadID@entry=188913) at /root/source/msquic/src/core/worker.c:489
    6. #5  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x607040, ThreadID=ThreadID@entry=188913, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3eecec8) at /root/source/msquic/src/core/worker.c:644
    7. #6  0x00007ffff7931e00 in QuicWorkerThread (Context=0x607040) at /root/source/msquic/src/core/worker.c:727
    8. #7  0x00007ffff72c114a in start_thread () from /usr/lib64/libpthread.so.0
    9. #8  0x00007ffff6ff0dc3 in clone () from /usr/lib64/libc.so.6

    1.3.6 handshake

    1.3.6.1 worker分配connection

    ```c
  9. QuicBindingCreateConnection 中的会创建connection，并分配给worker
  10. QuicLibraryGetWorker 获取worker，worker_index是根据收到报文的Datagram->PartitionIndex确定的（epoll_wait）
  11. PartitionIndex=SocketContext->ProcContext->Index，是for循环的index（0、1、2、3）
  12. Worker->IdealProcessor和【3】中的index数值一样

0 QuicConnAlloc (Registration=0x68a010, Datagram=Datagram@entry=0x76d338) at /root/source/msquic/src/core/connection.c:76

1 0x00007ffff78e1a8d in QuicBindingCreateConnection (Binding=Binding@entry=0x741720, Datagram=Datagram@entry=0x76d338) at /root/source/msquic/src/core/binding.c:1321

2 0x00007ffff78e229b in QuicBindingDeliverDatagrams (Binding=Binding@entry=0x741720, DatagramChain=0x76d338, DatagramChainLength=DatagramChainLength@entry=1) at /root/source/msquic/src/core/binding.c:1557

3 0x00007ffff78e2755 in QuicBindingReceive (Socket=, RecvCallbackContext=0x741720, DatagramChain=0x0) at /root/source/msquic/src/core/binding.c:1712

4 0x00007ffff792665e in CxPlatSocketContextRecvComplete (SocketContext=SocketContext@entry=0x746840, MessagesReceived=1) at /root/source/msquic/src/platform/datapath_epoll.c:1599

5 0x00007ffff79288e6 in CxPlatSocketContextProcessEvents (EventPtr=, Events=1) at /root/source/msquic/src/platform/datapath_epoll.c:1820

6 0x00007ffff7928ad5 in CxPlatDataPathWorkerThread (Context=0x665360) at /root/source/msquic/src/platform/datapath_epoll.c:2788

7 0x00007ffff6ad314a in start_thread () from /usr/lib64/libpthread.so.0

8 0x00007ffff6802dc3 in clone () from /usr/lib64/libc.so.6

  1. <a name="hNcP1"></a>
  2. #### 1.3.6.2 握手中
  3. ```c
  4. #0  CxPlatTlsAddHandshakeDataCallback (Ssl=<optimized out>, Level=ssl_encryption_initial, Data=0x7fffd8008750 "\002", Length=123) at /root/source/msquic/src/platform/tls_openssl.c:510
  5. #1  0x00007ffff7976dbb in ssl3_do_write () from /root/source/msquic/build/bin/Debug/libmsquic.so
  6. #2  0x00007ffff796f2b6 in state_machine.part () from /root/source/msquic/build/bin/Debug/libmsquic.so
  7. #3  0x00007ffff795bce4 in SSL_do_handshake () from /root/source/msquic/build/bin/Debug/libmsquic.so
  8. #4  0x00007ffff792cb27 in CxPlatTlsProcessData (TlsContext=0x7fffd8000d10, DataType=DataType@entry=CXPLAT_TLS_CRYPTO_DATA, Buffer=0x7fffd4002b70 "\001", BufferLength=BufferLength@entry=0x7ffff0d2fb40, State=State@entry=0x7fffdc001390) at /root/source/msquic/src/platform/tls_openssl.c:1811
  9. #5  0x00007ffff78f8867 in QuicCryptoProcessData (Crypto=Crypto@entry=0x7fffdc001380, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1783
  10. #6  0x00007ffff78f8df1 in QuicCryptoInitializeTls (Crypto=Crypto@entry=0x7fffdc001380, SecConfig=<optimized out>, Params=Params@entry=0x7ffff0d2fc20) at /root/source/msquic/src/core/crypto.c:350
  11. #7  0x00007ffff78f3a81 in QuicConnSetConfiguration (Connection=0x7fffdc000b60, Configuration=0x668350) at /root/source/msquic/src/core/connection.c:2446
  12. #8  0x00007ffff78f5130 in QuicConnProcessApiOperation (Connection=Connection@entry=0x7fffdc000b60, ApiCtx=0x7fffd4003e60) at /root/source/msquic/src/core/connection.c:6649
  13. #9  0x00007ffff78f5566 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffdc000b60) at /root/source/msquic/src/core/connection.c:6825
  14. #10 0x00007ffff78d8e60 in QuicWorkerProcessConnection (Worker=Worker@entry=0x665ea0, Connection=Connection@entry=0x7fffdc000b60, ThreadID=ThreadID@entry=145063) at /root/source/msquic/src/core/worker.c:489
  15. #11 0x00007ffff78d9736 in QuicWorkerLoop (Worker=Worker@entry=0x665ea0, ThreadID=ThreadID@entry=145063, Reason=<optimized out>, Delay=Delay@entry=0x7ffff0d2fe48) at /root/source/msquic/src/core/worker.c:647
  16. #12 0x00007ffff78d98f6 in QuicWorkerThread (Context=0x665ea0) at /root/source/msquic/src/core/worker.c:730
  17. #13 0x00007ffff6ad314a in start_thread () from /usr/lib64/libpthread.so.0
  18. #14 0x00007ffff6802dc3 in clone () from /usr/lib64/libc.so.6

1.3.6.3 握手完成

  1. #0  QuicCryptoProcessTlsCompletion (Crypto=Crypto@entry=0x7fffdc001380) at /root/source/msquic/src/core/crypto.c:1306
  2. #1  0x00007ffff78f86df in QuicCryptoProcessDataComplete (Crypto=Crypto@entry=0x7fffdc001380, RecvBufferConsumed=<optimized out>) at /root/source/msquic/src/core/crypto.c:1635
  3. #2  0x00007ffff78f8879 in QuicCryptoProcessData (Crypto=Crypto@entry=0x7fffdc001380, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1791
  4. #3  0x00007ffff78f8df1 in QuicCryptoInitializeTls (Crypto=Crypto@entry=0x7fffdc001380, SecConfig=<optimized out>, Params=Params@entry=0x7ffff1530c20) at /root/source/msquic/src/core/crypto.c:350
  5. #4  0x00007ffff78f3a81 in QuicConnSetConfiguration (Connection=0x7fffdc000b60, Configuration=0x668350) at /root/source/msquic/src/core/connection.c:2446
  6. #5  0x00007ffff78f5130 in QuicConnProcessApiOperation (Connection=Connection@entry=0x7fffdc000b60, ApiCtx=0x7fffd4003e60) at /root/source/msquic/src/core/connection.c:6649
  7. #6  0x00007ffff78f5566 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffdc000b60) at /root/source/msquic/src/core/connection.c:6825
  8. #7  0x00007ffff78d8e60 in QuicWorkerProcessConnection (Worker=Worker@entry=0x665ea0, Connection=Connection@entry=0x7fffdc000b60, ThreadID=ThreadID@entry=145376) at /root/source/msquic/src/core/worker.c:489
  9. #8  0x00007ffff78d9736 in QuicWorkerLoop (Worker=Worker@entry=0x665ea0, ThreadID=ThreadID@entry=145376, Reason=<optimized out>, Delay=Delay@entry=0x7ffff1530e48) at /root/source/msquic/src/core/worker.c:647
  10. #9  0x00007ffff78d98f6 in QuicWorkerThread (Context=0x665ea0) at /root/source/msquic/src/core/worker.c:730
  11. #10 0x00007ffff6ad314a in start_thread () from /usr/lib64/libpthread.so.0
  12. #11 0x00007ffff6802dc3 in clone () from /usr/lib64/libc.so.6

1.3.6.4 生成src cid~1

  1. #0  QuicConnCidNewBackupSource (Data=Data@entry=0x7f62ec004bc8 "", Connection=Connection@entry=0x7f62e8000b60) at /root/source/msquic/src/core/connection.c:61
  2. #1  0x00007f631a72ad35 in QuicCidNewRandomSource (Prefix=0x0, PrefixLength=0 '\000', PartitionID=44491, ServerID=0x7f62e8000c54, Connection=0x7f62e8000b60) at /root/source/msquic/src/core/library.h:455
  3. #2  QuicConnGenerateNewSourceCid (Connection=Connection@entry=0x7f62e8000b60, IsInitial=IsInitial@entry=1 '\001') at /root/source/msquic/src/core/connection.c:852
  4. #3  0x00007f631a708fb4 in QuicListenerAcceptConnection (Listener=Listener@entry=0x1c91250, Connection=Connection@entry=0x7f62e8000b60, Info=Info@entry=0x7f63077fd930) at /root/source/msquic/src/core/listener.c:579
  5. #4  0x00007f631a7230a6 in QuicBindingAcceptConnection (Binding=<optimized out>, Connection=Connection@entry=0x7f62e8000b60, Info=Info@entry=0x7f63077fd930) at /root/source/msquic/src/core/binding.c:556
  6. #5  0x00007f631a73d3a3 in QuicCryptoProcessData (Crypto=Crypto@entry=0x7f62e8001380, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1764
  7. #6  0x00007f631a73d82a in QuicCryptoProcessFrame (Crypto=Crypto@entry=0x7f62e8001380, KeyType=<optimized out>, Frame=Frame@entry=0x7f63077fda70) at /root/source/msquic/src/core/crypto.c:1255
  8. #7  0x00007f631a731615 in QuicConnRecvFrames (Connection=Connection@entry=0x7f62e8000b60, Path=<optimized out>, Packet=Packet@entry=0x1cd1a30, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4225
  9. #8  0x00007f631a7341ba in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x7f62e8000b60, Path=<optimized out>, Path@entry=0x7f62e8000c88, BatchCount=<optimized out>, Datagrams=Datagrams@entry=0x7f63077fdc90, Cipher=Cipher@entry=0x7f63077fdc10 "A\004\341\350\303ɪ\n$-P\205\241i\220\371\330\023", RecvState=RecvState@entry=0x7f63077fdcd4) at /root/source/msquic/src/core/connection.c:5093
  10. #9  0x00007f631a734b6f in QuicConnRecvDatagrams (Connection=Connection@entry=0x7f62e8000b60, DatagramChain=<optimized out>, DatagramChain@entry=0x1cd1418, DatagramChainCount=DatagramChainCount@entry=1, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5298
  11. #10 0x00007f631a73508f in QuicConnFlushRecv (Connection=Connection@entry=0x7f62e8000b60) at /root/source/msquic/src/core/connection.c:5442
  12. #11 0x00007f631a737cc4 in QuicConnDrainOperations (Connection=Connection@entry=0x7f62e8000b60) at /root/source/msquic/src/core/connection.c:6875
  13. #12 0x00007f631a71b3f6 in QuicWorkerProcessConnection (Worker=Worker@entry=0x1c940e0, Connection=Connection@entry=0x7f62e8000b60, ThreadID=ThreadID@entry=910958) at /root/source/msquic/src/core/worker.c:504
  14. #13 0x00007f631a71bcd5 in QuicWorkerLoop (Worker=Worker@entry=0x1c940e0, ThreadID=ThreadID@entry=910958, Reason=<optimized out>, Delay=Delay@entry=0x7f63077fde48) at /root/source/msquic/src/core/worker.c:676
  15. #14 0x00007f631a71be95 in QuicWorkerThread (Context=0x1c940e0) at /root/source/msquic/src/core/worker.c:759
  16. #15 0x00007f631991514a in start_thread () from /usr/lib64/libpthread.so.0
  17. #16 0x00007f6319644dc3 in clone () from /usr/lib64/libc.so.6

1.3.6.5 生成src cid~2

  1. #0  QuicBindingAddSourceConnectionID (Binding=0x7416f0, SourceCid=SourceCid@entry=0x7fffdc001d20) at /root/source/msquic/src/core/binding.c:567
  2. #1  0x00007ffff78e84e6 in QuicConnGenerateNewSourceCid (Connection=Connection@entry=0x7fffdc000b60, IsInitial=IsInitial@entry=0 '\000') at /root/source/msquic/src/core/connection.c:845
  3. #2  0x00007ffff78e8970 in QuicConnGenerateNewSourceCids (Connection=Connection@entry=0x7fffdc000b60, ReplaceExistingCids=ReplaceExistingCids@entry=0 '\000') at /root/source/msquic/src/core/connection.c:951
  4. #3  0x00007ffff78f7c09 in QuicCryptoProcessTlsCompletion (Crypto=Crypto@entry=0x7fffdc001380, cid=cid@entry=0x0, cid_len=cid_len@entry=0) at /root/source/msquic/src/core/crypto.c:1571
  5. #4  0x00007ffff78f86ef in QuicCryptoProcessDataComplete (Crypto=Crypto@entry=0x7fffdc001380, RecvBufferConsumed=<optimized out>) at /root/source/msquic/src/core/crypto.c:1647
  6. #5  0x00007ffff78fa9fa in QuicCryptoProcessData (Crypto=Crypto@entry=0x7fffdc001380, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1806
  7. #6  0x00007ffff78fb019 in QuicCryptoProcessFrame (Crypto=Crypto@entry=0x7fffdc001380, KeyType=<optimized out>, Frame=Frame@entry=0x7fffeaffca70) at /root/source/msquic/src/core/crypto.c:1253
  8. #7  0x00007ffff78eef2e in QuicConnRecvFrames (Connection=Connection@entry=0x7fffdc000b60, Path=<optimized out>, Packet=Packet@entry=0x7fffd80011c0, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4203
  9. #8  0x00007ffff78f1ad3 in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x7fffdc000b60, Path=<optimized out>, Path@entry=0x7fffdc000c88, BatchCount=<optimized out>, Datagrams=Datagrams@entry=0x7fffeaffcc90, Cipher=Cipher@entry=0x7fffeaffcc10 "\233\331\004\341\017\243\226\342a\317\034\200c\337\033\063\001", RecvState=RecvState@entry=0x7fffeaffccd4) at /root/source/msquic/src/core/connection.c:5071
  10. #9  0x00007ffff78f2488 in QuicConnRecvDatagrams (Connection=Connection@entry=0x7fffdc000b60, DatagramChain=<optimized out>, DatagramChain@entry=0x7fffd8000ba8, DatagramChainCount=DatagramChainCount@entry=3, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5276
  11. #10 0x00007ffff78f29a8 in QuicConnFlushRecv (Connection=Connection@entry=0x7fffdc000b60) at /root/source/msquic/src/core/connection.c:5420
  12. #11 0x00007ffff78f5578 in QuicConnDrainOperations (Connection=Connection@entry=0x7fffdc000b60) at /root/source/msquic/src/core/connection.c:6831
  13. #12 0x00007ffff78d8e60 in QuicWorkerProcessConnection (Worker=Worker@entry=0x667280, Connection=Connection@entry=0x7fffdc000b60, ThreadID=ThreadID@entry=159523) at /root/source/msquic/src/core/worker.c:489
  14. #13 0x00007ffff78d9736 in QuicWorkerLoop (Worker=Worker@entry=0x667280, ThreadID=ThreadID@entry=159523, Reason=<optimized out>, Delay=Delay@entry=0x7fffeaffce48) at /root/source/msquic/src/core/worker.c:647
  15. #14 0x00007ffff78d98f6 in QuicWorkerThread (Context=0x667280) at /root/source/msquic/src/core/worker.c:730
  16. #15 0x00007ffff6ad314a in start_thread () from /usr/lib64/libpthread.so.0
  17. #16 0x00007ffff6802dc3 in clone () from /usr/lib64/libc.so.6

1.3.6.6 密钥生成

  1. #0  CxPlatTlsSetEncryptionSecretsCallback (Ssl=<optimized out>, Level=ssl_encryption_handshake, ReadSecret=0x7f0eb400558c "\266\252\216a\361\373\256\303*\264\332\006k&\023\350\257۪\t", WriteSecret=0x7f0eb40055cc "|\351\302EuWI:\251", SecretLen=48) at /root/source/msquic/src/platform/tls_openssl.c:359
  2. #1  0x00007f0ee9d9a627 in quic_set_encryption_secrets () from /root/source/msquic/build/bin/Debug/libmsquic.so
  3. #2  0x00007f0ee9dc115a in tls13_change_cipher_state () from /root/source/msquic/build/bin/Debug/libmsquic.so
  4. #3  0x00007f0ee9db7b75 in ossl_statem_server_post_work () from /root/source/msquic/build/bin/Debug/libmsquic.so
  5. #4  0x00007f0ee9da917a in state_machine.part () from /root/source/msquic/build/bin/Debug/libmsquic.so
  6. #5  0x00007f0ee9d961a4 in SSL_do_handshake () from /root/source/msquic/build/bin/Debug/libmsquic.so
  7. #6  0x00007f0ee9d70305 in CxPlatTlsProcessData (TlsContext=0x7f0eb4005180, DataType=DataType@entry=CXPLAT_TLS_CRYPTO_DATA, Buffer=0x7f0eb4002b70 "\001", BufferLength=BufferLength@entry=0x7f0ed6ffcb20, State=State@entry=0x7f0ec00013a8) at /root/source/msquic/src/platform/tls_openssl.c:1800
  8. #7  0x00007f0ee9d3ac86 in QuicCryptoProcessData (Crypto=Crypto@entry=0x7f0ec0001398, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1789
  9. #8  0x00007f0ee9d3b210 in QuicCryptoInitializeTls (Crypto=Crypto@entry=0x7f0ec0001398, SecConfig=<optimized out>, Params=Params@entry=0x7f0ed6ffcc00) at /root/source/msquic/src/core/crypto.c:356
  10. #9  0x00007f0ee9d35952 in QuicConnSetConfiguration (Connection=0x7f0ec0000b60, Configuration=0x12bc740) at /root/source/msquic/src/core/connection.c:2501
  11. #10 0x00007f0ee9d37015 in QuicConnProcessApiOperation (Connection=Connection@entry=0x7f0ec0000b60, ApiCtx=0x7f0eb4003e60) at /root/source/msquic/src/core/connection.c:6735
  12. #11 0x00007f0ee9d37835 in QuicConnDrainOperations (Connection=Connection@entry=0x7f0ec0000b60) at /root/source/msquic/src/core/connection.c:7037
  13. #12 0x00007f0ee9d1a5c5 in QuicWorkerProcessConnection (Worker=Worker@entry=0x139fe60, Connection=Connection@entry=0x7f0ec0000b60, ThreadID=ThreadID@entry=10048) at /root/source/msquic/src/core/worker.c:499
  14. #13 0x00007f0ee9d1af91 in QuicWorkerLoop (Worker=Worker@entry=0x139fe60, ThreadID=ThreadID@entry=10048, Reason=<optimized out>, Delay=Delay@entry=0x7f0ed6ffce38, CurBackupConn=CurBackupConn@entry=0x139ff40, BackupConnCnt=BackupConnCnt@entry=0x7f0ed6ffce34) at /root/source/msquic/src/core/worker.c:698
  15. #14 0x00007f0ee9d1b1db in QuicWorkerThread (Context=0x139fe60) at /root/source/msquic/src/core/worker.c:791
  16. #15 0x00007f0ee8d0114a in start_thread () from /usr/lib64/libpthread.so.0
  17. #16 0x00007f0ee8a30dc3 in clone () from /usr/lib64/libc.so.6

1.3.7 socket

1.3.7.1 create

  1. #0  CxPlatSocketContextStartReceive (SocketContext=SocketContext@entry=0x7421a0) at /root/source/msquic/src/platform/datapath_epoll.c:1463
  2. #1  0x00007ffff7927601 in CxPlatSocketCreateUdp (Datapath=0x665000, Config=Config@entry=0x7fffffffdea0, NewBinding=NewBinding@entry=0x741740) at /root/source/msquic/src/platform/datapath_epoll.c:1943
  3. #2  0x00007ffff78e004e in QuicBindingInitialize (UdpConfig=UdpConfig@entry=0x7fffffffdea0, NewBinding=NewBinding@entry=0x73e2a0) at /root/source/msquic/src/core/binding.c:134
  4. #3  0x00007ffff78c3279 in QuicLibraryGetBinding (UdpConfig=UdpConfig@entry=0x7fffffffdea0, NewBinding=NewBinding@entry=0x73e2a0) at /root/source/msquic/src/core/library.c:1572
  5. #4  0x00007ffff78c622e in MsQuicListenerStart (Handle=0x73e1e0, AlpnBuffers=<optimized out>, AlpnBufferCount=1, LocalAddress=0x7fffffffdf20) at /root/source/msquic/src/core/listener.c:274
  6. #5  0x0000000000403423 in RunServer (argc=argc@entry=4, argv=argv@entry=0x7fffffffe058) at /root/source/msquic/src/tools/sample/sample.c:605
  7. #6  0x0000000000403e8e in main (argc=4, argv=0x7fffffffe058) at /root/source/msquic/src/tools/sample/sample.c:1069

1.3.8 worker

1.3.8.1 wait

  1. #0  0x00007efe2351e2fc in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
  2. #1  0x00007efe2431ef0b in CxPlatInternalEventWaitForever (Event=0x18cdea0) at /root/source/msquic/src/inc/quic_platform_posix.h:797
  3. #2  QuicWorkerThread (Context=0x18cdea0) at /root/source/msquic/src/core/worker.c:736
  4. #3  0x00007efe2351814a in start_thread () from /usr/lib64/libpthread.so.0
  5. #4  0x00007efe23247dc3 in clone () from /usr/lib64/libc.so.6

1.3.9 keepalive

1.3.9.1 执行1

  1. #0  QuicConnResetIdleTimeout (Connection=Connection@entry=0x7f897c000b60) at /root/source/msquic/src/core/connection.c:5621
  2. #1  0x00007f89a52d3089 in QuicConnRecvDatagrams (Connection=Connection@entry=0x7f897c000b60, DatagramChain=0x0, DatagramChain@entry=0x202b058, DatagramChainCount=DatagramChainCount@entry=1, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5393
  3. #2  0x00007f89a52d33be in QuicConnFlushRecv (Connection=Connection@entry=0x7f897c000b60) at /root/source/msquic/src/core/connection.c:5466
  4. #3  0x00007f89a52d633a in QuicConnDrainOperations (Connection=Connection@entry=0x7f897c000b60) at /root/source/msquic/src/core/connection.c:6986
  5. #4  0x00007f89a52b959d in QuicWorkerProcessConnection (Worker=Worker@entry=0x2021330, Connection=Connection@entry=0x7f897c000b60, ThreadID=ThreadID@entry=951846) at /root/source/msquic/src/core/worker.c:504
  6. #5  0x00007f89a52b9f2b in QuicWorkerLoop (Worker=Worker@entry=0x2021330, ThreadID=ThreadID@entry=951846, Reason=<optimized out>, Delay=Delay@entry=0x7f8991ffae48) at /root/source/msquic/src/core/worker.c:693
  7. #6  0x00007f89a52ba12d in QuicWorkerThread (Context=0x2021330) at /root/source/msquic/src/core/worker.c:782
  8. #7  0x00007f89a44b314a in start_thread () from /usr/lib64/libpthread.so.0
  9. #8  0x00007f89a41e2dc3 in clone () from /usr/lib64/libc.so.6

1.3.9.2 执行2

  1. #0  QuicConnProcessKeepAliveOperation (Connection=0x7f4f24000b60) at /root/source/msquic/src/core/connection.c:5659
  2. #1  0x00007f4f5410ad45 in QuicConnProcessExpiredTimer (Connection=Connection@entry=0x7f4f24000b60, Type=<optimized out>) at /root/source/msquic/src/core/connection.c:6804
  3. #2  0x00007f4f5410b4ef in QuicConnDrainOperations (Connection=Connection@entry=0x7f4f24000b60) at /root/source/msquic/src/core/connection.c:7017
  4. #3  0x00007f4f540ee59d in QuicWorkerProcessConnection (Worker=Worker@entry=0x9a5e60, Connection=Connection@entry=0x7f4f24000b60, ThreadID=ThreadID@entry=951505) at /root/source/msquic/src/core/worker.c:504
  5. #4  0x00007f4f540eef2b in QuicWorkerLoop (Worker=Worker@entry=0x9a5e60, ThreadID=ThreadID@entry=951505, Reason=<optimized out>, Delay=Delay@entry=0x7f4f46ffce48) at /root/source/msquic/src/core/worker.c:693
  6. #5  0x00007f4f540ef12d in QuicWorkerThread (Context=0x9a5e60) at /root/source/msquic/src/core/worker.c:782
  7. #6  0x00007f4f532e814a in start_thread () from /usr/lib64/libpthread.so.0
  8. #7  0x00007f4f53017dc3 in clone () from /usr/lib64/libc.so.6

2. client（quicsample）

2.1 调用堆栈

2.1.1 connection_cb

2.1.1.1 connected

  1. #0  ClientConnectionCallback (Connection=0x705270, Context=0x0, Event=0x7ffff2fee930) at /root/source/msquic/src/tools/sample/sample.c:767
  2. #1  0x00007ffff7939aaf in QuicConnIndicateEvent (Connection=Connection@entry=0x705270, Event=Event@entry=0x7ffff2fee930) at /root/source/msquic/src/core/connection.c:692
  3. #2  0x00007ffff7944f04 in QuicCryptoProcessTlsCompletion (Crypto=Crypto@entry=0x705a90) at /root/source/msquic/src/core/crypto.c:1582
  4. #3  0x00007ffff79455a3 in QuicCryptoProcessDataComplete (Crypto=Crypto@entry=0x705a90, RecvBufferConsumed=<optimized out>) at /root/source/msquic/src/core/crypto.c:1630
  5. #4  0x00007ffff7945652 in QuicCryptoProcessData (Crypto=Crypto@entry=0x705a90, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1786
  6. #5  0x00007ffff7945b22 in QuicCryptoProcessFrame (Crypto=Crypto@entry=0x705a90, KeyType=<optimized out>, Frame=Frame@entry=0x7ffff2feeaf0) at /root/source/msquic/src/core/crypto.c:1250
  7. #6  0x00007ffff793f445 in QuicConnRecvFrames (Connection=Connection@entry=0x705270, Path=<optimized out>, Packet=Packet@entry=0x7fffe4001180, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4203
  8. #7  0x00007ffff79409d5 in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x705270, Path=<optimized out>, Path@entry=0x705398, BatchCount=<optimized out>, Datagrams=Datagrams@entry=0x7ffff2feed10, Cipher=Cipher@entry=0x7ffff2feec90 "\273\213w\300h\253\303 Zp!\037\225\360W\255\001", RecvState=RecvState@entry=0x7ffff2feed54) at /root/source/msquic/src/core/connection.c:5071
  9. #8  0x00007ffff79410e2 in QuicConnRecvDatagrams (Connection=Connection@entry=0x705270, DatagramChain=<optimized out>, DatagramChain@entry=0x7fffe4000b68, DatagramChainCount=DatagramChainCount@entry=1, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5276
  10. #9  0x00007ffff794151b in QuicConnFlushRecv (Connection=Connection@entry=0x705270) at /root/source/msquic/src/core/connection.c:5420
  11. #10 0x00007ffff79434cd in QuicConnDrainOperations (Connection=Connection@entry=0x705270) at /root/source/msquic/src/core/connection.c:6831
  12. #11 0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x6216e0, Connection=Connection@entry=0x705270, ThreadID=ThreadID@entry=589663) at /root/source/msquic/src/core/worker.c:489
  13. #12 0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x6216e0, ThreadID=ThreadID@entry=589663, Reason=<optimized out>, Delay=Delay@entry=0x7ffff2feeec8) at /root/source/msquic/src/core/worker.c:644
  14. #13 0x00007ffff7931e00 in QuicWorkerThread (Context=0x6216e0) at /root/source/msquic/src/core/worker.c:727
  15. #14 0x00007ffff6ddb14a in start_thread () from /lib64/libpthread.so.0
  16. #15 0x00007ffff6b0adc3 in clone () from /lib64/libc.so.6

2.1.1.2 resumption ticket received

  1. #0  ClientConnectionCallback (Connection=0x705270, Context=0x0, Event=0x7ffff2fee560) at /root/source/msquic/src/tools/sample/sample.c:803
  2. #1  0x00007ffff7939aaf in QuicConnIndicateEvent (Connection=Connection@entry=0x705270, Event=Event@entry=0x7ffff2fee560) at /root/source/msquic/src/core/connection.c:692
  3. #2  0x00007ffff793c262 in QuicConnRecvResumptionTicket (Connection=0x705270, TicketLength=<optimized out>, Ticket=<optimized out>) at /root/source/msquic/src/core/connection.c:2146
  4. #3  0x00007ffff79649f7 in CxPlatTlsOnClientSessionTicketReceived (Ssl=<optimized out>, Session=0x7fffec036650) at /root/source/msquic/src/platform/tls_openssl.c:630
  5. #4  0x00007ffff797d0fa in ssl_update_cache () from /root/source/msquic/build/bin/Debug/libmsquic.so
  6. #5  0x00007ffff7994962 in tls_process_new_session_ticket () from /root/source/msquic/build/bin/Debug/libmsquic.so
  7. #6  0x00007ffff7995d05 in ossl_statem_client_process_message () from /root/source/msquic/build/bin/Debug/libmsquic.so
  8. #7  0x00007ffff7990634 in state_machine.part () from /root/source/msquic/build/bin/Debug/libmsquic.so
  9. #8  0x00007ffff79818b3 in SSL_process_quic_post_handshake () from /root/source/msquic/build/bin/Debug/libmsquic.so
  10. #9  0x00007ffff79662ec in CxPlatTlsProcessData (TlsContext=0x7fffec000c00, DataType=DataType@entry=CXPLAT_TLS_CRYPTO_DATA, Buffer=0x7fffec015ca0 "\004", BufferLength=BufferLength@entry=0x7ffff2feea30, State=State@entry=0x705aa0) at /root/source/msquic/src/platform/tls_openssl.c:1877
  11. #10 0x00007ffff7945640 in QuicCryptoProcessData (Crypto=Crypto@entry=0x705a90, IsClientInitial=IsClientInitial@entry=0 '\000') at /root/source/msquic/src/core/crypto.c:1778
  12. #11 0x00007ffff7945b22 in QuicCryptoProcessFrame (Crypto=Crypto@entry=0x705a90, KeyType=<optimized out>, Frame=Frame@entry=0x7ffff2feeaf0) at /root/source/msquic/src/core/crypto.c:1250
  13. #12 0x00007ffff793f445 in QuicConnRecvFrames (Connection=Connection@entry=0x705270, Path=<optimized out>, Packet=Packet@entry=0x7fffe4001180, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4203
  14. #13 0x00007ffff79409d5 in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x705270, Path=<optimized out>, Path@entry=0x705398, BatchCount=BatchCount@entry=1 '\001', Datagrams=Datagrams@entry=0x7ffff2feed10, Cipher=Cipher@entry=0x7ffff2feec90 "\350\\׃i\213\036\307\324\361\236G\270\016\067\242\001", RecvState=RecvState@entry=0x7ffff2feed54) at /root/source/msquic/src/core/connection.c:5071
  15. #14 0x00007ffff79412c0 in QuicConnRecvDatagrams (Connection=Connection@entry=0x705270, DatagramChain=0x0, DatagramChain@entry=0x7fffe4000b68, DatagramChainCount=DatagramChainCount@entry=1, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5336
  16. #15 0x00007ffff794151b in QuicConnFlushRecv (Connection=Connection@entry=0x705270) at /root/source/msquic/src/core/connection.c:5420
  17. #16 0x00007ffff79434cd in QuicConnDrainOperations (Connection=Connection@entry=0x705270) at /root/source/msquic/src/core/connection.c:6831
  18. #17 0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x6216e0, Connection=Connection@entry=0x705270, ThreadID=ThreadID@entry=589663) at /root/source/msquic/src/core/worker.c:489
  19. #18 0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x6216e0, ThreadID=ThreadID@entry=589663, Reason=<optimized out>, Delay=Delay@entry=0x7ffff2feeec8) at /root/source/msquic/src/core/worker.c:644
  20. #19 0x00007ffff7931e00 in QuicWorkerThread (Context=0x6216e0) at /root/source/msquic/src/core/worker.c:727
  21. #20 0x00007ffff6ddb14a in start_thread () from /lib64/libpthread.so.0
  22. #21 0x00007ffff6b0adc3 in clone () from /lib64/libc.so.6

2.1.1.3 connection register worker

  1. PartitionId = (高位)分区ID随机数值 | (低位)线程号
  3. #0  QuicConnAlloc (Registration=Registration@entry=0x620190, Datagram=Datagram@entry=0x0) at /root/source/msquic/src/core/connection.c:85
  4. #1  0x00007ffff792f7dc in MsQuicConnectionOpen (RegistrationHandle=0x620190, Handler=0x4045fb <ClientConnectionCallback>, Context=0x61d5d0 <client_main+80>, NewConnection=0x61d5f8 <client_main+120>) at /root/source/msquic/src/core/api.c:64
  5. #2  0x00000000004049d8 in RunClient (argc=argc@entry=4, argv=argv@entry=0x7fffffffdfd8) at /root/source/msquic/src/tools/sample/sample.c:1198
  6. #3  0x0000000000404d25 in main (argc=4, argv=0x7fffffffdfd8) at /root/source/msquic/src/tools/sample/sample.c:1311
  9. PartitionId低位取余(cpu数量) = 线程号，作为worker索引
  10. TODO（需要做负载均衡，平衡每个worker的工作量）
  12. #0  QuicRegistrationQueueNewConnection (Registration=Registration@entry=0x620190, Connection=Connection@entry=0x7054c0) at /root/source/msquic/src/core/registration.c:373
  13. #1  0x00007ffff792f7fa in MsQuicConnectionOpen (RegistrationHandle=0x620190, Handler=0x4045fb <ClientConnectionCallback>, Context=0x61d5d0 <client_main+80>, NewConnection=0x61d5f8 <client_main+120>) at /root/source/msquic/src/core/api.c:72
  14. #2  0x00000000004049d8 in RunClient (argc=argc@entry=4, argv=argv@entry=0x7fffffffdfd8) at /root/source/msquic/src/tools/sample/sample.c:1198
  15. #3  0x0000000000404d25 in main (argc=4, argv=0x7fffffffdfd8) at /root/source/msquic/src/tools/sample/sample.c:1311

2.1.1.4 random worker update

  1. 【connection.c::QuicConnRecvDatagramBatch ---> 修改PartitionIndex，标记位true】
  2.     RecvState->PartitionIndex = Datagrams[i]->PartitionIndex % MsQuicLib.PartitionCount;
  3.     // RecvState->UpdatePartitionId = TRUE;
  5. 【connection.c::QuicConnRecvDatagrams ---> 修改Connection->PartitionID】
  6.     Connection->PartitionID = QuicPartitionIdCreate(RecvState.PartitionIndex);

2.1.2 stream_cb

2.1.2.1 stream sent

  1. #0  ClientStreamCallback (Stream=0x6456e0, Context=<optimized out>, Event=<optimized out>) at /root/source/msquic/src/tools/sample/sample.c:650
  2. #1  0x00007ffff792b1f8 in QuicStreamIndicateEvent (Stream=Stream@entry=0x6456e0, Event=Event@entry=0x7ffff2feece0) at /root/source/msquic/src/core/stream.c:410
  3. #2  0x00007ffff792cafa in QuicStreamSendBufferRequest (Stream=Stream@entry=0x6456e0, Req=Req@entry=0x6edf20) at /root/source/msquic/src/core/stream_send.c:475
  4. #3  0x00007ffff7958591 in QuicSendBufferFill (Connection=0x705270) at /root/source/msquic/src/core/send_buffer.c:181
  5. #4  0x00007ffff792d64f in QuicStreamSendFlush (Stream=0x6456e0) at /root/source/msquic/src/core/stream_send.c:590
  6. #5  0x00007ffff794329d in QuicConnProcessApiOperation (Connection=Connection@entry=0x705270, ApiCtx=0x702630) at /root/source/msquic/src/core/connection.c:6688
  7. #6  0x00007ffff79434a6 in QuicConnDrainOperations (Connection=Connection@entry=0x705270) at /root/source/msquic/src/core/connection.c:6825
  8. #7  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x6216e0, Connection=Connection@entry=0x705270, ThreadID=ThreadID@entry=589663) at /root/source/msquic/src/core/worker.c:489
  9. #8  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x6216e0, ThreadID=ThreadID@entry=589663, Reason=<optimized out>, Delay=Delay@entry=0x7ffff2feeec8) at /root/source/msquic/src/core/worker.c:644
  10. #9  0x00007ffff7931e00 in QuicWorkerThread (Context=0x6216e0) at /root/source/msquic/src/core/worker.c:727
  11. #10 0x00007ffff6ddb14a in start_thread () from /lib64/libpthread.so.0
  12. #11 0x00007ffff6b0adc3 in clone () from /lib64/libc.so.6

2.1.2.2 stream recv

  1. #2 QuicStreamRecvFlush 步骤：
  2. 1. QuicRecvBufferRead
  3.     1.1 RecvBuffer->WrittenRanges是收到报文后的range，RecvBuffer->BaseOffset是未ack的报文偏移量
  4.   1.2 从未ack的报文开始发送，填满RecvBuffer->AllocBufferLength
  5. 2. 如果之前没有这个range，即read的是无效包，则FIN
  6. 3. sample中的回调write函数
  7. 4. QuicStreamReceiveComplete
  8.     4.1 QuicRecvBufferDrain实现：
  9.           增加未ack部分，调整recv buffer的偏移量，返回值：第一段的缝隙是否读完(true)，否则(false)接着读
  11. #0  ClientStreamCallback (Stream=0x7fdc0edf7010, Context=0x61d588 <client_main+8>, Event=0x7fdc09eebd70) at /root/source/msquic/src/tools/sample/sample.c:677
  12. #1  0x00007fdc0e8281d7 in QuicStreamIndicateEvent (Stream=Stream@entry=0x7fdc0edf7010, Event=Event@entry=0x7fdc09eebd70) at /root/source/msquic/src/core/stream.c:410
  13. #2  0x00007fdc0e82948a in QuicStreamRecvFlush (Stream=0x7fdc0edf7010) at /root/source/msquic/src/core/stream_recv.c:776
  14. #3  0x00007fdc0e8405fe in QuicConnDrainOperations (Connection=Connection@entry=0x16af4c0) at /root/source/msquic/src/core/connection.c:6841
  15. #4  0x00007fdc0e82e5c0 in QuicWorkerProcessConnection (Worker=Worker@entry=0x15cb6e0, Connection=Connection@entry=0x16af4c0, ThreadID=ThreadID@entry=1985749) at /root/source/msquic/src/core/worker.c:490
  16. #5  0x00007fdc0e82ec4b in QuicWorkerLoop (Worker=Worker@entryQuicStreamWriteStreamFrames=0x15cb6e0, ThreadID=ThreadID@entry=1985749, Reason=<optimized out>, Delay=Delay@entry=0x7fdc09eebec8) at /root/source/msquic/src/core/worker.c:645
  17. #6  0x00007fdc0e82ee4a in QuicWorkerThread (Context=0x15cb6e0) at /root/source/msquic/src/core/worker.c:732
  18. #7  0x00007fdc0dcd814a in start_thread () from /lib64/libpthread.so.0
  19. #8  0x00007fdc0da07dc3 in clone () from /lib64/libc.so.6

2.1.2.3 stream接收回调

主要函数：QuicStreamRecvFlush
bcaktrace - 图1

变量 case 1 case 2 case 3 case 4
有事件 Yes No Yes No
WrittenRanges 【0，1】 - 【0，2】【4】 【0，4】
BaseOffset 0 - 1 2
WrittenRangeLength 【1】 - 【2】 【3，4】
BaseOffset更新？ 更新到1 - 更新到2 更新到4
BufferStart更新？ 已同步，清空 - 增加【2】长度 已同步，清空
MorePendingData - - Yes，下一次Range找不到，DataAvailable=0，RecvPendingLength=0导致退出 -

2.1.3 send

2.1.3.1 api send

  1. #0  QuicStreamCopyFromSendRequests (Stream=Stream@entry=0x7fffec027770, Offset=0, Buf=0x7fffec02c4e1 "\251\355W\252P\371w\236\016\242\070$\002\\y\367\350\234\360h?W\263\177\003bGE\241\367\067\063\fѰ\330\036\f\377f\025\236\354\a\223.h\263G\320%9\030\261\266\204_t#\364l\301w\204\v\207`M\223\365\271\244.Ui\vD\304\373%\017\201\341\361\323jVؽ\274\355P\245,\f}\352\245\347\223\rb\361N#\265\344\217:\002Mf\366\240\205\251\034\307F\037H8\301&*\023\256\236\234:\335:\006;\233e\370\201\350\260ߓ\331\027\017\212WM\243\t\034\310lbXަ\252ӲZ\032m\356\001\005\313\307\330\032J:H\355d\364\021\253*?Z\253\247\345\371\364P\235\372\224\005/\022Lm=\202\374\330}"..., Len=60) at /root/source/msquic/src/core/stream_send.c:632
  2. #1  0x00007ffff792cf6e in QuicStreamWriteOneFrame (Stream=Stream@entry=0x7fffec027770, ExplicitDataLength=ExplicitDataLength@entry=1 '\001', Offset=Offset@entry=0, FramePayloadBytes=FramePayloadBytes@entry=0x7ffff3ff0a0e, FrameBytes=FrameBytes@entry=0x7ffff3ff0a0c, Buffer=Buffer@entry=0x7fffec02c4dd "\377\303}\001\251\355W\252P\371w\236\016\242\070$\002\\y\367\350\234\360h?W\263\177\003bGE\241\367\067\063\fѰ\330\036\f\377f\025\236\354\a\223.h\263G\320%9\030\261\266\204_t#\364l\301w\204\v\207`M\223\365\271\244.Ui\vD\304\373%\017\201\341\361\323jVؽ\274\355P\245,\f}\352\245\347\223\rb\361N#\265\344\217:\002Mf\366\240\205\251\034\307F\037H8\301&*\023\256\236\234:\335:\006;\233e\370\201\350\260ߓ\331\027\017\212WM\243\t\034\310lbXަ\252ӲZ\032m\356\001\005\313\307\330\032J:H\355d\364\021\253*?Z\253\247\345\371\364P\235\372\224\005/\022Lm="..., PacketMetadata=0x7ffff3ff0c48) at /root/source/msquic/src/core/stream_send.c:747
  3. #2  0x00007ffff792d7ce in QuicStreamWriteStreamFrames (Stream=Stream@entry=0x7fffec027770, ExplicitDataLength=1 '\001', PacketMetadata=0x7ffff3ff0c48, BufferLength=BufferLength@entry=0x7ffff3ff0a50, Buffer=0x7fffec02c4dd "\377\303}\001\251\355W\252P\371w\236\016\242\070$\002\\y\367\350\234\360h?W\263\177\003bGE\241\367\067\063\fѰ\330\036\f\377f\025\236\354\a\223.h\263G\320%9\030\261\266\204_t#\364l\301w\204\v\207`M\223\365\271\244.Ui\vD\304\373%\017\201\341\361\323jVؽ\274\355P\245,\f}\352\245\347\223\rb\361N#\265\344\217:\002Mf\366\240\205\251\034\307F\037H8\301&*\023\256\236\234:\335:\006;\233e\370\201\350\260ߓ\331\027\017\212WM\243\t\034\310lbXަ\252ӲZ\032m\356\001\005\313\307\330\032J:H\355d\364\021\253*?Z\253\247\345\371\364P\235\372\224\005/\022Lm="...) at /root/source/msquic/src/core/stream_send.c:916
  4. #3  0x00007ffff792e140 in QuicStreamSendWrite (Stream=Stream@entry=0x7fffec027770, Builder=Builder@entry=0x7ffff3ff0ab0) at /root/source/msquic/src/core/stream_send.c:1091
  5. #4  0x00007ffff795805e in QuicSendFlush (Send=Send@entry=0x705ce0) at /root/source/msquic/src/core/send.c:1244
  6. #5  0x00007ffff79435c5 in QuicConnDrainOperations (Connection=Connection@entry=0x705270) at /root/source/msquic/src/core/connection.c:6845
  7. #6  0x00007ffff79315c7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x6209a0, Connection=Connection@entry=0x705270, ThreadID=ThreadID@entry=1459978) at /root/source/msquic/src/core/worker.c:489
  8. #7  0x00007ffff7931c52 in QuicWorkerLoop (Worker=Worker@entry=0x6209a0, ThreadID=ThreadID@entry=1459978, Reason=<optimized out>, Delay=Delay@entry=0x7ffff3ff0ec8) at /root/source/msquic/src/core/worker.c:644
  9. #8  0x00007ffff7931e00 in QuicWorkerThread (Context=0x6209a0) at /root/source/msquic/src/core/worker.c:727
  10. #9  0x00007ffff6ddb14a in start_thread () from /lib64/libpthread.so.0
  11. #10 0x00007ffff6b0adc3 in clone () from /lib64/libc.so.6

2.1.3.2 socket send

  1. #0  CxPlatSocketSendInternal (Socket=0xf48000, LocalAddress=0xf42148, RemoteAddress=0xf42164, SendData=0xe95200, IsPendedSend=IsPendedSend@entry=0 '\000') at /root/source/msquic/src/platform/datapath_epoll.c:2608
  2. #1  0x00007ffff79616ce in CxPlatSocketSend (Socket=<optimized out>, LocalAddress=<optimized out>, RemoteAddress=<optimized out>, SendData=<optimized out>, IdealProcessor=IdealProcessor@entry=3) at /root/source/msquic/src/platform/datapath_epoll.c:2700
  3. #2  0x00007ffff7934dba in QuicBindingSend (Binding=0xead040, LocalAddress=<optimized out>, RemoteAddress=<optimized out>, SendData=0xe95200, BytesToSend=0, DatagramsToSend=1, IdealProcessor=3) at /root/source/msquic/src/core/binding.c:1792
  4. #3  0x00007ffff7959ecf in QuicPacketBuilderSendBatch (Builder=Builder@entry=0x7ffff22e1a10) at /root/source/msquic/src/core/packet_builder.c:972
  5. #4  0x00007ffff795a04b in QuicPacketBuilderFinalize (Builder=Builder@entry=0x7ffff22e1a10, FlushBatchedDatagrams=FlushBatchedDatagrams@entry=1 '\001') at /root/source/msquic/src/core/packet_builder.c:928
  6. #5  0x00007ffff79552fa in QuicSendFlush (Send=Send@entry=0xf42a70) at /root/source/msquic/src/core/send.c:1304
  7. #6  0x00007ffff794060f in QuicConnDrainOperations (Connection=Connection@entry=0xf42000) at /root/source/msquic/src/core/connection.c:6845
  8. #7  0x00007ffff792e5c0 in QuicWorkerProcessConnection (Worker=Worker@entry=0xe4d3f0, Connection=Connection@entry=0xf42000, ThreadID=ThreadID@entry=2168192) at /root/source/msquic/src/core/worker.c:490
  9. #8  0x00007ffff792ec4b in QuicWorkerLoop (Worker=Worker@entry=0xe4d3f0, ThreadID=ThreadID@entry=2168192, Reason=<optimized out>, Delay=Delay@entry=0x7ffff22e1e48) at /root/source/msquic/src/core/worker.c:645
  10. #9  0x00007ffff792ee4a in QuicWorkerThread (Context=0xe4d3f0) at /root/source/msquic/src/core/worker.c:732
  11. #10 0x00007ffff69fd14a in start_thread () from /lib64/libpthread.so.0
  12. #11 0x00007ffff672cdc3 in clone () from /lib64/libc.so.6

2.1.3.3 stream发送

bcaktrace - 图2

2.1.4 recv

2.1.4.1 socket recv

  1. CxPlatSocketContextPrepareReceive
  2. 1. MsgHdr->msg_iov = &SocketContext->RecvIov[i];
  3. 2. SocketContext->RecvIov[i].iov_base = CurrentBlock->RecvPacket.Buffer;
  4. 3. CurrentBlock = SocketContext->CurrentRecvBlocks[i]
  6. 通过Connection->ReceiveQueueTail传递收到的socket消息
  7. #0  QuicConnQueueRecvDatagrams (Connection=Connection@entry=0xf42000, DatagramChain=DatagramChain@entry=0xe43508, DatagramChainLength=DatagramChainLength@entry=2) at /root/source/msquic/src/core/connection.c:2945
  8. #1  0x00007ffff793426e in QuicBindingDeliverDatagrams (Binding=Binding@entry=0xead040, DatagramChain=0xe43508, DatagramChainLength=DatagramChainLength@entry=2) at /root/source/msquic/src/core/binding.c:1560
  9. #2  0x00007ffff7934920 in QuicBindingReceive (Socket=<optimized out>, RecvCallbackContext=0xead040, DatagramChain=0x0) at /root/source/msquic/src/core/binding.c:1700
  10. #3  0x00007ffff795fa44 in CxPlatSocketContextRecvComplete (SocketContext=SocketContext@entry=0xf480c0, MessagesReceived=2) at /root/source/msquic/src/platform/datapath_epoll.c:1593
  11. #4  0x00007ffff7961539 in CxPlatSocketContextProcessEvents (EventPtr=<optimized out>, Events=1) at /root/source/msquic/src/platform/datapath_epoll.c:1814
  12. #5  0x00007ffff796168c in CxPlatDataPathWorkerThread (Context=0xe42200) at /root/source/msquic/src/platform/datapath_epoll.c:2773
  13. #6  0x00007ffff69fd14a in start_thread () from /lib64/libpthread.so.0
  14. #7  0x00007ffff672cdc3 in clone () from /lib64/libc.so.6

2.1.4.2 data recv

  1. #1---> QuicRecvBufferWrite---> 重传打印
  2. #1 QuicStreamProcessStreamFrame 步骤：
  3. 1.QuicRecvBufferWrite
  4.   1.1 确认异常情况，报文不是重传、没有越界、【缝隙包】
  5.   1.2 找到报文范围，QuicRangeAddRange：将[0，1]+[3，5] ===> [0，2]+[3，5]
  6.   1.3 如果现有报文是[0,2]，收到报文[5,5]，则变量赋值情况：(1)*ReadyToRead=0（不deliver） (2)*WriteLength!=0（有stream）
  7. 2.deliver时发送 FLUSH_STREAM_RECV 事件
  9. #0  QuicStreamRecvQueueFlush (Stream=Stream@entry=0x7fbfff8f1010) at /root/source/msquic/src/core/stream_recv.c:141
  10. #1  0x00007fbfff323f0e in QuicStreamProcessStreamFrame (Stream=Stream@entry=0x7fbfff8f1010, EncryptedWith0Rtt=EncryptedWith0Rtt@entry=0 '\000', Frame=Frame@entry=0x7fbffc1e9a40) at /root/source/msquic/src/core/stream_recv.c:439
  11. #2  0x00007fbfff324165 in QuicStreamRecv (Stream=Stream@entry=0x7fbfff8f1010, EncryptedWith0Rtt=<optimized out>, FrameType=FrameType@entry=QUIC_FRAME_STREAM_6, BufferLength=BufferLength@entry=1040, Buffer=Buffer@entry=0x7fbfec00b183 "\016", Offset=Offset@entry=0x7fbffc1e9b3c, UpdatedFlowControl=0x7fbffc1e9b3f "") at /root/source/msquic/src/core/stream_recv.c:586
  12. #3  0x00007fbfff337b74 in QuicConnRecvFrames (Connection=Connection@entry=0x15e14e0, Path=<optimized out>, Packet=Packet@entry=0x7fbfec00b740, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4340
  13. #4  0x00007fbfff338d9a in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x15e14e0, Path=<optimized out>, Path@entry=0x15e1608, BatchCount=BatchCount@entry=1 '\001', Datagrams=Datagrams@entry=0x7fbffc1e9d10, Cipher=Cipher@entry=0x7fbffc1e9c90 "\225\207-\217\367\200N\333Ϊ; 6,\b\220", RecvState=RecvState@entry=0x7fbffc1e9d54) at /root/source/msquic/src/core/connection.c:5071
  14. #5  0x00007fbfff339681 in QuicConnRecvDatagrams (Connection=Connection@entry=0x15e14e0, DatagramChain=0x0, DatagramChain@entry=0x7fbfec00b128, DatagramChainCount=DatagramChainCount@entry=1, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5336
  15. #6  0x00007fbfff3398dc in QuicConnFlushRecv (Connection=Connection@entry=0x15e14e0) at /root/source/msquic/src/core/connection.c:5420
  16. #7  0x00007fbfff33b88e in QuicConnDrainOperations (Connection=Connection@entry=0x15e14e0) at /root/source/msquic/src/core/connection.c:6831
  17. #8  0x00007fbfff3297a7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x14fc320, Connection=Connection@entry=0x15e14e0, ThreadID=ThreadID@entry=50050) at /root/source/msquic/src/core/worker.c:490
  18. #9  0x00007fbfff329e32 in QuicWorkerLoop (Worker=Worker@entry=0x14fc320, ThreadID=ThreadID@entry=50050, Reason=<optimized out>, Delay=Delay@entry=0x7fbffc1e9ec8) at /root/source/msquic/src/core/worker.c:645
  19. #10 0x00007fbfff32a0eb in QuicWorkerThread (Context=0x14fc320) at /root/source/msquic/src/core/worker.c:757
  20. #11 0x00007fbffe7d314a in start_thread () from /lib64/libpthread.so.0
  21. #12 0x00007fbffe502dc3 in clone () from /lib64/libc.so.6

2.1.4.3 stream接收处理

主要函数：QuicStreamProcessStreamFrame
bcaktrace - 图3

变量 case 1 case 2 case 3 case 4
WriteLength 【1】的长度 【4】的长度 =0 =0
Frame->Length 【1】的长度 【4】的长度 【2】的长度 【3】的长度
WrittenRanges 【0，1】 【0，1】【4】 【0，2】【4】 【0，4】
BaseOffset 0 1 1 2
ReadyToDeliver Yes No Yes Yes

2.1.4.4 ack recv

  1. #0  QuicStreamOnAck (Stream=0x7fffec027770, PacketFlags=..., FrameMetadata=FrameMetadata@entry=0x7fffec034378) at /root/source/msquic/src/core/stream_send.c:1360
  2. #1  0x00007ffff794dfb0 in QuicLossDetectionOnPacketAcknowledged (LossDetection=LossDetection@entry=0x705c78, EncryptLevel=EncryptLevel@entry=QUIC_ENCRYPT_LEVEL_1_RTT, Packet=0x7fffec034340) at /root/source/msquic/src/core/loss_detection.c:537
  3. #2  0x00007ffff794f666 in QuicLossDetectionProcessAckBlocks (LossDetection=LossDetection@entry=0x705c78, Path=Path@entry=0x7055e8, EncryptLevel=EncryptLevel@entry=QUIC_ENCRYPT_LEVEL_1_RTT, AckDelay=0, AckBlocks=AckBlocks@entry=0x705ac8, InvalidAckBlock=InvalidAckBlock@entry=0x7ffff47edaf0 "") at /root/source/msquic/src/core/loss_detection.c:1405
  4. #3  0x00007ffff794f839 in QuicLossDetectionProcessAckFrame (LossDetection=LossDetection@entry=0x705c78, Path=Path@entry=0x7055e8, EncryptLevel=EncryptLevel@entry=QUIC_ENCRYPT_LEVEL_1_RTT, FrameType=FrameType@entry=QUIC_FRAME_ACK, BufferLength=BufferLength@entry=5, Buffer=Buffer@entry=0x7fffec0040f3 "\002\t", Offset=0x7ffff47edb3c, InvalidFrame=0x7ffff47edaf0 "") at /root/source/msquic/src/core/loss_detection.c:1510
  5. #4  0x00007ffff793c202 in QuicConnRecvFrames (Connection=Connection@entry=0x7054c0, Path=<optimized out>, Packet=Packet@entry=0x7fffec0046b0, ECN=ECN@entry=CXPLAT_ECN_NON_ECT) at /root/source/msquic/src/core/connection.c:4162
  6. #5  0x00007ffff793d823 in QuicConnRecvDatagramBatch (Connection=Connection@entry=0x7054c0, Path=<optimized out>, Path@entry=0x7055e8, BatchCount=BatchCount@entry=1 '\001', Datagrams=Datagrams@entry=0x7ffff47edd10, Cipher=Cipher@entry=0x7ffff47edc90 "\300\027\237v@DF\375\037\025\345\213VF\n\316p", RecvState=RecvState@entry=0x7ffff47edd54) at /root/source/msquic/src/core/connection.c:5071
  7. #6  0x00007ffff793e10a in QuicConnRecvDatagrams (Connection=Connection@entry=0x7054c0, DatagramChain=0x0, DatagramChain@entry=0x7fffec004098, DatagramChainCount=DatagramChainCount@entry=1, IsDeferred=IsDeferred@entry=0 '\000') at /root/source/msquic/src/core/connection.c:5336
  8. #7  0x00007ffff793e365 in QuicConnFlushRecv (Connection=Connection@entry=0x7054c0) at /root/source/msquic/src/core/connection.c:5420
  9. #8  0x00007ffff7940317 in QuicConnDrainOperations (Connection=Connection@entry=0x7054c0) at /root/source/msquic/src/core/connection.c:6831
  10. #9  0x00007ffff792d9e7 in QuicWorkerProcessConnection (Worker=Worker@entry=0x620300, Connection=Connection@entry=0x7054c0, ThreadID=ThreadID@entry=80203) at /root/source/msquic/src/core/worker.c:490
  11. #10 0x00007ffff792e072 in QuicWorkerLoop (Worker=Worker@entry=0x620300, ThreadID=ThreadID@entry=80203, Reason=<optimized out>, Delay=Delay@entry=0x7ffff47edec8) at /root/source/msquic/src/core/worker.c:645
  12. #11 0x00007ffff792e32b in QuicWorkerThread (Context=0x620300) at /root/source/msquic/src/core/worker.c:757
  13. #12 0x00007ffff6dd714a in start_thread () from /lib64/libpthread.so.0
  14. #13 0x00007ffff6b06dc3 in clone () from /lib64/libc.so.6

3. 异常

3.1 ctrl+c

客户端ctrl+c，服务器不会收到信息，服务器调用接口也会成功，超时正常断开（因为超时短）
image.png
超时时间足够长时，会收到传输层的断开，然后关闭stream、conn
image.png

3.2 关闭后调用

  1. # 关闭stream
  2. MsQuic->StreamSend(Stream, SendBuffer, 1, QUIC_SEND_FLAG_FIN, SendBuffer)
  3. MsQuic->StreamClose(Stream);
  4. # 关闭conn
  5. MsQuic->ConnectionClose(Connection)
  8. # 调用stream send
  9. MsQuic->StreamSend(TestStream, SendBuffer, 1, QUIC_SEND_FLAG_NONE, SendBuffer)

image.png

3.3 优雅释放流

  1. 服务端
  2.     MsQuic->StreamShutdown(Stream, QUIC_STREAM_SHUTDOWN_FLAG_GRACEFUL, 0);
  4. 客户端
  5.     MsQuic->StreamShutdown(Stream, QUIC_STREAM_SHUTDOWN_FLAG_GRACEFUL, 0);
  7. StreamSend 函数中增加 QUIC_SEND_FLAG_FIN 标记位，相当于调用 StreamShutdown 函数 + QUIC_STREAM_SHUTDOWN_FLAG_GRACEFUL

image.png
image.png

3.4 崩溃释放流

  1. 服务端
  2.     MsQuic->StreamShutdown(TestStream, QUIC_STREAM_SHUTDOWN_FLAG_ABORT, 0);

image.png
image.png

3.5 立刻释放连接

  1. 服务端直接关闭
  2.     MsQuic->StreamShutdown(Stream, QUIC_STREAM_SHUTDOWN_FLAG_IMMEDIATE, 0);
  3.     MsQuic->ConnectionShutdown(Connection, QUIC_CONNECTION_SHUTDOWN_FLAG_SILENT, 0);

image.png
image.png

3.6 释放连接

  1. 服务端
  2.     MsQuic->StreamShutdown(TestStream, QUIC_STREAM_SHUTDOWN_FLAG_GRACEFUL, 0);
  3.     MsQuic->ConnectionShutdown(TestConn, QUIC_CONNECTION_SHUTDOWN_FLAG_NONE, 0);
  5. 客户端
  6.     MsQuic->StreamShutdown(Stream, QUIC_STREAM_SHUTDOWN_FLAG_GRACEFUL, 0);
  7.     MsQuic->ConnectionShutdown(Connection, QUIC_CONNECTION_SHUTDOWN_FLAG_NONE, 0);

image.png
image.png