go net/http Client 使用——长连接客户端的使用

go net/http Client使用——长连接客户端的使用_菜的抠脚的博客-CSDN博客 - 图1

hardecz 2017-12-06 15:46:40 go net/http Client使用——长连接客户端的使用_菜的抠脚的博客-CSDN博客 - 图2
17705 go net/http Client使用——长连接客户端的使用_菜的抠脚的博客-CSDN博客 - 图3
go net/http Client使用——长连接客户端的使用_菜的抠脚的博客-CSDN博客 - 图4
收藏 3

最后发布: 2017-12-06 15:46:40 首次发布: 2017-12-06 15:46:40

go net/http Client 使用总结

Client 数据结构

  1. `2.  // A Client is an HTTP client. Its zero value (DefaultClient) is a
  3. 3.  // usable client that uses DefaultTransport.
  5. 4.  //
  7. 5.  // The Client's Transport typically has internal state (cached TCP
  9. 6.  // connections), so Clients should be reused instead of created as
  11. 7.  // needed. Clients are safe for concurrent use by multiple goroutines.
  13. 8.  //
  15. 9.  // A Client is higher-level than a RoundTripper (such as Transport)
  17. 10.  // and additionally handles HTTP details such as cookies and
  19. 11.  // redirects.
  21. 12.  type  Client  struct  {
  23. 13.      // Transport specifies the mechanism by which individual
  25. 14.      // HTTP requests are made.
  27. 15.      // If nil, DefaultTransport is used.
  29. 16.      Transport  RoundTripper
  32. 18.      // CheckRedirect specifies the policy for handling redirects.
  34. 19.      // If CheckRedirect is not nil, the client calls it before
  36. 20.      // following an HTTP redirect. The arguments req and via are
  38. 21.      // the upcoming request and the requests made already, oldest
  40. 22.      // first. If CheckRedirect returns an error, the Client's Get
  42. 23.      // method returns both the previous Response (with its Body
  44. 24.      // closed) and CheckRedirect's error (wrapped in a url.Error)
  46. 25.      // instead of issuing the Request req.
  48. 26.      // As a special case, if CheckRedirect returns ErrUseLastResponse,
  50. 27.      // then the most recent response is returned with its body
  52. 28.      // unclosed, along with a nil error.
  54. 29.      //
  56. 30.      // If CheckRedirect is nil, the Client uses its default policy,
  58. 31.      // which is to stop after 10 consecutive requests.
  60. 32.      CheckRedirect  func(req  *<span class="" "="">Request,  via  []*Request)  error
  63. 34.      // Jar specifies the cookie jar.
  65. 35.      // If Jar is nil, cookies are not sent in requests and ignored
  67. 36.      // in responses.
  69. 37.      Jar  CookieJar
  72. 39.      // Timeout specifies a time limit for requests made by this
  74. 40.      // Client. The timeout includes connection time, any
  76. 41.      // redirects, and reading the response body. The timer remains
  78. 42.      // running after Get, Head, Post, or Do return and will
  80. 43.      // interrupt reading of the Response.Body.
  82. 44.      //
  84. 45.      // A Timeout of zero means no timeout.
  86. 46.      //
  88. 47.      // The Client cancels requests to the underlying Transport
  90. 48.      // using the Request.Cancel mechanism. Requests passed
  93. 50.                  ay still set Request.Cancel; both will
  95. 51.      // cancel the request.
  97. 52.      //
  99. 53.      // For compatibility, the Client will also use the deprecated
  101. 54.      // CancelRequest method on Transport if found. New
  103. 55.      // RoundTripper implementations should use Request.Cancel
  105. 56.      // instead of implementing CancelRequest.
  107. 57.      Timeout  time.Duration
  109. 58.  }`

注意Transport数据结构，对应如下：

  1. `1.  // Transport is an implementation of RoundTripper that supports HTTP,
  3. 2.  // HTTPS, and HTTP proxies (for either HTTP or HTTPS with CONNECT).
  5. 3.  //
  7. 4.  // By default, Transport caches connections for future re-use.
  9. 5.  // This may leave many open connections when accessing many hosts.
  11. 6.  // This behavior can be managed using Transport's CloseIdleConnections method
  13. 7.  // and the MaxIdleConnsPerHost and DisableKeepAlives fields.
  15. 8.  //
  17. 9.  // Transports should be reused instead of created as needed.
  19. 10.  // Transports are safe for concurrent use by multiple goroutines.
  21. 11.  //
  23. 12.  // A Transport is a low-level primitive for making HTTP and HTTPS requests.
  25. 13.  // For high-level functionality, such as cookies and redirects, see Client.
  27. 14.  //
  29. 15.  // Transport uses HTTP/1.1 for HTTP URLs and either HTTP/1.1 or HTTP/2
  31. 16.  // for HTTPS URLs, depending on whether the server supports HTTP/2.
  33. 17.  // See the package docs for more about HTTP/2.
  35. 18.  type  Transport  struct  {
  37. 19.      idleMu  sync.Mutex
  39. 20.      wantIdle  bool  // user has requested to close all idle conns
  41. 21.      idleConn  map[connectMethodKey][]*persistConn  // most recently used at end
  43. 22.      idleConnCh  map[connectMethodKey]chan  *persistConn
  45. 23.      idleLRU  connLRU
  48. 25.      reqMu  sync.Mutex
  50. 26.      reqCanceler  map[*Request]func()
  53. 28.      altMu  sync.RWMutex
  55. 29.      altProto  map[string]RoundTripper  // nil or map of URI scheme => RoundTripper
  58. 31.      // Proxy specifies a function to return a proxy for a given
  60. 32.      // Request. If the function returns a non-nil error, the
  62. 33.      // request is aborted with the provided error.
  64. 34.      // If Proxy is nil or returns a nil *URL, no proxy is used.
  66. 35.      Proxy  func(*Request)  (*url.URL,  error)
  69. 37.      // DialContext specifies the dial function for creating unencrypted TCP connections.
  71. 38.      // If DialContext is nil (and the deprecated Dial below is also nil),
  73. 39.      // then the transport dials using package net.
  75. 40.      DialContext  func(ctx  context.Context,  network,  addr  string)  (net.Conn,  error)
  78. 42.      // Dial specifies the dial function for creating unencrypted TCP connections.
  80. 43.      //
  82. 44.      // Deprecated: Use DialContext instead, which allows the transport
  84. 45.      // to cancel dials as soon as they are no longer needed.
  86. 46.      // If both are set, DialContext takes priority.
  88. 47.      Dial  func(network,  addr  string)  (net.Conn,  error)
  91. 49.      // DialTLS specifies an optional dial function for creating
  93. 50.      // TLS connections for non-proxied HTTPS requests.
  95. 51.      //
  97. 52.      // If DialTLS is nil, Dial and TLSClientConfig are used.
  99. 53.      //
  101. 54.      // If DialTLS is set, the Dial hook is not used for HTTPS
  103. 55.      // requests and the TLSClientConfig and TLSHandshakeTimeout
  105. 56.      // are ignored. The returned net.Conn is assumed to already be
  107. 57.      // past the TLS handshake.
  109. 58.      DialTLS  func(network,  addr  string)  (net.Conn,  error)
  112. 60.      // TLSClientConfig specifies the TLS configuration to use with
  114. 61.      // tls.Client. If nil, the default configuration is used.
  116. 62.      TLSClientConfig  *tls.Config
  119. 64.      // TLSHandshakeTimeout specifies the maximum amount of time waiting to
  121. 65.      // wait for a TLS handshake. Zero means no timeout.
  123. 66.      TLSHandshakeTimeout  time.Duration
  126. 68.      // DisableKeepAlives, if true, prevents re-use of TCP connections
  128. 69.      // between different HTTP requests.
  130. 70.      DisableKeepAlives  bool
  133. 72.      // DisableCompression, if true, prevents the Transport from
  135. 73.      // requesting compression with an "Accept-Encoding: gzip"
  137. 74.      // request header when the Request contains no existing
  139. 75.      // Accept-Encoding value. If the Transport requests gzip on
  141. 76.      // its own and gets a gzipped response, it's transparently
  143. 77.      // decoded in the Response.Body. However, if the user
  145. 78.      // explicitly requested gzip it is not automatically
  147. 79.      // uncompressed.
  149. 80.      DisableCompression  bool
  152. 82.      // MaxIdleConns controls the maximum number of idle (keep-alive)
  154. 83.      // connections across all hosts. Zero means no limit.
  156. 84.      MaxIdleConns  int
  159. 86.      // MaxIdleConnsPerHost, if non-zero, controls the maximum idle
  161. 87.      // (keep-alive) connections to keep per-host. If zero,
  163. 88.      // DefaultMaxIdleConnsPerHost is used.
  165. 89.      MaxIdleConnsPerHost  int
  168. 91.      // IdleConnTimeout is the maximum amount of time an idle
  170. 92.      // (keep-alive) connection will remain idle before closing
  172. 93.      // itself.
  174. 94.      // Zero means no limit.
  176. 95.      IdleConnTimeout  time.Duration
  179. 97.      // ResponseHeaderTimeout, if non-zero, specifies the amount of
  181. 98.      // time to wait for a server's response headers after fully
  183. 99.      // writing the request (including its body, if any). This
  185. 100.      // time does not include the time to read the response body.
  187. 101.      ResponseHeaderTimeout  time.Duration
  190. 103.      // ExpectContinueTimeout, if non-zero, specifies the amount of
  192. 104.      // time to wait for a server's first response headers after fully
  194. 105.      // writing the request headers if the request has an
  196. 106.      // "Expect: 100-continue" header. Zero means no timeout.
  198. 107.      // This time does not include the time to send the request header.
  200. 108.      ExpectContinueTimeout  time.Duration
  203. 110.      // TLSNextProto specifies how the Transport switches to an
  205. 111.      // alternate protocol (such as HTTP/2) after a TLS NPN/ALPN
  207. 112.      // protocol negotiation. If Transport dials an TLS connection
  209. 113.      // with a non-empty protocol name and TLSNextProto contains a
  211. 114.      // map entry for that key (such as "h2"), then the func is
  213. 115.      // called with the request's authority (such as "example.com"
  215. 116.      // or "example.com:1234") and the TLS connection. The function
  217. 117.      // must return a RoundTripper that then handles the request.
  219. 118.      // If TLSNextProto is nil, HTTP/2 support is enabled automatically.
  221. 119.      TLSNextProto  map[string]func(authority  string,  c  *tls.Conn)  RoundTripper
  224. 121.      // MaxResponseHeaderBytes specifies a limit on how many
  226. 122.      // response bytes are allowed in the server's response
  228. 123.      // header.
  230. 124.      //
  232. 125.      // Zero means to use a default limit.
  234. 126.      MaxResponseHeaderBytes  int64
  237. 128.      // nextProtoOnce guards initialization of TLSNextProto and
  239. 129.      // h2transport (via onceSetNextProtoDefaults)
  241. 130.      nextProtoOnce  sync.Once
  243. 131.      h2transport  *http2Transport  // non-nil if http2 wired up
  246. 133.      // TODO: tunable on max per-host TCP dials in flight (Issue 13957)
  248. 134.  }`

实际上是一个连接池，所以在使用的过程中通常是初始化一个Transport和Client，在每个处理函数中都共用该连接池，例如如下 Demo：

  1. `
  3. 1.  package  main
  6. 3.  import  (
  8. 4.      "bytes"
  10. 5.      "io/ioutil"
  12. 6.      "log"
  14. 7.      "net"
  16. 8.      "net/http"
  18. 9.      "time"
  20. 10.  )
  23. 12.  var  (
  25. 13.      httpClient  *http.Client
  27. 14.  )
  30. 16.  // init HTTPClient
  32. 17.  func  init()  {
  34. 18.      httpClient  =  createHTTPClient()
  36. 19.  }
  39. 21.  const  (
  41. 22.      MaxIdleConns  int  =  100
  43. 23.      MaxIdleConnsPerHost  int  =  100
  45. 24.      IdleConnTimeout  int  =  90
  47. 25.  )
  50. 27.  // createHTTPClient for connection re-use
  52. 28.  func  createHTTPClient()  *http.Client  {
  54. 29.      client  :=  &http.Client{
  56. 30.          Transport:  &http.Transport{
  58. 31.              Proxy:  http.ProxyFromEnvironment,
  60. 32.              DialContext:  (&net.Dialer{
  62. 33.                  Timeout:  30  *  time.Second,
  64. 34.                  KeepAlive:  30  *  time.Second,
  66. 35.              }).DialContext,
  68. 36.              MaxIdleConns:  MaxIdleConns,
  70. 37.              MaxIdleConnsPerHost:  MaxIdleConnsPerHost,
  72. 38.              IdleConnTimeout: time.Duration(_IdleConnTimeout_)*  time.Second,
  74. 39.    },
  77. `
  3.   Timeout: 20 \* time._Second_,
  1.  `1.    }
  3. 2.      return  client
  5. 3.  }
  8. 5.  func  main()  {
  10. 6.      var  endPoint  string  =  "https://localhost:8080/doSomething"
  13. 8.      req,  err  :=  http.NewRequest("POST",  endPoint,  bytes.NewBuffer([]byte("Post this data")))
  15. 9.      if  err  !=  nil  {
  17. 10.          log.Fatalf("Error Occured. %+v",  err)
  19. 11.      }
  21. 12.      req.Header.Set("Content-Type",  "application/x-www-form-urlencoded")
  24. 14.      // use httpClient to send request
  26. 15.      response,  err  :=  httpClient.Do(req)
  28. 16.      if  err  !=  nil  &&  response  ==  nil  {
  30. 17.          log.Fatalf("Error sending request to API endpoint. %+v",  err)
  32. 18.      }  else  {
  34. 19.          // Close the connection to reuse it
  36. 20.          defer  response.Body.Close()
  39. 22.          // Let's check if the work actually is done
  41. 23.          // We have seen inconsistencies even when we get 200 OK response
  43. 24.          body,  err  :=  ioutil.ReadAll(response.Body)
  45. 25.          if  err  !=  nil  {
  47. 26.              log.Fatalf("Couldn't parse response body. %+v",  err)
  49. 27.          }
  52. 29.          log.Println("Response Body:",  string(body))
  54. 30.      }
  57. 32.  }`

下面详细讲解Transport结构需要注意的几个参数设置：

  • DisableKeepAlives
  1. `
  3. 1.  // DisableKeepAlives, if true, prevents re-use of TCP connections
  5. 2.  // between different HTTP requests.
  8. `

表示是否开启 http keepalive 功能，也即是否重用连接，默认开启 (false)

  • MaxIdleConns
  1. `
  3. 1.  // MaxIdleConns controls the maximum number of idle (keep-alive)
  5. 2.  // connections across all hosts. Zero means no limit.
  8. `

表示连接池对所有 host 的最大链接数量，host 也即 dest-ip，默认为无穷大（0），但是通常情况下为了性能考虑都要严格限制该数目（实际使用中通常利用压测 二分得到该参数的最佳近似值）。太大容易导致客户端和服务端的 socket 数量剧增，导致内存吃满，文件描述符不足等问题；太小则限制了连接池的 socket 数量，资源利用率较低。

  • MaxIdleConnsPerHost
  1. `
  3. 1.  // MaxIdleConnsPerHost, if non-zero, controls the maximum idle
  5. 2.  // (keep-alive) connections to keep per-host. If zero,
  7. 3.  // DefaultMaxIdleConnsPerHost is used.
  10. `

表示连接池对每个 host 的最大链接数量，从字面意思也可以看出：

  1. `MaxIdleConnsPerHost <= MaxIdleConns`

如果客户端只需要访问一个 host，那么最好将MaxIdleConnsPerHost与MaxIdleConns设置为相同，这样逻辑更加清晰。

  • IdleConnTimeout
  1. `
  3. 1.  // IdleConnTimeout is the maximum amount of time an idle
  5. 2.  // (keep-alive) connection will remain idle before closing 
  7. 3.  // itself.
  9. 4.  // Zero means no limit.
  12. `

空闲 timeout 设置，也即 socket 在该时间内没有交互则自动关闭连接（注意：该 timeout 起点是从每次空闲开始计时，若有交互则重置为 0）, 该参数通常设置为分钟级别，例如：90 秒。

  • DialContext
  1. `
  3. 1.  // DialContext specifies the dial function for creating unencrypted TCP connections.
  5. 2.  // If DialContext is nil (and the deprecated Dial below is also nil),
  7. 3.  // then the transport dials using package net.
  9. 4.  DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
  12. `

该函数用于创建 http（非 https）连接，通常需要关注Timeout和KeepAlive参数。前者表示建立 Tcp 链接超时时间；后者表示底层为了维持 http keepalive 状态 每隔多长时间发送 Keep-Alive 报文。Timeout通常设置为 30s（网络环境良好），KeepAlive通常设置为 30s(与 IdleConnTimeout 要对应)。

补充说明

Transport数据结构中的MaxConnsPerHost参数目前正在研发中（Go 项目），其表示对每个 host 可以发出的最大连接个数（包括长链接和短链接），是非常有用的参数，可以用来对 socket 数量进行限制，配合MaxIdleConns和MaxIdleConnsPerHost一起使用。

net/http Transport MaxConnsPerHost

参考