04-代理服务器 - 04-2-L7负载均衡 - 《Nginx 学习笔记》

1. http_upstream_module
2. nginx_upstream_check_module
3. Example
- 3.1. Basic SLB

1. http_upstream_module

1.1. upstream

Directives

Syntax: upstream name {...}
Default: Close
Context: http

Instroduction

Defines a group of servers,in addition,servers listening in TCP and UNIX-domian sockets can be mixed.

1.2. server

Directives

Syntax: server address [paremters] ;
Default: Close
Context: upstream

Instroduction

Defines the address and other parameters of a server.If a port is not specified , the port 80 is used.

Paremeters
- weigth=number

sets the weight of the server ,default 1.

fail_timeout=time

If the counts that nginx sends request to proxied server unsuccessful in fail timeout,nginx will consider the proxied server unavailable,then in fail timeout may no request send to proxied servers. Default 10 seconds.

max_fails=number

sets the number of unsuccessful attempts to communicate with the server that should happen in the duration set by the fail_timeout parameter to consider the server unavailable for a duration also set by the fail_timeout parameter.

backup

Marks the server as a backup server.If all primary servers unavailable ,the backup server will be used.

down

Marks the server as a unavailable server.

slow_start=time

If the backend server is recoverd from unavaliable status,it will delay slow_start time to used.It will useful.
But in hash or ip_hash load balancing methods ,the parameter can’t be used.
In version 1.14.2,the directive(slow_start) is uninvalid.We can achieve delay start by nginx_upstream_check_module.

max_conns=number

Limits the maximum number of simultaneous active connections to the proxied server.default 0,that means no limit.
If idle keepalive connections , multiple workers,and shared memory are enabled,the total numnber of active and idle connections to the proxied server may exceed the max_conns value.

1.3. ip_hash

Directives

Syntax: ip_hash ;
Default: Close 
Context: upstream

Instroduction

Specifies load balancing method bases on client IP address.After 1.2.2 and 1.3.2,ip_hash supportes IPV6.
But in sometime , $remoteaddr is not client address,it may be proxy server IP or CDN ip.ip_hash_ is very useful when client connects to nginx server direct.
Notice:
If one of the servers need to be temporarily removed,it should be marked with the down paramter .
If need keepalive session , it suggests use cache server,such as redis.

1.4. hash key

Directives

Syntax: hash key ;
Default: Close 
Context: upstream

Instroduction

The hash key is used to solve ip_hash directive can’t get client ip issue.The hash key can use $X-Real_IP instead of $remote_addr.

1.5. keepalive

Directives

Syntax: keepalive connections ;
Default: Close
Context: upstream

Instroduction

Sets maximum idle keepalive connections of each worker process.When this number is exceeded,the least recently used connection will close.
The directive doesn’t limit the total number of connection of each worker process can open.The value should be set to a number small enough to let upstream servers process new incomming as well.
The directive should be set after load balancer methods.In most time , it seted in last line.

http example

In http upstream keepalive,the http upstream keepalive,the http verision should be set to 1.1 and the Connection head shoul be cleared(default connection will be set to Close).

upstream http_backend {
    server 127.0.0.1:8080;
    keepalive 16;
}
server {
    ...
    location /http/ {
        proxy_pass http://http_backend;
        proxy_http_version 1.1 ;
        proxy_set_header Connection "";
        ...
    }
}

memcached example

upstream memcached_backend {
  server 127.0.0.1:11211;
  server 10.0.0.2:11211;
  keepalive 32;
}
server {
  ...
  location /memcached/ {
      set $memcached_key $uri;
      memcached_pass memcached_backend;
  }
}

FastCGI

In fastcgi upstream keepalive,the fastcgi_keep_conn must be set on.

upstream fastcgi_backend {
    server 127.0.0.1:9000;
    keepalive 8;
}
server {
    ...
    location /fastcgi/ {
        fastcgi_pass fastcgi_backend;
        fastcgi_keep_conn on;
        ...
    }
}

1.6. keepalive_requests

Directives

Syntax: keepalive_requests number ;
Default: keepalive_requests 100 ;
Context: upstream

Instroduction

Sets the maximum number of requests that can be served through one keepalive connection. After the maximum number of requests is made, the connection is closed.The direcitve apperared in 1.15.3.

1.7. keepalive_timeout

Directives

Syntax: keepalive_timeout time;
Default: keepalive_timeout 60s ;
Context: upstream

Instroduction

Sets a timeout during which an idle keepalive connection to an upstream server will stay open.The direcitve apperared in 1.15.3.

1.8. least_conn

Directives

Syntax: least_conn;
Default: Close
Context: upstream

Instroduction

Specifies that a group should use a load balancing method where a request is passed to the server with the least number of active connections, taking into account weights of servers.

2. nginx_upstream_check_module

2.1. Install module

download: https://github.com/yaoweibin/nginx_upstream_check_module
document: http://tengine.taobao.org/document_cn/http_upstream_check_cn.html

[root@centos-81 apps]# git clone https://github.com/yaoweibin/nginx_upstream_check_module.git
[root@centos-81 apps]# cd nginx-1.14.2/
[root@centos-81 nginx-1.14.2]# patch -p1 < ../nginx_upstream_check_module/check_1.14.0+.patch

[root@centos-81 nginx-1.14.2]# ./configure —prefix=/opt/apps/nginx —user=nginx —group=nginx —with-compat —with-file-aio —with-threads —with-http_addition_module —with-http_auth_request_module —with-http_dav_module —with-http_flv_module —with-http_gunzip_module —with-http_gzip_static_module —with-http_mp4_module —with-http_random_index_module —with-http_realip_module —with-http_secure_link_module —with-http_slice_module —with-http_ssl_module —with-http_stub_status_module —with-http_sub_module —with-http_v2_module —with-mail —with-mail_ssl_module —with-stream —with-stream_realip_module —with-stream_ssl_module —with-stream_ssl_preread_module —with-cc-opt=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong —param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC’ —with-ld-opt=’-Wl,-z,relro -Wl,-z,now -pie’ —add-module=../nginx_upstream_check_module ## add module to old nginx
[root@centos-81 nginx-1.14.2]# make -j 4

[root@centos-81 nginx-1.14.2]# mv /opt/apps/nginx/sbin/nginx /opt/apps/nginx/sbin/nginx.old
[root@centos-81 nginx-1.14.2]# cp objs/nginx /opt/apps/nginx/sbin/
[root@centos-81 nginx-1.14.2]# /opt/apps/nginx/sbin/nginx -c /opt/apps/nginx/conf/nginx.conf -t
[root@centos-81 nginx-1.14.2]# kill -USR2 $(cat /opt/apps/nginx/logs/nginx.pid)
[root@centos-81 nginx-1.14.2]# kill -WINCH $(cat /opt/apps/nginx/logs/nginx.pid.oldbin)
[root@centos-81 nginx-1.14.2]# kill -QUIT $(cat /opt/apps/nginx/logs/nginx.pid.oldbin)

2.2. check

Directives

Syntax: check interval=milliseconds [fall=count] [rise=count] [timeout=milliseconds] [default_down=true|false] [type=tcp|http|ssl_hello|mysql|ajp] [port=check_port]
Default: check interval=30000 fall=5 rise=2 timeout=1000 default_down=true type=tcp ;
Context: upstream

Instroduction

Add health check for the upstream servers.

interval

The check request’s interval time.If upstream server unhealth,the interval time is equal to the value,but if last check is successful the next check need 60s+ .

fall

Sets unsuccessful counts that server will marke down.

rise

Sets successful counts that server will marke up.

timeout

Sets check request timeout.

default_down

Specified upstream server default status,default is true.

type

the check protocol type:

  - tcp: a simple TCP socket connect and peek one byte.
  - ssl_hello: send a client SSL hello packet and receive the server SSL hello packet.
  - http: send a http request packet, receive and parse the http response to diagnose if the upstream server is alive.
  - mysql: connect to the mysql server, receive the greeting response to diagnose if the upstream server is alive.
  - ajp: send an AJP Cping packet, receive and parse the AJP Cpong response to diagnose if the upstream server is alive.

port

specify the check port in the backend servers. It can be different with the original servers port. Default the port is 0 and it means the same as the original backend server.

2.3. check_keepalive_requests

Directives

Syntax: check_keepalive_requests number ;
Default: 1
Context: upstream

Instroduction

The directive specifies the number of requests sent on a connection, the default vaule 1 indicates that tengine will certainly close the connection after a request.
If the number > 1,you need specify HTTP/1.1.

2.4. check_http_send

Directives

Syntax: check_http_send request_header ;
Default: check_http_send "GET / HTTP/1.0\r\n\r\n"
Context: upstream

Instroduction

If the check type is http, the check function will send this http packet to the upstream server. Suggest:
check_http_send “HEAD /check.html HTTP/1.1\nHost: 127.0.0.1\nConnection: keep-alive\n\n” ;

2.5. check_http_expect_alive

Directives

Syntax: check_http_expect_alive http_2xx|http_3xx|http_4xx|http_5xx ;
Default: check_http_expect_alive http_2xx http_3xx ;
Context: upsteam

Instroduction

These status codes indicate the upstream server’s http response is OK and the check response is successful.

2.6. check_shm_size

Directives

Syntax: check_shm_size size ;
Default: check_shm_size 1M ;
Context: http

Instroduction

Default size is one megabytes. If you want to check thousands of servers, the shared memory may be not enough, you can enlarge it with this directive.

3. Example

3.1. Basic SLB

3.1.1. Nginx Configuration

server {
    listen       80;
    server_name  localhost;
    location / {
        proxy_pass http://test_proxy ;
        proxy_buffers 16 4k ;
        proxy_connect_timeout  15 ;
        proxy_read_timeout     15 ;
        proxy_send_timeout     10 ;
        proxy_set_header X-Real-IP $remote_addr ;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for ;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html ;
    }
}
upstream test_proxy {
    server 192.168.1.50:8080 ;
    server 192.168.1.50:8081 weight=2;
    server 192.168.1.50:8082 ;
    server 192.168.1.50:8083 ;
    keepalive          64 ;
    check interval=3000 fall=2 rise=3 timeout=1000 default_down=true type=http ;
    check_keepalive_requests 100 ;
    check_http_send "HEAD /check.html HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: keep-alive\r\n\r\n" ;
    check_http_expect_alive http_2xx http_3xx ;
}

3.1.2. LB Test

[root@centos-81 ~]# for i in {1..100};do curl -s 192.168.1.81/ ;done | sort |uniq -c

     20 vhost01
     40 vhost02
     20 vhost03
     20 vhost04

3.1.3. Health Check Test

[root@centos-50 ~]# cd /usr/share/nginx/html/vhost01/ ; mv check.html check.html.bak
### Wait 1 minute for next health check.If check unsuccessful,the next check just 3s later.
[root@centos-81 ~]# for i in {1..100};do curl -s 192.168.1.81/ ;done | sort |uniq -c

     50 vhost02
     25 vhost03
     25 vhost04

[root@centos-50 ~]# grep HEAD /var/log/nginx/access.log

### Health Check log.You can close the check log.
- - - [11/Jan/2019:07:04:18 +0800] "HEAD /check.html HTTP/1.1" 200 0 "-" "-" "-"
- - - [11/Jan/2019:07:05:07 +0800] "HEAD /check.html HTTP/1.1" 404 0 "-" "-" "-"
- - - [11/Jan/2019:07:05:12 +0800] "HEAD /check.html HTTP/1.1" 404 0 "-" "-" "-"
- - - [11/Jan/2019:07:05:15 +0800] "HEAD /check.html HTTP/1.1" 404 0 "-" "-" "-"
- - - [11/Jan/2019:07:05:18 +0800] "HEAD /check.html HTTP/1.1" 404 0 "-" "-" "-"
- - - [11/Jan/2019:07:05:21 +0800] "HEAD /check.html HTTP/1.1" 404 0 "-" "-" "-"