1. L4 LB
2. Example
- 2.1. LB for Kubernets Master
  - 2.1.1. Nginx Config
  - 2.1.2. Test
- 2.2. Proxy To HTTP Server
  - 2.2.1. Nginx Config
    - 2.2.1.1. L4 Proxy Server
    - 2.2.1.2. Web Server
  - 2.2.2. Test

1. L4 LB

Nginx L4 LB process TCP/UPD by stream directive by 7 stages:

1.    Post-accept
        The first phase after accepting a client connection. The ngx_stream_realip_module module is invoked at this phase.
2.    Pre-access
        Preliminary check for access. The ngx_stream_limit_conn_module and ngx_stream_set_module modules are invoked at this phase.
3.    Access
        Client access limitation before actual data processing. The ngx_stream_access_module module is invoked at this phase.
4.    SSL
        TLS/SSL termination. The ngx_stream_ssl_module module is invoked at this phase.
5.    Preread
        Reading initial bytes of data into the preread buffer to allow modules such as ngx_stream_ssl_preread_module analyze the data before its processing.
6.    Content
        Mandatory phase where data is actually processed, usually proxied to upstream servers, or a specified value is returned to a client.
7.    Log
        The final phase where the result of a client session processing is recorded. The ngx_stream_log_module module is invoked at this phase.

1.1. ngx_stream_core_module

1.1.1. stream

Directives

Syntax: stream {...}
Default: Close
Context: main

Instroduction

Provides the configuration file context in which the stream server directives are specified.

1.1.2. server

Directives

Syntax: server {...}
Default: Close
Context: stream

Instroduction

Configuration of server, some like L7 virtual host server .

1.1.3. listen

Directives

Syntax: listen [address:]port [ssl] [udp] [proxy_protocol] [backlog=number] [rcvbuf=size] [sndbuf=size] [bind] [ipv6only=on|off] [reuseport] [so_keepalive=on|off|[keepidle]:[keepintvl]:[keepcnt]];
Default: Close
Context: stream

Instroduction

Listen address and port for socket on which the server will accept connections. The address and port support multiple formats, such as:

listen 6443;
listen 127.0.0.1:6443;
listen 0.0.0.0:6443;
listen 0.0.0.0:6443-6446;
listen unix:/var/run/nginx.sock;

ssl parameter allows specifying that all connections accept on the port must be work in SSL mode.
udp parameter specifying use UDP protocol, default is tcp protocol.In upd mode, you shoude specify reuseportparameter to handle package from same port and address.
protocolparameter allow all connections use proxy protocol.If fronted server(L4 LB) use open proxy protocol, the current must be sepcified protocol parameter, usualy the parameter used for tranfer client to backend server.
backlogparameter limit maxinum length queue for connections. General linux platform is 511 default.
other parameters instruduction doc

1.1.4. preread_buffer_size

Directives

Syntax: preread_buffer_size size
Default: preread_buffer_size 16k;
Context: stream, server

Instroduction

Specifies a size of the preread buffer

1.1.5. preread_timeout

Directives

Syntax: preread_timeout  timeout
Default: preread_timeout 30;
Context: stream, server

Instroduction

Specifies a timeout of the preread phase.

1.1.6. resolver

Directives

Syntax: resolver address ... [valid=time] [ipv6=on|off] [status_zone=zone] 
Default: Close
Context: stream, server

Instroduction

Specifies dns server to resolve upstream server form domian name to IP address.
valid parameter specified dns cache life.
status_zoneparameter (1.17.1) ，is not community feature.

1.1.7. preread_timeout

Directives

Syntax: resolver_timeout timeout
Default: resolver_timeout 30;
Context: stream, server

Instroduction

Sets a timeout for name resolution.

1.2. ngx_stream_proxy_module

The ngx_stream_proxy_module module (1.9.0) allows proxying data streams over TCP, UDP (1.9.13), and UNIX-domain sockets.

1.2.1. proxy_pass

Directives

Syntax: proxy_pass address;
Default: Close
Context: server

Instroduction

Sets proxied server address, the address can be specified by IP, domain name, socket path or upstream name.For example:

proxy_pass 127.0.0.1:8080;
proxy_pass unix:/var/run/test.sock;
proxy_pass $upstream;

1.2.2. proxy_protocol

Directives

Syntax: proxy_protocol on|off;
Default: proxy_protocol off;
Context: stream, server

Instroduction

Add client IP to network package which transport to proxied server, the proxied server need use protocol to resolv request.

1.2.3. proxy_buffer_size

Directives

Syntax: proxy_buffer_size size;
Default: proxy_buffer_size 16k;
Context: stream, server

Instroduction

Sets the size of the buffer used for reading data from the proxied server. Also sets the size of the buffer used for reading data from the client.

1.2.4. proxy_connect_timeout

Directives

Syntax: proxy_connect_timeout timeout;
Default: proxy_connect_timeout 60s;
Context: stream, server

Instroduction

Defines a timeout for establishing a connection with a proxied server.

1.2.5. proxy_download_rate

Directives

Syntax: proxy_download_rate rate;
Default: proxy_download_rate 0;
Context: stream, server

Instroduction

Limit reading bytes per second from proxied server on one connection. Default 0 is no limit.

1.2.6. proxy_next_upstream

Directives

Syntax: proxy_next_upstream on | off;
Default: proxy_next_upstream on;
Context: stream, server

Instroduction

When a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server.

1.2.6. proxy_next_upstream

Directives

Syntax: proxy_next_upstream_timeout;
Default: proxy_next_upstream_timeout 0;
Context: stream, server

Instroduction

Limits the time allowed to pass a connection to the next server. The 0 value turns off this limitation.

1.2.6. proxy_next_upstream_tries

Directives

Syntax: proxy_next_upstream_tries rate;
Default: proxy_next_upstream_tries 0;
Context: stream, server

Instroduction

Limits the number of possible tries for passing a connection to the next server. The 0 value turns off this limitation.

1.3. ngx_stream_realip_module

1.3.1. set_real_ip_from

Directives

Syntax: set_real_ip_from address|CIDR|unix;
Default: Close
Context: stream, server

Instroduction

Defines trusted address that are know to send corrent replacement addresses.
If specified, the tow variables can use for next process:

$realip_remote_addr: real client IP
$realip_remote_port: keeps the original client port
1.4. ngx_stream_log_module
1.4.1. log_format

Directives

Syntax: log_format name [escape=default|json|none] string ...;
Default: Close
Context: stream

Instroduction

Sets log format, some variables refer: core_module, all variables

1.4.2. access_log

Directives

Syntax: access_log path format [buffer=size] [gzip[=level]] [flush=time] [if=condition];
              access_log off;
Default: Close
Context: stream, server

Instroduction

Sets log access log configure:
buffer paramter specified for cache access log on memory, it useful when requests is too many.
gzip parameter specified compress access log, used zactorzgrepto view logs.When use gzipcompress, the cache will be cache on memory, defautl buffer is 64k, default compress is 1.

1.5. ngx_stream_upstream_module

1.5.1. upstream

Directives

Syntax: upstream name { ... }
Default: Close
Context: stream

Instroduction

Defines a group of servers. Servers can listen on different ports. In addition, servers listening on TCP and UNIX-domain sockets can be mixed.

1.5.2. server

Directives

Syntax: server address [parameters];
Default: Close
Context: upstream

Instroduction

Sets proxied server configration.
weigth=num sets the weight of the server, by default, 1.
max_conns=num limits the maximum number of simultaneous connections to the proxied server.Default 0.
max_fails=numsets the number of unsuccessful attempts to communicate with the server that should happen in the duration set by the fail_timeout parameter to consider the server unavailable.Default 1.
fail_timeout=timethe period of time the server will be considered unavailable.Default 10.
backupmarks the server as a backup server. Connections to the backup server will be passed when the primary servers are unavailable.
down marks the server as permanently unavailable.

1.5.3. hash

Directives

Syntax: hash key [consistent];
Default: Close
Context: upstream

Instroduction

Specifies a load balancing method for a server group where the client-server mapping is based on the hashed key value. The key can contain text, variables. such as: hash $remote_addr

1.5.4. least_conn

Directives

Syntax: least_conn
Default: Close
Context: upstream

Instroduction

Specifies that a group should use a load balancing method where a connection is passed to the server with the least number of active connections, taking into account weights of servers.

2. Example

2.1. LB for Kubernets Master

In my kubernetes cluster, there has three master: 10.4.7.51:6443-10.4.7.53:6443. Nginx LB deploy on 10.4.7.59.The configruation:

2.1.1. Nginx Config

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 65535;
}
stream {
    log_format proxy '$time_local|$remote_addr|$upstream_addr|$protocol|$status|'
                     '$session_time|$upstream_connect_time|$bytes_sent|$bytes_received|'
                     '$upstream_bytes_sent|$upstream_bytes_received' ;
    access_log /var/log/nginx/access_stream.log proxy;
    error_log  /var/log/nginx/error_stream.log warn;
    include /etc/nginx/conf.d/stream/*.conf;
}

upstream kube-apiserver {
    server 10.4.7.51:6443 ;
    server 10.4.7.52:6443 ;
    server 10.4.7.53:6443 ;
}
server {
    listen 10.4.7.59:6443 backlog=65535 so_keepalive=on;
    allow 10.4.7.0/24;
    allow 192.168.0.0/16;
    allow 172.19.0.0/16;
    allow 172.24.0.0/16;
    deny all;
    proxy_connect_timeout 3s;
    proxy_next_upstream on;
    proxy_next_upstream_timeout 5;
    proxy_next_upstream_tries 1;
    proxy_pass kube-apiserver;
    access_log /var/log/nginx/kube-apiserver.log proxy;
}

2.1.2. Test

[root@centos-7-59 nginx]# cat /var/log/nginx/kube-apiserver.log
09/May/2021:14:57:16 +0800|10.4.7.1|10.4.7.51:6443|TCP|200|0.282|0.000|46393|4323|4323|46393
09/May/2021:14:57:17 +0800|10.4.7.1|10.4.7.51:6443|TCP|200|0.014|0.002|13935|1776|1776|13935
09/May/2021:14:57:30 +0800|10.4.7.1|10.4.7.52:6443|TCP|502|3.004|-|0|0|0|0
09/May/2021:14:57:34 +0800|10.4.7.1|10.4.7.52:6443|TCP|502|3.003|-|0|0|0|0
09/May/2021:14:57:35 +0800|10.4.7.1|10.4.7.53:6443|TCP|200|0.012|0.000|13936|1776|1776|13936
09/May/2021:14:57:37 +0800|10.4.7.1|10.4.7.53:6443|TCP|200|0.014|0.000|13936|1776|1776|13936
09/May/2021:14:58:27 +0800|10.4.7.51|10.4.7.53:6443|TCP|200|0.013|0.000|1546|1408|1408|1546

2.2. Proxy To HTTP Server

When Nginx proxy TCP requset to backend server, the request $remote_ip will be change to nginx IP.So we need add client to TCP request.

2.2.1. Nginx Config

2.2.1.1. L4 Proxy Server

stream {
    log_format proxy '$time_local|$remote_addr|$upstream_addr|$protocol|$status|'
                     '$session_time|$upstream_connect_time|$bytes_sent|$bytes_received|'
                     '$upstream_bytes_sent|$upstream_bytes_received' ;
    access_log /var/log/nginx/stream_access.log proxy ;
    error_log /var/log/nginx/stream_error.log info;
    include /etc/nginx/stream/*.conf;
}

server {
    listen 80;
    proxy_protocol on;
    proxy_pass 10.4.7.82:8080;
}

2.2.1.2. Web Server

http {
    log_format access '$time_local|$http_x_real_ip|$remote_addr|$http_x_forwarded_for|$upstream_addr|'
                      '$request_method|$server_protocol|$host|$request_uri|$http_referer|$http_user_agent|'
                      '$proxy_host|$status' ;
    access_log  /var/log/nginx/access.log  access;
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    include /etc/nginx/conf.d/*.conf;
    server {
        # the 80 port to accept HTTP request direct requset
        # the 8080 port to accept proxy protcol request
        listen       80 default_server;
        listen       8080 proxy_protocol;
        set_real_ip_from 10.4.7.81;
        real_ip_header proxy_protocol;
        location / {
            return 200 "hello world\r\n";
        }
    }
}

2.2.2. Test

# the first log: client direct access webserver by http://10.4.7.82
# the second log: client access proxy server by http://10.4.7.81
[root@centos-82 ~]# tail /var/log/nginx/access.log
09/May/2021:15:41:53 +0800|-|10.4.7.1|-|-|GET|HTTP/1.1|10.4.7.82|/|-|curl/7.68.0|-|200
09/May/2021:15:41:55 +0800|-|10.4.7.1|-|-|GET|HTTP/1.1|10.4.7.81|/|-|curl/7.68.0|-|200

# when you direct access 10.4.7.82:8080, nginx will raise error:
[root@centos-82 ~]# cat /var/log/nginx/error.log
2021/05/09 15:42:51 [error] 1672#0: *8 broken header: "GET / HTTP/1.1
Host: 10.4.7.82:8080
User-Agent: curl/7.68.0
Accept: */*
" while reading PROXY protocol, client: 10.4.7.1, server: 0.0.0.0:8080

Nginx 学习笔记

04-3-L4负载均衡

1. L4 LB

1.1. ngx_stream_core_module

1.1.1. stream

1.1.2. server

1.1.3. listen

1.1.4. preread_buffer_size

1.1.5. preread_timeout

1.1.6. resolver

1.1.7. preread_timeout

1.2. ngx_stream_proxy_module

1.2.1. proxy_pass

1.2.2. proxy_protocol

1.2.3. proxy_buffer_size

1.2.4. proxy_connect_timeout

1.2.5. proxy_download_rate

1.2.6. proxy_next_upstream

1.2.6. proxy_next_upstream

1.2.6. proxy_next_upstream_tries

1.3. ngx_stream_realip_module

1.3.1. set_real_ip_from

1.4. ngx_stream_log_module

1.4.1. log_format

1.4.2. access_log

1.5. ngx_stream_upstream_module

1.5.1. upstream

1.5.2. server

1.5.3. hash

1.5.4. least_conn

2. Example

2.1. LB for Kubernets Master

2.1.1. Nginx Config

2.1.2. Test

2.2. Proxy To HTTP Server

2.2.1. Nginx Config

2.2.1.1. L4 Proxy Server

2.2.1.2. Web Server

2.2.2. Test