1. 处理磁盘IO的方式

Nginx在做今天文件服务器和缓存服务器时，针对客户端请求的文件，需要频繁的从磁盘或者缓存中读取并发送给客户端。在这个过程中，根据文件的大小、文件的随机性、内存大小需要使用不同的磁盘IO方式来提高运行效率。
如果文件存在与缓存中，那么将缓存中的文件发送给套接字是非常迅速的。如果缓存中不存在这个文件，那么需要从磁盘中读取，这个过程又会涉及到文件系统缓存和用户空间缓存的两次拷贝问题。如果文件比较大，频繁的更新缓存也会浪费事件。Nginx针对这些场景，提供了 sendfile、directio、aio、io任务卸载多种方式。无论使用哪种方式，性能最好的无疑是将所有的文件都缓存到文件系统缓存中，但生产环境中往往没有这么大的内存可以使用。

image.png

image.png
07-5-I/O Directives - 图3

1.1. sendfile

在普通模式中，如果文件不在文件缓存中，那么应用程序先通过特权模式切换，将文件读取到文件系统缓存，然后再拷贝到用户空间中，最后再发送到socket缓冲区。
使用Linux系统提供的sendfile，可以实现将文件系统缓冲中的内容直接发送到socket套接字缓冲区，减少了一次文件拷贝，降低的IO的阻塞时间。

1.2. directio

在大文件的下载的场景中，如果每次都将非热点的大文件先读入文件系统缓存，那么会导致文件系统缓存中很多小文件缓存被释放，进而导致缓存命中率降低，造成更多的磁盘IO。
针对大文件，没必要将其放入缓存中，尤其是非热点文件，通过直接IO跳过文件系统缓存是最佳的选择。

1.3. AIO

AIO是Linux操作系统提供的一种异步IO接口(Linux native aio)，在文件读取方面的性能非常高，当Nginx调用aio接口时，不需要等待返回，交由操作系统来完成。

1.4. 线程池

虽然Linux提供了sendfile、directio、aio的方式来增强磁盘io的性能，但是总体来说，在大量io的场景中，无论采用哪种方式，都会影响工作进程本身的循环运行，进而影响后续队列的处理速度。为此Nginx提供了线程池的方式来卸载磁盘io的任务，将繁重的io任务交给线程池来完成，从而避免主进程(worker processes)阻塞。
线程池可以参考：https://www.infoq.cn/article/thread-pools-boost-performance-9x

2. 指令

2.1. sendfile

  • Directives

    1. Syntax: sendfile on|off ;
    2. Default: sendfile on ;
    3. Context: http,server,location,if in loaction

  • Instroduction

根据sendfile()的工作原理，这个指令针对静态资源服务器比较管用，尤其是小文件非常多的情况下，但是针对反向代理的情况，并不能提高性能。

2.2. sendfile_max_chunk

  • Directives

    1. Syntax: sendfile_max_chunk size ;
    2. Default: sendfile_max_chunk 0 ;
    3. Context: http,server,location

  • Instroduction

用于限制单次sendfile()系统调用的数据量，用于防止一个快速连阻塞工作进程。

2.3. directio

  • Directives

    1. Syntax: directio size|off ;
    2. Default: directio off
    3. Context: http,server,location

  • Instroduction

Enables the use of the O_DIRECT flag to open files.When reading files that are larger than or equal to the specified _size._If use directio,the sendfile will be disabled.

2.4. threads

  • Directives

    1. Syntax: thread_pool name threads=number [max_queue=number] ;
    2. Default: thread_pool default threads=32 max_queue=65535 ;
    3. Context: main

  • Instroduction

Defines named thread pools used for multi-threaded reading and sending of files without blocking worker processes.The directive appeared in version 1.7.11.
The threads parameter defines the number of threads in the pool.
In the event that all threads in the pool are busy, a new task will wait in the queue.
The max_queue parameter limits the number of tasks allowed to be waiting in the queue. By default, up to 65536 tasks can wait in the queue. When the queue overflows, the task is completed with an error.

2.5. aio

  • Directives

    1. Syntax: aio on|off|threads[=pool] ;
    2. Default: aio off
    3. Context: http,server,location

  • Instroduction

Enables or disables the use of asynchronous file I/O (AIO) on FreeBSD and Linux.On FreeBSD, AIO can be used starting from FreeBSD 4.3. Prior to FreeBSD 11.0.On Linux, AIO can be used starting from kernel version 2.6.22,but it is necessary to enabledirectio.
When both AIO and sendfile are enabled on Linux, AIO is used for files that are larger than or equal to the size specified in the directio directive, while sendfile is used for files of smaller sizes or when directio is disabled.
Finally, files can be read and sent using multi-threading (1.7.11), without blocking a worker process.
By default, multi-threading is disabled, it should be enabled with the —with-threads configuration parameter.
Currently, multi-threading is compatible only with the epoll, kqueue, and eventportmethods. Multi-threaded sending of files is only supported on Linux.

2.6. Files Cache

2.6.1. open_file_cache

  • Directives

    1. Syntax: open_file_cache off ;
    2.           open_file_cache max=number [inactive=time] ;
    3. Default: open_file_cache off ;
    4. Context: http,server,location

  • Instroduction

Sets a cache that store:

  • open file descriptors, their sizes and modification times;
  • information on existence of directories;
  • file lookup errors, such as “file not found”, “no read permission”, and so on.

Parameters:

  • off: disable the cache
  • max: sets max elements in the cache.Removes the least recently element from cache when the cache overflow.
  • inactive: sets inactive time that used for remove element from cache.Defaut 60s.

    2.6.2. open_file_cache_min_uses

  • Directives

    1. Syntax: open_file_cache_min_uses number ;
    2. Default: open_file_cache_min_uses 1 ;
    3. Context: http,server,location

  • Instroduction

Sets minumum number of file accesss during the inactive time that sets by open_file_cache.

2.6.3. open_file_cache_valid

  • Directives

    1. Syntax: open_file_cache_valid time ;
    2. Default: open_file_cache_valid 60s ;
    3. Context: http,server,location

  • Instroduction

Sets a time after which open_file_cache elements should be check.

2.6.4. open_file_cache_errors

  • Directives

    1. Syntax: open_file_cache_errors on|off ;
    2. Default: open_file_cache_errors on ;
    3. Context: http,server,location

  • Instroduction

Enable or disables caching file lookup by open_file_cache directive.

3. Example

3.1. Directio Test

3.1.1 Use Filesystem Cache

  1.     location /download/ {
  2.         limit_rate 2048k ;
  3.         sendfile on ;
  4.         # directio 10m ;
  5.         # aio threads ;
  6.     }

[root@centos-81 ~]# echo 3 > /proc/sys/vm/drop_caches
[root@centos-81 ~]# wget —no-check-certificate https://www.heyang.com/download/book.zip ## 47M
[root@centos-81 ~]# pcstat /opt/website/download/book.zip ## Cached all pages

  1. |--------------------------------+----------------+------------+-----------+---------|
  2. | Name                           | Size           | Pages      | Cached    | Percent |
  3. |--------------------------------+----------------+------------+-----------+---------|
  4. | /opt/website/download/book.zip | 48425357       | 11823      | 11823     | 100.000 |
  5. |--------------------------------+----------------+------------+-----------+---------|

3.1.2. Use Directio

  1.     location /download/ {
  2.         limit_rate 2048k ;
  3.         sendfile on ;
  4.         directio 10m ;
  5.         aio threads ;
  6.     }

[root@centos-81 ~]# echo 3 > /proc/sys/vm/drop_caches
[root@centos-81 ~]# wget —no-check-certificate https://www.heyang.com/download/book.zip
[root@centos-81 ~]# pcstat /opt/website/download/book.zip

  1. |--------------------------------+----------------+------------+-----------+---------|
  2. | Name                           | Size           | Pages      | Cached    | Percent |
  3. |--------------------------------+----------------+------------+-----------+---------|
  4. | /opt/website/download/book.zip | 48425357       | 11823      | 7         | 000.059 |
  5. |--------------------------------+----------------+------------+-----------+---------|

3.2. 性能测试

暂时没有做测试，可参考 https://www.infoq.cn/article/thread-pools-boost-performance-9x