big request body cause send file error

[chanjarster]

use openresty as a http reverse proxy, upload a file about 1.1M go error describe below.

• image: openresty/openresty:jammy
• nginx version: openresty/1.25.3.2
• nginx.conf in /usr/local/openresty/nginx/conf/nginx.conf (open the debug log):

# nginx.conf  --  docker-openresty
#
# This file is installed to:
#   `/usr/local/openresty/nginx/conf/nginx.conf`
# and is the file loaded by nginx at startup,
# unless the user specifies otherwise.
#
# It tracks the upstream OpenResty's `nginx.conf`, but removes the `server`
# section and adds this directive:
#     `include /etc/nginx/conf.d/*.conf;`
#
# The `docker-openresty` file `nginx.vh.default.conf` is copied to
# `/etc/nginx/conf.d/default.conf`.  It contains the `server section
# of the upstream `nginx.conf`.
#
# See https://github.com/openresty/docker-openresty/blob/master/README.md#nginx-config-files
#

#user  nobody;
#worker_processes 1;

# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;

error_log  logs/error.log  debug;

#pid        logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    # See Move default writable paths to a dedicated directory (#119)
    # https://github.com/openresty/docker-openresty/issues/119
    client_body_temp_path /var/run/openresty/nginx-client-body;
    proxy_temp_path       /var/run/openresty/nginx-proxy;
    fastcgi_temp_path     /var/run/openresty/nginx-fastcgi;
    uwsgi_temp_path       /var/run/openresty/nginx-uwsgi;
    scgi_temp_path        /var/run/openresty/nginx-scgi;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    # Don't reveal OpenResty version to clients.
    # server_tokens off;
}

• provide custom config in /etc/nginx/conf.d/:

upstream up_servers {
    server echo:8080 max_conns=0 max_fails=0;
}
map $http_upgrade $connection_upgrade {
    default Upgrade;
    ''      close;
}
server {
    listen 80;
    client_max_body_size 32m;
    client_header_timeout 30s;
    client_header_buffer_size 4k;
    client_body_timeout 10s;
    large_client_header_buffers 32 64k;
    proxy_connect_timeout 15s;
    proxy_read_timeout 900s;
    proxy_send_timeout 10s;
    proxy_buffers 8 16k;
    proxy_buffer_size 16k;    
    location / {
        proxy_pass http://up_servers;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

• run container:

docker network create --attachable test-network
docker run -d \
                --name echo \
                --network test-network \
                -p 8888:8080 \
                inanimate/echo-server:latest
docker run -d  \
                --name openresty \
                --network test-network \
                -v $(pwd)/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf \
                -v $(pwd)/conf.d:/etc/nginx/conf.d \
                -p 8080:80 \
                "openresty/openresty:jammy"

• upload a file over 1M size:

curl --location --request POST 'http://localhost:8080/' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Accept: */*' \
--header 'Host: localhost:8080' \
--header 'Connection: keep-alive' \
--header 'Content-Type: multipart/form-data; boundary=--------------------------132917510733278170156063' \
--form 'foo=@"/path/to/file"'

• go error below:

2024/11/05 03:07:49 [notice] 1#1: using the "epoll" event method
2024/11/05 03:07:49 [notice] 1#1: openresty/1.25.3.2
2024/11/05 03:07:49 [notice] 1#1: built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 
2024/11/05 03:07:49 [notice] 1#1: OS: Linux 6.1.32-0-virt
2024/11/05 03:07:49 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/11/05 03:07:49 [notice] 1#1: start worker processes
2024/11/05 03:07:49 [notice] 1#1: start worker process 7
2024/11/05 03:08:04 [warn] 7#7: *1 a client request body is buffered to a temporary file /var/run/openresty/nginx-client-body/0000000001, client: 192.168.160.1, server: , request: "POST / HTTP/1.1", host: "localhost:8080"
2024/11/05 03:08:04 [alert] 7#7: *1 sendfile() failed (9: Bad file descriptor) while sending request to upstream, client: 192.168.160.1, server: , request: "POST / HTTP/1.1", upstream: "http://192.168.160.2:8080/", host: "localhost:8080"
192.168.160.1 - - [05/Nov/2024:03:08:04 +0000] "POST / HTTP/1.1" 200 25 "-" "Apifox/1.0.0 (https://apifox.com)"

Did some google and got this:

https://forum.nginx.org/read.php?21,118976,278732#msg-278732