Nginx: connection reset by peer while reading upstream

Nginx logs "connection reset by peer while reading upstream" when the upstream backend server closes the TCP connection unexpectedly before Nginx finishes reading the response.

The full error appears in the Nginx error log as:

failed (104: Connection reset by peer) while reading upstream

When Nginx Produces This Error

Nginx produces the "connection reset by peer" error when the upstream server sends a TCP RST (reset) packet while Nginx is reading the response body or headers. This interrupts the data transfer and causes Nginx to return a 502 Bad Gateway to the client.

Nginx also logs this error when a reverse proxy connection is reused through keepalive, but the upstream server closes the idle connection before Nginx sends the next request. The upstream server's keepalive timeout may be shorter than the Nginx keepalive timeout, creating a race condition.

What Causes the "Connection Reset by Peer" Error in Nginx

Nginx receives a connection reset when the upstream server process crashes mid-response.A segfault, out-of-memory kill, or unhandled exception in the backend application causes the operating system to send a TCP RST to all open connections. Nginx receives this reset while waiting for response data.

Nginx receives a connection reset when the upstream server's keepalive timeout is shorter than the Nginx keepalive timeout.Nginx reuses idle upstream connections to reduce latency. If the upstream server closes the connection after its own keepalive timeout, Nginx discovers the closed connection only when it attempts to send the next request.

Nginx receives a connection reset when a firewall or load balancer terminates idle connections.Network devices between Nginx and the upstream server may drop connections that have been idle for too long. The upstream server is unaware of the dropped connection and continues normally until the next packet triggers a reset.

How to Fix the "Connection Reset by Peer" Error in Nginx

1. Enable proxy_socket_keepalive in the Nginx location block. This directive enables TCP keepalive probes on the connection between Nginx and the upstream server. Keepalive probes detect dead connections before Nginx tries to send a request on them.

server {
    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_socket_keepalive on;
    }
}

2. Set the proxy_next_upstream directive to retry on connection resets. Nginx can forward the request to another upstream server in the group when the first server resets the connection.

upstream backend {
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_next_upstream error timeout http_502;
    }
}

3. Align the Nginx keepalive timeout with the upstream server's keepalive timeout. Set the Nginx keepalive_timeout for upstream connections to a value lower than the upstream server's timeout. This ensures Nginx closes idle connections before the upstream server does.

upstream backend {
    server 127.0.0.1:3000;
    keepalive 32;
    keepalive_timeout 60s;
}

4. Test the configuration and reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

How to Verify the Fix

Nginx stops logging "connection reset by peer" errors when upstream connections remain stable. Monitor the error log after applying the fix:

tail -f /var/log/nginx/error.log | grep "reset by peer"

Send sustained traffic to the server and confirm that no new reset errors appear. A load testing tool such as ab (Apache Bench) or wrk can generate enough requests to test connection reuse behavior.

Edge Cases and Variations

Nginx returns connection reset errors intermittently under high traffic.The upstream server may reach its maximum connection limit and reject new connections with a reset. Increase the upstream server's connection limit or add more backend servers to the upstream group.

Nginx returns connection reset errors for large response bodies only.The upstream server may terminate the connection when the response exceeds a size threshold. Check the upstream application's response size limits and Nginx's proxy_buffer_size and proxy_buffers directives.

Related Nginx Errors

Nginx: 502 Bad Gateway-- The 502 error is the client-facing result of a connection reset. The error log identifies whether the cause was a reset, a timeout, or a refused connection.

Nginx: 504 Gateway Timeout-- A 504 indicates the upstream did not respond in time. A connection reset indicates the upstream actively closed the connection.