Describe the bug
When connecting to nginx with https using http2 the following error appears in the logs intermittently and results in client facing 502 errors;
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
To Reproduce
Steps to reproduce the behavior:
- Install nginx container (such as nginx:alpine) and cloudflared:latest
- Use an nginx config file with http2 on (Using a short timeout or request limit triggers GOAWAY frames faster);
server {
listen 443 ssl default_server;
http2 on;
keepalive_timeout 1s;
keepalive_requests 1;
}
- Use a cloudflared config file with http2 on;
ingress:
- service: https://nginx:443
originRequest:
http2Origin: true
- Start sending requests through cloudflared and view logs. You will see the error;
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
- Setting
http2Origin: false resolves the issue.
Expected behavior
Cloudflared should see the GOAWAY frame and retry the request.
Environment and versions
- Cloudflared Version: 2024.6.0
Additional context
Appears to be an issue with http.Request missing a no-op GetBody() method.
The GOAWAY frame (type=0x7) is used to initiate graceful shutdown of a connection by a server. Nginx sends GOAWAY frames when a keepalive connection needs to end (e.g. has hit keepalive_timeout or keepalive_requests limits). This happens regularly with default nginx http2 configs.
Describe the bug
When connecting to nginx with https using http2 the following error appears in the logs intermittently and results in client facing 502 errors;
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this errorTo Reproduce
Steps to reproduce the behavior:
http2Origin: falseresolves the issue.Expected behavior
Cloudflared should see the GOAWAY frame and retry the request.
Environment and versions
Additional context
Appears to be an issue with
http.Requestmissing a no-opGetBody()method.The GOAWAY frame (type=0x7) is used to initiate graceful shutdown of a connection by a server. Nginx sends GOAWAY frames when a keepalive connection needs to end (e.g. has hit
keepalive_timeoutorkeepalive_requestslimits). This happens regularly with default nginx http2 configs.