502 Bad Gateway in Nginx Reverse Proxy
Gejala
- Browser displays "502 Bad Gateway" error page with Nginx branding
- Nginx error log shows `connect() failed (111: Connection refused) while connecting to upstream`
- Error appears intermittently under load but returns to normal during low-traffic periods
- API clients receive HTTP 502 with an HTML error body instead of JSON
- Health check endpoints return 502 while direct backend access (bypassing Nginx) works fine
- Nginx error log shows `connect() failed (111: Connection refused) while connecting to upstream`
- Error appears intermittently under load but returns to normal during low-traffic periods
- API clients receive HTTP 502 with an HTML error body instead of JSON
- Health check endpoints return 502 while direct backend access (bypassing Nginx) works fine
Penyebab Utama
- Upstream application server (Gunicorn, uWSGI, Node.js) is down or crashed and not listening on its configured port
- Incorrect `proxy_pass` target in Nginx config — wrong port, socket path, or hostname that doesn't resolve
- Upstream process ran out of worker slots and is refusing new connections during a traffic spike
- Unix socket file referenced in `proxy_pass` was deleted after app restart and not recreated
- Firewall or `iptables` rule blocking Nginx from reaching the upstream on localhost or a private IP
Diagnosis
**Step 1: Check if the upstream process is running**
```bash
# For Gunicorn on a TCP port
ss -tlnp | grep 8000
# For a Unix socket
ls -la /run/gunicorn/gunicorn.sock
sudo systemctl status gunicorn
```
**Step 2: Read the Nginx error log**
```bash
sudo tail -n 50 /var/log/nginx/error.log
# Look for: 'connect() failed', 'no live upstreams', 'recv() failed'
```
**Step 3: Validate the Nginx proxy_pass target**
```bash
sudo nginx -T | grep proxy_pass
# Manually curl the upstream to confirm it responds
curl -v http://127.0.0.1:8000/
```
**Step 4: Check upstream application logs**
```bash
sudo journalctl -u gunicorn --since '5 min ago' --no-pager
```
**Step 5: Test Nginx config and reload**
```bash
sudo nginx -t && sudo systemctl reload nginx
```
```bash
# For Gunicorn on a TCP port
ss -tlnp | grep 8000
# For a Unix socket
ls -la /run/gunicorn/gunicorn.sock
sudo systemctl status gunicorn
```
**Step 2: Read the Nginx error log**
```bash
sudo tail -n 50 /var/log/nginx/error.log
# Look for: 'connect() failed', 'no live upstreams', 'recv() failed'
```
**Step 3: Validate the Nginx proxy_pass target**
```bash
sudo nginx -T | grep proxy_pass
# Manually curl the upstream to confirm it responds
curl -v http://127.0.0.1:8000/
```
**Step 4: Check upstream application logs**
```bash
sudo journalctl -u gunicorn --since '5 min ago' --no-pager
```
**Step 5: Test Nginx config and reload**
```bash
sudo nginx -t && sudo systemctl reload nginx
```
Resolusi
**Fix 1: Restart the upstream process**
```bash
sudo systemctl restart gunicorn
# Verify it's listening
ss -tlnp | grep 8000
```
**Fix 2: Correct a wrong proxy_pass port**
```nginx
# /etc/nginx/sites-available/myapp
location / {
proxy_pass http://127.0.0.1:8000; # match your app port
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
```
**Fix 3: Increase Gunicorn workers to handle load**
```bash
# /etc/systemd/system/gunicorn.service
ExecStart=/srv/app/.venv/bin/gunicorn \
--workers 4 \
--bind 127.0.0.1:8000 \
myapp.wsgi:application
```
**Fix 4: Add upstream keepalive and timeout tuning**
```nginx
upstream backend {
server 127.0.0.1:8000;
keepalive 32;
}
server {
location / {
proxy_pass http://backend;
proxy_connect_timeout 5s;
proxy_read_timeout 60s;
}
}
```
```bash
sudo systemctl restart gunicorn
# Verify it's listening
ss -tlnp | grep 8000
```
**Fix 2: Correct a wrong proxy_pass port**
```nginx
# /etc/nginx/sites-available/myapp
location / {
proxy_pass http://127.0.0.1:8000; # match your app port
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
```
**Fix 3: Increase Gunicorn workers to handle load**
```bash
# /etc/systemd/system/gunicorn.service
ExecStart=/srv/app/.venv/bin/gunicorn \
--workers 4 \
--bind 127.0.0.1:8000 \
myapp.wsgi:application
```
**Fix 4: Add upstream keepalive and timeout tuning**
```nginx
upstream backend {
server 127.0.0.1:8000;
keepalive 32;
}
server {
location / {
proxy_pass http://backend;
proxy_connect_timeout 5s;
proxy_read_timeout 60s;
}
}
```
Pencegahan
- **Systemd dependency**: Use `After=gunicorn.service` and `Requires=gunicorn.service` in the Nginx unit so it won't start before the upstream is ready
- **Process supervision**: Run upstream apps under systemd with `Restart=always` so crashes auto-recover
- **Upstream health checks**: If using Nginx Plus or a load balancer, enable active health checks on the upstream group
- **Capacity planning**: Set worker count to `(2 × CPU cores) + 1` as a baseline, then load-test and tune
- **Log rotation**: Ensure log files don't grow to fill disk — a full disk can prevent Gunicorn from starting
- **Process supervision**: Run upstream apps under systemd with `Restart=always` so crashes auto-recover
- **Upstream health checks**: If using Nginx Plus or a load balancer, enable active health checks on the upstream group
- **Capacity planning**: Set worker count to `(2 × CPU cores) + 1` as a baseline, then load-test and tune
- **Log rotation**: Ensure log files don't grow to fill disk — a full disk can prevent Gunicorn from starting