Intermediate 10 min FTP 421

421 Service Unavailable — Connection Limit Reached

Triệu chứng

- New FTP connections are refused with "421 There are too many connections from your internet address"
- "421 Maximum number of connections exceeded" on the first connect attempt
- Existing long-running FTP sessions continue to work while new ones are blocked
- The error is intermittent and correlates with automated scripts or batch jobs that open many parallel FTP connections
- `ss -tn | grep :21` on the server shows many connections in ESTABLISHED or TIME_WAIT state from the same source IP

Nguyên nhân gốc rễ

  • vsftpd `max_per_ip` limit reached — the server enforces a maximum number of simultaneous connections from a single IP address (default 0 = unlimited, but hosting providers often set it to 2–5)
  • Automated scripts opening many parallel FTP connections for batch uploads without explicitly calling QUIT to close sessions afterward
  • FTP client not properly closing data connections after each transfer, leaving zombie sessions that count against the per-IP limit until they time out
  • vsftpd `max_clients` global limit reached — total connections across all IPs has hit the server-wide ceiling
  • OS-level `ulimit -n` (open file descriptors) reached — each FTP control and data connection consumes a file descriptor, and a low limit blocks new ones

Chẩn đoán

**Step 1 — Count active connections from your IP**

```bash
# On the server:
ss -tn sport = :21 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
# Each line: <count> <source_IP>
# High count for one IP confirms per-IP limit is being hit
```

**Step 2 — Check vsftpd connection limits**

```bash
grep -E 'max_clients|max_per_ip' /etc/vsftpd.conf
# Default if not set:
# max_clients=0 (unlimited)
# max_per_ip=0 (unlimited)
# Hosting providers may set max_per_ip=3 or similar
```

**Step 3 — Check the vsftpd log for 421 events**

```bash
sudo grep '421' /var/log/vsftpd.log | tail -20
# Look for: FAIL LOGIN or connection limit messages
```

**Step 4 — Check OS file descriptor limits**

```bash
# Current soft limit for the vsftpd process:
sudo cat /proc/$(pgrep vsftpd | head -1)/limits | grep 'open files'

# System-wide open FDs:
cat /proc/sys/fs/file-nr
# Format: <used> <unused> <max>
```

**Step 5 — Identify connection leaks in scripts**

```bash
# Watch connections in real time:
watch -n1 'ss -tn sport = :21 | wc -l'
# Count should drop after each batch job finishes;
# if it keeps growing, there is a leak
```

Giải quyết

**Fix 1 — Increase vsftpd connection limits**

```ini
# /etc/vsftpd.conf
max_clients=100
max_per_ip=10
```

```bash
sudo systemctl restart vsftpd
```

**Fix 2 — Fix connection leaks in Python scripts**

```python
import ftplib

# Wrong: connection left open after exception
def upload_file_bad(path: str) -> None:
ftp = ftplib.FTP('ftp.example.com', 'user', 'pass')
ftp.storbinary(f'STOR {path}', open(path, 'rb'))
# No ftp.quit() — leak!

# Correct: use context manager (Python 3.11+ natively, or wrap manually)
def upload_file_good(path: str) -> None:
with ftplib.FTP('ftp.example.com') as ftp:
ftp.login('user', 'pass')
with open(path, 'rb') as f:
ftp.storbinary(f'STOR {path}', f)
# ftp.quit() called automatically on __exit__
```

**Fix 3 — Serialize parallel FTP uploads**

```python
# Instead of opening N connections for N files, reuse one connection:
def upload_batch(files: list[str]) -> None:
with ftplib.FTP('ftp.example.com') as ftp:
ftp.login('user', 'pass')
for path in files:
with open(path, 'rb') as f:
ftp.storbinary(f'STOR {path}', f)
```

**Fix 4 — Raise OS file descriptor limits**

```bash
# /etc/security/limits.conf
vsftpd soft nofile 65536
vsftpd hard nofile 65536

# Or in the systemd unit override:
# /etc/systemd/system/vsftpd.service.d/limits.conf
[Service]
LimitNOFILE=65536
```

Phòng ngừa

- Always use a context manager (`with ftplib.FTP(...) as ftp`) or explicit `ftp.quit()` in finally blocks to ensure connections are released, especially in scripts that run on a schedule
- Reuse a single FTP connection for batch operations rather than opening one connection per file — it is faster and avoids hitting per-IP limits
- Set conservative but realistic `max_per_ip` limits (e.g., 10) and monitor connection counts with `ss -tn sport = :21` in your observability stack
- Prefer SFTP with connection pooling libraries (Paramiko SSHClient) for automated batch uploads — the SSH keepalive mechanism prevents zombie sessions
- Add connection count metrics to your server monitoring dashboard and alert when per-IP connections approach the limit

Mã trạng thái liên quan

Thuật ngữ liên quan