431 Request Header Fields Too Large — Cookie Overflow

- Certain users receive 431 Request Header Fields Too Large errors while other users on the same site are unaffected
- The affected users tend to be long-time users or those who have visited many marketing campaigns that set A/B testing cookies
- Browser DevTools Application tab shows dozens of cookies set for the domain, with total size exceeding 8KB
- Clearing cookies for the domain resolves the 431 immediately but the problem recurs as cookies accumulate again over days or weeks
- The Nginx error log shows `request_line: large client headers buffer` or the 431 appears as a raw Nginx error page rather than your app's error page

**Step 1 — Measure total cookie size in DevTools**

Open DevTools → Network tab → click any failing request → Request Headers → find the Cookie header and measure its length:
```javascript
// Run in DevTools console to measure cookie size:
const cookieSize = new TextEncoder().encode(document.cookie).length;
console.log(`Cookie size: ${cookieSize} bytes (limit: ~8192)`);
console.log(`Cookie count: ${document.cookie.split(';').length}`);
document.cookie.split(';').forEach(c => {
const [name, val] = c.trim().split('=');
console.log(`${name}: ${(val || '').length} bytes`);
});
```

**Step 2 — Reproduce with curl by sending a large Cookie header**

```bash
# Simulate a user with large cookies:
LARGE_COOKIE=$(python3 -c "print('test=' + 'x' * 8000)")
curl -v https://example.com/ -H "Cookie: $LARGE_COOKIE"
# < HTTP/1.1 431 Request Header Fields Too Large
# Confirms the threshold and which layer returns 431
```

**Step 3 — Check Nginx header buffer configuration**

```bash
ssh server
grep -r 'large_client_header_buffers' /etc/nginx/
# Default: large_client_header_buffers 4 8k;
# Total: 4 × 8192 = 32768 bytes, but any single header must fit in 8192
```

**Step 4 — Audit which cookies are set and by whom**

```bash
# Use browser DevTools → Application → Cookies to list all cookies:
# Note which tool owns each cookie:
# _ga, _gid → Google Analytics (usually small)
# _fbp → Facebook Pixel
# ab_test_* → A/B testing tools (can be very large)
# session → your app session
```

**Fix 1: Increase Nginx header buffer (temporary — fix root cause too)**

```nginx
# /etc/nginx/nginx.conf (or site config)
http {
# Increase buffer for large Cookie headers
large_client_header_buffers 4 32k;
client_header_buffer_size 4k;
}
```
```bash
sudo nginx -t && sudo systemctl reload nginx
```

**Fix 2: Consolidate application cookies into one JSON cookie**

```python
import json
from django.http import HttpResponse

def set_user_preferences(response: HttpResponse, prefs: dict) -> None:
# BAD: One cookie per preference
# response.set_cookie('theme', 'dark')
# response.set_cookie('language', 'en')
# response.set_cookie('notifications', 'on')

# GOOD: Single JSON cookie for all preferences
response.set_cookie(
'user_prefs',
json.dumps(prefs),
max_age=365 * 24 * 60 * 60,
httponly=True,
samesite='Lax',
)
```

**Fix 3: Scope third-party cookies correctly to limit spreading**

```javascript
// Set cookies with restrictive Path to avoid sending on every request:
document.cookie = 'ab_test_data=...; Path=/experiments; Max-Age=86400; SameSite=Lax';
// This cookie is sent ONLY to /experiments/* URLs, not all requests
```

**Fix 4: Store large data server-side, send only a session ID in cookie**

```python
# Use server-side session storage (Django default: DB or cache backend)
# settings.py
SESSION_ENGINE = 'django.contrib.sessions.backends.cache' # Redis/Memcache
SESSION_COOKIE_AGE = 1209600 # 2 weeks
# Cookie only stores: sessionid=abc123 (small, constant size)
# All actual data lives in Redis, keyed by session ID
```

**Fix 5: Clean up legacy cookies in the response**

```python
# Identify and delete cookies that are no longer needed:
DEPRECATED_COOKIES = ['old_ab_test', 'legacy_pref_v1', 'analytics_v1']

class CookieCleanupMiddleware:
def __call__(self, request, get_response):
response = get_response(request)
for cookie_name in DEPRECATED_COOKIES:
if cookie_name in request.COOKIES:
response.delete_cookie(cookie_name)
return response
```

- **Conduct a cookie audit at the start of every analytics/marketing integration** — document what each cookie stores, its size, its expiry, and its domain scope before it reaches production
- **Set a cookie budget** (e.g., under 4KB total per domain) as a performance and reliability constraint and enforce it in code review and CI with a cookie size check
- **Use server-side session storage instead of client-side cookies** for application state — store only a small session ID token in the cookie and keep all payload in Redis or a database
- **Scope third-party cookies to specific paths** using the `Path` attribute to prevent analytics or A/B testing cookies from being sent on every API request to the same domain