RESOURCE_EXHAUSTED — gRPC Message Size Limit Exceeded
Síntomas
- gRPC calls fail with `StatusCode.RESOURCE_EXHAUSTED` and message: "Received message larger than max (4194304 vs. 4194304)"
- The error only appears with large payloads — small requests succeed
- Server log: `RESOURCE_EXHAUSTED: Received message larger than max (X vs 4194304)`
- Error is consistent for a specific RPC endpoint that returns large lists or binary blobs
- Works in development with small datasets but fails in production with full data
- The error only appears with large payloads — small requests succeed
- Server log: `RESOURCE_EXHAUSTED: Received message larger than max (X vs 4194304)`
- Error is consistent for a specific RPC endpoint that returns large lists or binary blobs
- Works in development with small datasets but fails in production with full data
Causas raíz
- gRPC default max message size is 4MB (4,194,304 bytes) — exceeded by a large response
- List RPC returning all records without pagination — size grows unbounded with data
- Binary blob (image, file) embedded directly in a Protobuf `bytes` field
- Both client and server must be configured with matching max sizes — client limit applies to received messages, server limit to received requests
- Nested Protobuf messages accumulating into a payload larger than expected
Diagnóstico
**Step 1 — Identify the failing RPC and payload size**
```python
# Python gRPC — catch RESOURCE_EXHAUSTED and log the size
import grpc
try:
response = stub.ListItems(request)
except grpc.RpcError as e:
if e.code() == grpc.StatusCode.RESOURCE_EXHAUSTED:
print(f'Message too large: {e.details()}')
```
**Step 2 — Measure the actual serialized size**
```python
# Check the response size before sending
response = build_response()
size_bytes = response.ByteSize()
print(f'Response size: {size_bytes / 1024 / 1024:.2f} MB')
```
**Step 3 — Check current configured limits**
```python
# Python gRPC — server options
server = grpc.server(
futures.ThreadPoolExecutor(max_workers=10),
options=[
('grpc.max_receive_message_length', -1), # -1 = unlimited
('grpc.max_send_message_length', -1),
]
)
# Python gRPC — channel (client) options
channel = grpc.insecure_channel(
'localhost:50051',
options=[('grpc.max_receive_message_length', 16 * 1024 * 1024)]
)
```
```python
# Python gRPC — catch RESOURCE_EXHAUSTED and log the size
import grpc
try:
response = stub.ListItems(request)
except grpc.RpcError as e:
if e.code() == grpc.StatusCode.RESOURCE_EXHAUSTED:
print(f'Message too large: {e.details()}')
```
**Step 2 — Measure the actual serialized size**
```python
# Check the response size before sending
response = build_response()
size_bytes = response.ByteSize()
print(f'Response size: {size_bytes / 1024 / 1024:.2f} MB')
```
**Step 3 — Check current configured limits**
```python
# Python gRPC — server options
server = grpc.server(
futures.ThreadPoolExecutor(max_workers=10),
options=[
('grpc.max_receive_message_length', -1), # -1 = unlimited
('grpc.max_send_message_length', -1),
]
)
# Python gRPC — channel (client) options
channel = grpc.insecure_channel(
'localhost:50051',
options=[('grpc.max_receive_message_length', 16 * 1024 * 1024)]
)
```
Resolución
**Fix 1 — Increase the message size limit (quick fix)**
```python
# Server — increase max receive message to 16MB
options = [
('grpc.max_receive_message_length', 16 * 1024 * 1024),
('grpc.max_send_message_length', 16 * 1024 * 1024),
]
server = grpc.server(executor, options=options)
# Client — must match server's send limit
channel = grpc.secure_channel(
'api.example.com:443', credentials,
options=[('grpc.max_receive_message_length', 16 * 1024 * 1024)]
)
```
**Fix 2 — Add pagination to list RPCs (correct long-term fix)**
```protobuf
// items.proto
message ListItemsRequest {
int32 page_size = 1; // max 100
string page_token = 2;
}
message ListItemsResponse {
repeated Item items = 1;
string next_page_token = 2; // empty = last page
}
```
**Fix 3 — Use server-side streaming for large responses**
```protobuf
// Instead of a large unary response:
rpc ListAllItems (ListItemsRequest) returns (stream Item) {}
```
```python
# Client side: iterate the stream
for item in stub.ListAllItems(request):
process(item)
```
**Fix 4 — Serve binary blobs via a URL, not inline Protobuf**
Return a signed URL pointing to S3 or GCS instead of embedding file bytes in a `bytes` field.
```python
# Server — increase max receive message to 16MB
options = [
('grpc.max_receive_message_length', 16 * 1024 * 1024),
('grpc.max_send_message_length', 16 * 1024 * 1024),
]
server = grpc.server(executor, options=options)
# Client — must match server's send limit
channel = grpc.secure_channel(
'api.example.com:443', credentials,
options=[('grpc.max_receive_message_length', 16 * 1024 * 1024)]
)
```
**Fix 2 — Add pagination to list RPCs (correct long-term fix)**
```protobuf
// items.proto
message ListItemsRequest {
int32 page_size = 1; // max 100
string page_token = 2;
}
message ListItemsResponse {
repeated Item items = 1;
string next_page_token = 2; // empty = last page
}
```
**Fix 3 — Use server-side streaming for large responses**
```protobuf
// Instead of a large unary response:
rpc ListAllItems (ListItemsRequest) returns (stream Item) {}
```
```python
# Client side: iterate the stream
for item in stub.ListAllItems(request):
process(item)
```
**Fix 4 — Serve binary blobs via a URL, not inline Protobuf**
Return a signed URL pointing to S3 or GCS instead of embedding file bytes in a `bytes` field.
Prevención
- Add pagination to all list RPCs from day one — set a maximum `page_size` (e.g., 100 items) enforced server-side
- Measure Protobuf message sizes in unit tests and add a size assertion to catch unexpected growth
- Never embed binary files in Protobuf messages — use object storage (S3, GCS) and return URLs instead
- Use server-side streaming for any response that may grow unbounded, such as exports or log queries
- Set matching `max_receive_message_length` on both client and server — mismatches cause confusing failures
- Measure Protobuf message sizes in unit tests and add a size assertion to catch unexpected growth
- Never embed binary files in Protobuf messages — use object storage (S3, GCS) and return URLs instead
- Use server-side streaming for any response that may grow unbounded, such as exports or log queries
- Set matching `max_receive_message_length` on both client and server — mismatches cause confusing failures