002 - Claim Check Pattern
ADR Metadata ACCEPTED
Date
2024-01-01
Deciders
Architecture Team
Context
Section titled “Context”The system processes payloads that can contain large base64 images or extensive metadata (e.g., OCR results). AWS services like SQS (256KB limit) and DynamoDB (400KB item limit) have hard constraints on message/item size. Passing large payloads through the messaging system creates bottlenecks and can exceed these limits.
Decision
Section titled “Decision”We will implement the Claim Check Pattern.
- Storage: Large payloads (>400KB or a configured threshold) are stored in Amazon S3.
- Reference: The message passed between services (SQS, DynamoDB) contains only a reference (URI/Key) to the S3 object—the “Claim Check”.
- Retrieval: The consumer service uses the claim check to retrieve the full payload from S3 only when needed.
Consequences
Section titled “Consequences” ✓ Positive Consequences
- Scalability: We are not limited by message broker size constraints.
- Performance: Messaging infrastructure remains lightweight and fast.
- Cost: S3 is cheaper for storage than DynamoDB or high-throughput SQS for large data.
✗ Negative Consequences
- Latency: Extra round-trip time to fetch data from S3.
- Complexity: Services need logic to handle both “inline” data (small) and “claim check” data (large), plus S3 retention management (Lifecycle Policies).