Project Overview

This project involved architecting and building a production-grade connector that synchronises product catalogue data and order information bi-directionally between multiple Shopify stores and an external system via authenticated API integration. The scale of the operation — 10,000+ products, 50+ orders per minute at peak — required a deeply engineered, fault-tolerant backend architecture rather than a naive webhook-to-API approach.

Key Technical Challenges & Solutions

Rate-Limit-Safe API Orchestration

• Adaptive Rate Limiting: Implemented queue-based API orchestration using plimit for adaptive concurrency control, with retry/backoff strategies to stay within Shopify's and the external system's rate limits simultaneously.
• Per-Page Query Batching: Used Prisma-backed sequential processing with per-page batching via AWS ECS workers, ensuring no API threshold was breached regardless of catalogue size.
• Shopify Bulk Operations API: Leveraged Shopify's Bulk Operations API (GraphQL bulk queries and mutations) for large-scale product data reads and updates — the only efficient approach at this data volume.

Idempotent Webhook Handling

• Deduplication Logic: Shopify fires multiple product/update webhook events for a single merchant action. Built deduplication logic using a Prisma-backed event store to prevent redundant downstream API calls and maintain data consistency across all integrated systems.
• Idempotent Processing: Designed all sync operations to be fully idempotent — safe to re-process in case of failure or retry, with no risk of duplicate orders or corrupted product state.

Queue Architecture on AWS

• Deployed Laravel Jobs processed by SQS queues and Supervisor-managed ECS workers for reliable background job execution at scale.
• Processed high-volume webhook traffic (50+ orders/min) without blocking the main application thread.
• Built merchant-facing tools for monitoring sync status, managing product mappings, and viewing error logs with auto-retry controls.

Results

• Successfully synced 10,000+ products without a single rate-limit violation on either system.
• Eliminated duplicate processing through idempotent webhook deduplication.
• Reduced manual intervention by 100% — all sync operations automated end-to-end.