← Back to Work

Data Engineering · 2024 · Taqtful

Furniture Clinic

30,000+ Amazon orders/month through automated pipelines

Role: Backend Developer

Node.jsMongoDBPower BIAmazon SP APIAdvertising API

30k+

Orders / Month

Processed automatically

-50%

Reporting Overhead

Average reduction

15 min

Report Refresh

Down from manual hours

0

Manual Effort

Order tracking eliminated

The Challenge

  • —Processing and storing 30,000+ Amazon orders monthly with high accuracy and minimal manual effort
  • —Integrating Amazon Seller Central and Advertising API data into a unified pipeline
  • —Delivering real-time business insights through Power BI dashboards connected to a live MongoDB backend
  • —Automating daily data sync jobs to keep dashboards current without manual intervention

Process

  • 01Designed scalable MongoDB schema optimized for high-volume order storage and client-specific data segmentation
  • 02Integrated Amazon Seller Central and Advertising API using Node.js with automated daily sync jobs
  • 03Built data transformation pipelines to normalize API responses into consistent MongoDB documents
  • 04Connected MongoDB to Power BI using a live data connector for real-time dashboard updates

The Solution

  • ✓Automated pipeline processing 30,000+ Amazon orders per month with high accuracy
  • ✓Integrated Seller Central and Advertising API data into unified daily reporting
  • ✓Power BI dashboards delivering real-time business insights and performance analytics
  • ✓Reduced reporting overhead by 40 to 60% by eliminating manual data collection

Architecture

1

Amazon SP API Ingestion

Node.js service handles SP API authentication, rate limiting, and pagination. Orders, refunds, and inventory changes are captured via polling with idempotent upsert into MongoDB.

2

Normalisation & Storage

Raw SP API payloads are transformed into a consistent schema before storage. MongoDB's flexible schema allowed rapid iteration on the data model as reporting requirements evolved.

3

Power BI Dashboards

Direct query mode connects Power BI to MongoDB Atlas. Finance and ops teams have live dashboards for revenue by SKU, fulfilment SLAs, and return rates — no CSV exports needed.

Takeaways

  • →Gained deep experience with large-scale data pipeline design and Amazon API integration
  • →Built reusable pipeline patterns later adopted across other projects at Qbatch
  • →Learned the importance of schema design for analytics-heavy workloads