Pass Gallery migration
A resumable, distributed pipeline that moves 500k+ images per client between gallery platforms in under 24 hours. It preserves folder structure and metadata, and recovers from mid-run failure without restarting the whole job.
The problem
A client arrives already on PicTime, Pixieset, ShootProof, or Zenfolio, with hundreds of thousands of images and years of folders built around how they shoot. The job was to move all of it onto Pass Gallery (500k+ images for a single client) without losing the shape of it. Same galleries, same sets, same metadata. Nothing renamed, nothing flattened, nothing uploaded twice.
What existed before me was a person doing this by hand. It took days, and it didn't survive interruptions: a dropped connection or a provider throttling the work halfway through meant starting that client over from scratch. That last part is the one I cared about. Not “make it fast” so much as “make it so a failure near the end of a run isn't a disaster.”
The real constraint
So the constraint was never throughput. It was that the work had to be resumable. A run that copies most of a client's catalogue and then dies is worse than one that never started. By then I've spent the provider's patience and my own time, and if the only move left is to run it again from the top, I risk uploading everything a second time.
That reframed the whole thing. Every image had to be a unit of work I could account for on its own (done, not done, or failed). On failure, the run had to pick up from the last thing that worked, not the first.
The shape of the system
I built it as stages that don't trust each other. Ingest pulls a client's catalogue down from their old provider and drops units of work onto an SQS queue. A pool of Node.js workers on ECS takes them off and does the actual copying. Lambda handles the handoffs between stages. MongoDB holds the state (what's done, what's left), which is the piece that lets a dead run come back to life.
The decisions
Three decisions carried most of the weight. For each I've written down what I picked; the alternatives I weighed and why I dropped them are the ones I still owe this page.
Decouple ingest from upload with a queue
I put SQS between pulling images and uploading them so neither side could hold the other hostage. When a provider started throttling me, the workers didn't sit idle. They kept draining whatever was already queued. And a queue gives a failed unit somewhere obvious to go: back of the line, try again.
Hold job + checkpoint state in MongoDB
Nothing counts as done until it's written to MongoDB. That way recovery isn't special-case code I have to trust under pressure. It's a question I ask at startup: what's already finished, and what's left? The run continues from the answer.
Run workers on ECS, coordinate with Lambda
The copying itself is long and memory-hungry, not something I wanted to wedge inside a Lambda timeout. That runs on ECS containers. Lambda I kept for what it's actually good at: the short, event-shaped glue between stages.
The result
It does the thing it had to do. A client's 500k+ images move in under 24 hours instead of days, with the galleries, sets, and metadata intact on the other side. And when a run dies, it resumes from its last checkpoint. The cost of a failure is the work since that checkpoint, not the whole client.