Building Fiction Drafts: WordPress backups that resume where they stopped

I've been building Fiction Drafts, a WordPress backup plugin, and the thing that shaped its whole architecture is not a feature. It's the fact that PHP will kill your request in the middle of the work and not tell anyone.

Every backup plugin I've used on a site of real size eventually dies mid-run. Not because copying files is difficult — it isn't — but because a 40 GB uploads directory doesn't fit inside a 30-second max_execution_time, and the usual escapes are all closed on the hosting most WordPress sites actually run on. You can't raise the limit. You can't shell out to mysqldump. You can chunk the work with AJAX from the browser, which works right up until someone closes the tab.

So the constraint isn't the work. It's the request. And once you accept that no single request will finish the job, the design question stops being "how do we make this fast" and becomes "how does this survive being interrupted at any point."

A stage that can't be interrupted isn't a stage

A backup in Fiction Drafts is five stages — prepare, dump the database, scan the files, build the archive, finalize — and every one of them implements the same interface:

interface Stage {
    public function id(): string;
    public function label(): string;
    public function appliesTo( BackupJob $job ): bool;

    public function run(
        BackupJob $job,
        StageCursor $cursor,
        TimeBudget $budget
    ): StageResult;
}

The invariant lives in run(). It receives a time budget and must return before that budget is exhausted, handing back a cursor that describes exactly where to resume. In practice that means a loop over units of work that checks $budget->exhausted() at the top of each iteration and bails the moment it's true.

What comes back says which of the two things happened:

StageResult::complete( $processed, $total );
StageResult::incomplete( $cursor, $processed, $total );

complete is the only thing that advances the pipeline. incomplete means the stage ran out of clock, and the cursor is the position to hand back next time. StageCursor is deliberately dumb — a serializable bag of stage-defined scalars — because the runner should never need to understand what's inside it.

The budget is 20 seconds by default, which is deliberately well under a typical 30-second limit: the step still has to persist its cursor and schedule its successor after the clock runs out, and that work has to fit too. It also watches memory, not just time:

public function exhausted(): bool {
    if ( $this->elapsed() >= (float) $this->seconds ) {
        return true;
    }

    return $this->memoryExhausted();
}

That second condition exists because one pathological row or one enormous file should end a step cleanly, with a cursor written down, rather than fatally. A fatal error takes the cursor with it.

"Return when the budget is exhausted" is an infinite queue

The invariant above turned out to be half-written, and it took a test to notice.

Read it literally: a stage checks exhausted() at the top of each unit and returns as soon as it's true. Now hand a stage a budget that is already exhausted — a zero-second budget in a test, or a real one where setup ate the clock. The stage checks the budget before doing anything, sees it's spent, and returns incomplete with the cursor it was given. Unchanged. The runner sees an unfinished stage and schedules another step. Which does the same thing. Forever.

Nothing errors. The queue just fills up with a job that will never move.

The fix is two layers, because one wasn't enough:

1. A stage does at least one unit of work before consulting the clock. Forward progress becomes structural instead of hoped-for. A step may overrun its budget slightly; it will never do nothing.
2. The runner refuses to re-enqueue an unfinished step that handed back the cursor it was given, and fails the job with a stated reason.

The second layer is the one worth arguing about, since the first already fixes every stage I wrote. It exists because stages arrive through a public filter — fiction_drafts/stages — so an upload stage or an encryption stage can be added without touching core code. A third-party stage that gets rule 1 wrong should produce a failed job with a clear message, not an infinite queue on someone else's site.

Every piece of state a resume depends on is written in one class, StageRunner, and nowhere else. That's the only reason this stays tractable: there's exactly one place resumability can be got wrong.

Where you resume from depends on what you're writing

I assumed "save your position and pick it up later" was one problem. It's one problem per output format.

The database stage writes a SQL dump. Its cursor is a byte length: on resume, rewind the file to that length and keep appending. Anything written past it belonged to a step that didn't finish, so it's discarded.

That does not work for the archive, and I found out by measuring rather than reasoning. A zip file's central directory is written at the end of the file. Truncating a zip to a byte offset that was valid five entries ago produces something no reader will open. So the archive cursor counts entries, and every resume calls truncateTo() before adding anything. The principle is identical — discard whatever the persisted cursor doesn't account for — but the unit an archive admits is not the unit a text file admits.

Two smaller things fell out of the same stage, both of the "silent until a restore" variety:

The volume rollover has to happen before the add, not after. Archives get split into volumes, and if a volume were sealed after the entry that overflowed it, that entry would sit in the old volume while the cursor said it belonged to the new one. The union of the volumes would then differ from the file manifest by exactly one file. No error, no warning, one file missing.

Volumes cap at 60,000 entries regardless of size. PclZip writes no ZIP64 record, so past 65,535 entries a volume's count wraps and many extractors read it as count mod 65536 — a restore that silently loses files with nothing anywhere to indicate it happened. ZipArchive handles more, but a ceiling that applies to both writers means the resume logic has one behaviour instead of two.

Two workers on one job stopped being merely wasteful

WP-Cron overlapping an admin-ajax tick is routine, not hypothetical, so two workers can enter the same job.

For most of the build that was wasteful and no worse — a repeated batch just overwrote itself. Once resuming started mutating the archive, it turned destructive: both workers truncate to the same entry count, and both then add again. You get duplicated entries in a file whose whole purpose is being trustworthy later.

So there's a MySQL-backed lock, and the handling is one line of intent:

if ( null !== $this->lock && ! $this->lock->acquire() ) {
    return;
}

Failing to take the lock means someone else is inside this job right now, and they will re-enqueue when they finish. Returning is the whole handling. No retry, no wait, no error — the work is already in hand.

Where it stands

0.1.0 is out. It exports: five profiles, archives split into volumes with a SHA-256 for each one, downloads behind a capability check and a single-use token, and no external requests of any kind — no telemetry, no account, no key to register.

It does not restore. That's deliberate for this release, and there's a test that fails the build if an import path is added to it. Restore is coming as its own screen with its own confirmation, because "the thing that overwrites your database" and "the thing that copies your database" should not share a button. Scheduled backups and Drive/Dropbox destinations are on the roadmap too.

The plugin page has the download, the screenshots and the docs, and the source is on GitHub. If you run WordPress on a host that won't give you a shell, this is the shape of the problem I'd solve the same way again.