The problem
A data_sync run is one long stream over a cursor. On core's engine:
- a deploy in the middle of a multi-day backfill kills the run
- a worker that dies leaves the run
running forever - one transient error at hour nine fails the whole run, throwing away hour one through eight
Install
yarn add @fullstackhouse/open-mercato-data-sync-durable @fullstackhouse/open-mercato-durable-work pg-boss
Name durable-work explicitly: it is a peer of this package, not a dependency, and yarn does not install peers for you. That is not tidiness — it exports a process-wide registry, so a second copy in the tree would mean job kinds register into one while the worker reads the other, silently. pg-boss is the client for the default transport; swap it for bullmq and ioredis to run on Redis.
The two packages release in lockstep at one version, and this package's peer range is rewritten to ^<that version> on every release — upgrade them together (yarn up '@fullstackhouse/open-mercato-*'). Still 0.x: pin the exact version in production.
src/modules.ts — the whole change is one from:
{ id: 'durable_work', from: '@fullstackhouse/open-mercato-durable-work' },
{ id: 'data_sync', from: '@fullstackhouse/open-mercato-data-sync-durable' }, // was '@open-mercato/core'
Then yarn generate && yarn db:migrate — a no-op for existing data_sync history, because the migration classes keep core's names — and run yarn mercato durable_work worker.
Reverting is the same line.
What changes
A run gets a lease. It commits its cursor under that lease, hands back at a batch boundary when its slice budget is spent or the process is stopping, and resumes from the committed cursor. The batch that spends the budget is always committed first, however long it took — the hand-back lands at the start of the next one — so every slice makes progress even when a single batch outlasts the whole budget. A transient error retries the slice instead of failing the run. A worker that dies is noticed by the reconciler within about a minute. One live run per integration, entity and direction.
That makes lease.sliceBudgetMs a soft limit: a slice runs until the first batch boundary past it, so it can outlast the budget by a whole batch — size the budget against the slowest batch, not the average one. Two to three times the slowest batch leaves room; a budget that a single batch can already exceed does not.
A slice that never reaches a boundary at all — an adapter stuck on a socket or a lock that never returns — is abandoned after DATA_SYNC_MAX_SLICE_MS. The run is re-driven from its last committed cursor, and parked as poison if it hangs every time, which releases the stream for a new start. The broker's in-flight timeout follows the deadline, so a slice that is merely slow is never redelivered while it runs.
Runs no job carries
A run started through core's start path becomes durable when its queue message is delivered. If that never happens — the process died before the enqueue, the message was lost, or every adoption was refused — the run would sit pending or running with no job, and core's overlap check would refuse every later start of that stream. A sweep on the reconciler's tick settles those runs once they have been quiet for the grace period:
- its job already finished: the run takes the job's outcome
- no job, and quiet for less than the adoption limit: a job is created and the run resumes
- no job, and quiet for longer: the run is failed rather than resumed from a stale cursor
- another live run holds the stream: the run is failed with that reason
A run it fails gets core's finalize tail — progress job, integration health, lifecycle event —
