The hardest failure in a job aggregator is not a broken request. It is a broken request that looks like valid new information.
01 / The problem
Make incomplete scans harmless.
An employer feed can stop halfway through pagination, time out after several successful pages, or return an implausible empty result. If a collector treats that partial view as the complete source, still-open roles can be marked missing or closed even though the system never observed a complete board.
I built the engine around an atomic scan-commit boundary. Pages and seen-record identities are staged under a monotonic scan generation, but presence and absence change only after the page graph is complete and every required page succeeded. Failed or partial scans leave prior presence unchanged; suspicious zero-result scans are quarantined instead of mass-closing jobs.
Observe. Retain the fetched evidence, attempt state, and field provenance without assuming that a source enumeration completed.
Quarantine. A partial, failed, stale, or suspiciously empty scan cannot establish absence; uncertainty remains explicit instead of becoming a bulk state change.
Commit. Only a complete, current scan generation can atomically support presence or absence and the downstream canonical event.
Simplified scan-state model. It explains one invariant, not the full private implementation.
02 / The work
Keep evidence and identity reversible.
Versioned adapters cover Greenhouse, Lever, Ashby, and generic JSON-LD. Raw response bytes are content-addressed by hash, while each fetch attempt retains its own snapshot and body-free audit. Field-level provenance survives normalization, so a current canonical value can still be traced back to the observation and deterministic rule that produced it.
A vendor requisition ID is evidence, not unconditional identity. Source occurrence, canonical job, and job family remain separate records; incompatible title or location facts can create a reviewable duplicate state instead of an automatic merge. Identity decisions are immutable and supersedable, which makes a mistaken match correctable without deleting the underlying observations.
03 / The judgment
Separate authority, then test the whole state machine.
The system gives each boundary less authority than the whole pipeline. Adapters transform supplied bytes but cannot fetch. The fetch gateway cannot declare source presence. Canonical projection writes job facts and transactional outbox events, while a separate idempotent projector rebuilds search documents. The public API serves only publication-eligible jobs whose exact serving source still has a current redistribution decision, through a source-safe listing contract that withholds acquisition evidence and direct destinations.
A deterministic offline harness drives fixtures through capture, parsing, scan finalization, freshness transitions, outbox projection, search, API, and server rendering. Live acquisition is a separate, bounded opt-in that fails closed when required source-policy evidence is absent or stale. The architecture remains a PostgreSQL-backed modular monolith: distributed infrastructure is deferred until a measured threshold or failure mode justifies the added system.
04 / Decisions
What I chose.
And why.
- Commit only complete scans
- A transport or pagination failure must not turn into a false closure signal.
- Quarantine suspicious zero-result scans
- An established source suddenly returning nothing is evidence of uncertainty before it is evidence that every role disappeared.
- Reversible canonical identity
- A mistaken duplicate decision should be correctable without losing the original observations or provenance.
- Source-safe publication boundary
- Public listing responses should expose approved job facts without leaking acquisition evidence, private workflow state, or source destinations.
- Modular monolith before distributed infrastructure
- Keep transactional invariants in one authoritative PostgreSQL system until a measured bottleneck justifies another platform boundary.
05 / Result
What came out of it.
An implemented private engine with versioned ATS adapters, immutable provenance, atomic scan semantics, reversible canonical identity, fenced workers, transactional outbox projection, source-safe listing APIs, and deterministic offline end-to-end verification.
Status & source notes
Offline fixtures and synthetic capacity tooling are not production throughput or verified internship coverage. This page does not assert that the engine is the deployed InternshipDeadlines backend. Live acquisition in the project is separately bounded and opt-in; source-policy records, acquisition data, runtime credentials, and worker configuration remain private.
Case study updated September 12, 2026.