The crawl frontier is a state machine.

Decide what counts as the same URL

URL identity comes before priority. RFC 3986 permits syntax normalization such as lowercasing schemes and hosts, normalizing percent-encoding of unreserved characters, and removing dot segments. It does not make query parameters disposable, paths case-insensitive, or trailing slashes equivalent on every server.

Keep the observed reference, the absolute resolved request URL, and a comparison key generated by documented rules. If a normalization rule later proves wrong, the source survives. Classify parameters as tracking-only, order-sensitive, content-bearing, or unknown without rewriting that observation.

Fragments usually do not trigger another HTTP retrieval, but still describe in-page navigation. Keep them on discovery edges, outside fetch keys. Reject user information, unsupported schemes, malformed ports, and out-of-scope hosts with named states rather than silently dropping them during parsing.

Record transitions, not queue positions

A pending flag hides the difference between a host delay, robots pause, retry, and scope rejection. Named states—discovered, admitted, scheduled, fetching, fetched, retryable, suppressed, and terminal—need entry conditions, allowed successors, and checkable invariants. The vocabulary can vary; the contract cannot be implicit.

Append transitions keyed by run, URL identity, attempt, and time. Rebuild current state from those events rather than trusting an overwritten status row. If a worker crashes between a row update and response capture, the record must expose that gap. A stale fetching lease can return to scheduled; a terminal response requires an explicit new attempt or run.

Give workers expiring leases, renewed only during useful work, and attach a monotonically increasing lease generation to each assignment. Reclaim abandoned items without erasing their attempts. When the old worker returns after reassignment, retain its response as late evidence but require the current lease generation before it can advance frontier state. Expiry alone cannot stop an already-running worker from writing.

Allocate capacity by origin

Global FIFO can let a large host occupy the workers. Partition eligible work by scheme, host, and port, with concurrency limits, minimum intervals, next-eligible times, and recent response health. The global scheduler chooses an origin; its local scheduler chooses a URL.

Fast successful responses may support more concurrency than timeouts or server errors. Adapt slowly within fixed limits. Back off the affected origin after transport failures, 429 responses, or repeated 5xx responses; unrelated origins keep their capacity. Persist the response history and controller inputs so the decision can be reproduced.

Reapply scope, robots and destination-origin capacity at each redirect hop. A URL admitted on one origin can redirect to another, and the original host’s permission or spare slot does not authorize dispatch there. A convenient HTTP client that follows the chain invisibly can bypass the scheduler’s own contract; capture and schedule the next hop explicitly when those boundaries matter.

Evaluate robots policy before dispatch. Cache retrieval time, status, expiry, and parser version. RFC 9309 defines access-failure and caching behavior; a more conservative operator policy remains a separate decision from the protocol-derived parse result.

• Use a fair origin selector so a deep host cannot starve smaller hosts.
• Apply random jitter within a bounded window to avoid synchronized retry bursts.
• Cap both per-origin concurrency and total active sockets.
• Re-evaluate host policy when a cached robots record expires or changes.
• Treat server health as a scheduling input, not as evidence about page quality.

Bound discovery before it runs away

Finite content can generate an effectively infinite address space: endless calendars, facet permutations, sort orders, session IDs, and internal-search queries. Waiting for the queue to empty is not a stopping rule for such sites.

Budget accepted URLs, depth, query combinations, path patterns, per-host pages, redirects, response bytes, render time, and run duration independently. A breach must record the rule, affected pattern, first example, and suppressed count. Otherwise a truncated sample can be mistaken for the site.

Trap rules have unequal failure costs: broad rules hide valid content; narrow rules can exhaust the run. Start with reversible deferral, group structural signatures, and inspect examples to learn which URL components change content. Exceptions that allow a URL should be narrower and more explicit than the rule they override.

Give retries a reason and a limit

For each failure, decide whether it is plausibly transient, when another attempt is eligible, and how many attempts remain. DNS timeouts, resets, 429 responses, and many 5xx responses can justify retries. Stable 404s, scope rejections, and robots denials generally require a new run or policy change. Preserve earlier responses and errors even after success.

Use exponential backoff, bounded jitter, and a maximum delay. Honor applicable Retry-After values within local safety limits. Save the classification rule with the computed next time: the time alone cannot explain the decision. Exhausted attempts become visible terminal failures in coverage reports.

At shutdown, reconcile URL identities separately from attempts. One admitted URL may have a failed attempt, an expired lease and a successful retry; counting three requests as three covered pages inflates coverage. Report the final disposition per admitted identity, the attempt history behind it, and any in-flight or deferred work remaining at the cutoff.

Idle workers do not prove completion. Reconcile discovery, deduplication, suppression, fetches, retries, and failures until every admitted item is terminal or explicitly unresolved. An unreconciled frontier cannot support a site-wide conclusion.

type FrontierTransition = {
  runId: string;
  fetchKey: string;
  attemptId: string;
  from: FrontierState;
  event: string;
  to: FrontierState;
  occurredAt: string;
  policyVersion: string;
  evidenceRef?: string;
  reason: string;
};

Reconcile every admitted URL

Before making a site-wide claim, trace every admitted URL to a terminal result or an explicit unresolved state. Throughput becomes useful once identity, host capacity, retries, and recorded transitions make that coverage defensible.

Sources

1. RFC 3986: Uniform Resource Identifier generic syntax

   Verified Jul 19, 2026

2. RFC 9309: Robots Exclusion Protocol

   Verified Jul 19, 2026

3. Google Search: URL structure best practices

   Verified Jul 19, 2026

4. Google Crawling Infrastructure: crawl budget management

   Verified Jul 19, 2026

5. RFC 9110: HTTP Semantics

   Verified Jul 19, 2026

Read next