Son of Anton Docs
Architecture

Queue, Idempotency and Durability

Delivery deduplication, dispatch leases, uncertain publication and the limits of checkpoint recovery.

This page describes the deployed-engine lineage, son-of-anton-review on feat/cloudflare-native. The Durable Object queue in cloudflare/ is a different backend from the native Cloudflare Queue. Queue-repair APIs in unmerged feat/greptile-operator-parity must not be presented as native queue administration.

Delivery is not exactly once

The native configuration consumes gilf-review-intents with max_batch_size: 1, max_batch_timeout: 5 and max_retries: 8; the configured dead-letter queue is gilf-review-dlq.

Ingress uses KV key delivery:<deliveryId>, with DEDUPE_TTL_SECONDS defaulting to seven days. KV is eventually consistent, so two deliveries can both pass the dedupe check. A D1 record is written before enqueue. A recording or enqueue failure attempts to remove the KV marker and returns 500. Those operations are not a cross-service transaction, and dedupe rollback itself is best-effort.

Do not describe this as exactly one review per head. Distinct manual commands intentionally have distinct identities, and remote publication can have an uncertain outcome.

Sources: engine cloudflare-native/wrangler.jsonc:56-69; cloudflare-native/src/ingress.js:295-346; cloudflare-native/src/consumer.js:287-304.

Review keys and dispatch leases

The ordinary key is owner/repo#prNumber@headSha. Manual comment requests can append :manual-<commentId>. A supplied reviewKey takes precedence. Missing-head intents are resolved before the dispatch claim; this avoids leaving normal comment reviews permanently keyed at @pending.

The dispatch lease is in publish_leases, not publish_ledger. Its action is container-dispatch or container-dispatch:shadow. Claiming is a conditional update that refuses succeeded leases and runs already marked superseded or abandoned.

LEASE_MS defaults to 30 minutes. A running lease becomes claimable when its recorded start/renewal time is older than the cutoff. The active consumer renews ownership and aborts dispatch if renewal fails. Another delivery retries while a holder is running; only a succeeded holder lets the duplicate be acknowledged.

The fencing migration checks a matching running action and holder for snapshots carrying dispatchLease. It does not itself compare the timestamp to a wall-clock expiry. A replaced holder cannot write through that fence; simply passing the time cutoff is not the same event as replacement.

Sources: engine cloudflare-native/src/consumer.js:17,33-105,287-304,344-419; cloudflare-native/migrations/003-runtime-lease-fencing.sql:5-29.

Publication claims do not expire into retries

Native publication uses an awaited transport backed by publish_ledger:

  1. The bridge flushes run state before claiming a publication action.
  2. A claim inserts a running action under the active dispatch identity.
  3. A duplicate succeeded action returns its stored result without a new claim.
  4. A duplicate non-succeeded action throws: remote receipt reconciliation is required.
  5. A failed request records uncertain, not proof that GitHub did nothing.
  6. Completion may record the remote result even after the dispatch lease changed, but only for the action's original claimant.

Unlike the dispatch lease, a native running or uncertain publication claim has no expiry-based resend path. Do not copy the older SQLite publication-lease behavior into the native runbook.

The GitHub client attempts a mutation once. Check creation carries external_id: reviewKey; PR reviews carry <!-- anton-review <reviewKey> -->; receipt comments carry <!-- gilf-review-receipt <reviewKey> -->. These are evidence for reconciliation. They are not evidence that every ambiguous write is automatically reconciled by the normal retry path. Investigate the remote object and ledger before authorizing another mutation.

Sources: engine cloudflare-native/src/d1-store-adapter.js:263-282; cloudflare-native/src/state-transport.js:490-524; cloudflare-native/src/broker-github-client.js:78-88; src/github-publisher.js:22-65; src/review-service.js:181-191.

What a checkpoint preserves

A container flush persists run state and buffered traces over the outbound Worker. The bridge restores drained traces after a failed flush and only advances its event cursor after acknowledgement. State and traces are separate writes within that request, so partial persistence remains possible.

The server waits for a successful final flush before returning success. During termination it attempts to record a phase marker and flush, with a 10-second exit timer. Failures are logged, not guaranteed recoverable.

On another attempt, durable analysis.status = succeeded lets the container skip the model and resume stored publication. A phase marker alone does not restore an interrupted model call or sandbox process. Work before a successful checkpoint can repeat.

Sources: engine cloudflare-native/src/d1-store-adapter.js:303-354; cloudflare-native/src/state-transport.js:558-579; cloudflare-native/container/entrypoint.mjs:49-68,95-148,421-477; src/review-service.js:350-364.

Dead letters and watchdog counts

The consumer handles unroutable messages and errors reaching MAX_ATTEMPTS (default 8) by calling its dead-letter helper and acknowledging the original message. MAX_ATTEMPTS uses the message's delivery-attempt count; it is separate from Wrangler's retry setting.

The helper first attempts the DLQ send, then records whether that send succeeded in cron_dead_letters. Both the send failure and a ledger-write failure are caught and logged. The original message is still acknowledged. A DLQ send is therefore not guaranteed, and a missing audit row can make the ledger undercount.

The watchdog projection reads unacknowledged ledger rows only when CRON_DEAD_LETTER_LEDGER=enabled; otherwise the count is unobservable, not zero. acknowledged_at IS NULL selects outstanding rows. No acknowledgement CLI was found in the audited engine bin/, scripts/, src/ or native src/; do not invent a purge or acknowledgement command. Recovery must preserve evidence and distinguish ledger acknowledgement from physical queue removal.

Sources: engine cloudflare-native/src/consumer.js:256-284,324-328,450-465; cloudflare-native/src/cron-queue.js:154-164; cloudflare-native/src/cron-state.js:98-101; cloudflare-native/src/cron-watchdog.js:112-117.

Node queue selection

src/worker-runner.js selects:

  1. Remote Durable Object queue when both GILF_QUEUE_URL and GILF_QUEUE_TOKEN are set.
  2. SQLite queue at GILF_QUEUE_DB_PATH when set.
  3. SQLite queue at GILF_DB_PATH when set.
  4. Otherwise, an in-memory queue.

The worker's lease derives from model and validation timeouts plus five minutes of slack unless a positive GILF_QUEUE_LEASE_SECONDS overrides it. This is not the native LEASE_MS setting. The parity /operator/api/queue/* repair routes forward to a Durable Object and then record an audit event; they do not operate on gilf-review-intents or provide a transaction across queue repair and audit storage.

Sources: engine src/worker-runner.js:14-39; parity cloudflare/src/review-state-api.js:544-572.

See Cloudflare-native runtime, Publish modes, and Operator API.

Key Broker

Separating the App private key from native review containers, without overstating the broker's authorization controls.

Observability and Cost

Trace lifecycle, persistence failures and cost estimates without treating missing telemetry as zero spend.

On this page

Delivery is not exactly onceReview keys and dispatch leasesPublication claims do not expire into retriesWhat a checkpoint preservesDead letters and watchdog countsNode queue selection