Troubleshooting

Start with the failing runtime and phase, not a blanket retry. R/ below means son-of-anton-review (feat/cloudflare-native); P/ means son-of-anton-operator-parity (feat/greptile-operator-parity). These are separate, unmerged trees. Commands below are operator procedures supported by source; they were not executed as part of this documentation update.

Collect the minimum evidence

Keep the delivery id, repository, PR number, head SHA, review key, phase, analysis status, publish status and exact error. Redact credentials and sensitive code from anything shared. A successful HTTP response, configured-provider badge or empty findings array does not establish a completed review.

Sources: R/cloudflare-native/src/ingress.js:267-269; R/server.mjs:86-98; R/src/model-provider.js:363-385; R/src/review-service.js:419-443; R/src/validation-executor.js:365-377.

Native diagnostics

Set ANTON_NATIVE to your native ingress origin. It is an example shell variable, not product configuration:

curl -sS -i "$ANTON_NATIVE/health"

From the review repository, inspect the configured native Worker's logs and recent stored run states:

wrangler tail --config cloudflare-native/wrangler.jsonc

wrangler d1 execute son-of-anton-review-state \
  --config cloudflare-native/wrangler.jsonc --remote \
  --command 'SELECT review_key,status FROM review_runs ORDER BY rowid DESC LIMIT 5'

Use your deployment's actual D1 database name/configuration if different. The checked-in binding names that database at R/cloudflare-native/wrangler.jsonc:73-81; the diagnostic commands appear in R/cloudflare-native/DEPLOY.md:529-531. Treat logs as sensitive. Do not print environment values while debugging.

Node diagnostics

Against the server's actual port, default 8787:

curl -sS -i http://127.0.0.1:8787/health

The operator CLI can inspect the existing SQLite run store:

node bin/gilf-review.mjs runs --db "$GILF_DB_PATH" --limit 20 --json
node bin/gilf-review.mjs show "$REVIEW_KEY" --db "$GILF_DB_PATH" --json
node bin/gilf-review.mjs usage --db "$GILF_DB_PATH" --limit 100 --json

REVIEW_KEY is the actual stored key; GILF_DB_PATH must identify the existing database. A wrong path may create/open a different SQLite store. The CLI opens a normal application store, not a read-only forensic connection; use a preserved copy if you need strictly non-mutating inspection. Do not use harness as a read-only probe: it can save a harness version.

Sources: R/src/operator-cli.js:70-89,553-644; R/bin/gilf-review.mjs:5-9. Successful CLI dispatch returns exit code 0; thrown errors print to stderr and exit 1. A command's success exit code is not the reviewed PR's verdict.

Webhook rejected or accepted but not queued

Both Node and native expect POST /github/webhooks, with x-github-delivery, x-github-event and x-hub-signature-256. Do not test it with unsigned empty JSON and interpret the rejection as downtime.

Native ingress attempts to delete the KV marker after record/enqueue failure so a retry can proceed. Oversized payloads over 96 KiB are offloaded when WEBHOOK_PAYLOADS exists; without it, a payload exceeding the Queue message limit can keep failing. A stored queued delivery record alone does not prove the send succeeded.

Sources: R/src/http-handler.js:19-61; R/cloudflare-native/src/ingress.js:63-132,139-170,227-259,275-346.

Review skipped by admission policy

Do not solve these with a forced publish. Inspect the saved policy and pinned metadata. Manual commands bypass only the automatic checks inside evaluateReviewPolicy, not repository disablement, command authorization or budget admission. Metadata-required reasons can defer rather than deny admission.

GILF_REPOS only seeds Node configuration; omission is not denial. Native does not enforce Node's branches.include/review.auto predicate in the same way. Use the runtime-specific controls in Repo scoping.

Sources: R/src/config.js:86-97; R/src/review-service.js:716-717,768-834; R/cloudflare-native/src/d1-store-adapter.js:153-172; R/src/operator-review-policy.js:154-176; R/src/review-policy-runtime.js:39-53.

Queue or container appears stuck

Native dispatch

• skipped_completed: an already successful dispatch lease was reconciled and acknowledged.
• skipped_locked: another owner holds a noncompleted lease; the delivery is retried, not silently discarded.
• skipped_obsolete: run is superseded or abandoned.
• ignored with no_review_command: native issue-comment dispatch recognizes review/rerun, not status/help as review work.
• container-dispatch-failed: inspect the logged HTTP status and response body. The follow-on container-dispatch-error is not the root cause by itself.
• lease-release-failed: the source explicitly leaves the lease running until expiry and reclaim. Do not delete a possibly active lease to hurry recovery.
• dead_lettered with unroutable or max_attempts: repair the missing identity or underlying dispatch error before scheduling fresh work. MAX_ATTEMPTS defaults to 8.

The consumer retains the actual container error body rather than reporting only a generic 500. A missing head has a specific backstop: review run ... has no headSha; the consumer must resolve the PR head before dispatch. Inspect PR-head resolution and broker access; do not invent a head SHA.

Sources: R/cloudflare-native/src/consumer.js:310-465; R/cloudflare-native/container/entrypoint.mjs:82-94.

Node worker

The server and worker must use the same durable queue or SQLite path. Without shared paths, each has an independent in-memory queue. The worker drains a bounded batch then exits; default GILF_WORKER_MAX_JOBS=1 and GILF_WORKER_CONCURRENCY=1 do not create a daemon.

lease_renew_failed and lease_lost_aborted indicate ownership trouble. The worker stops rather than risk duplicate publication. Set explicit model/validation timeouts and a sufficiently long lease: the worker's lease formula assumes a 10-minute model timeout while the runner defaults to 45 minutes.

Sources: R/server.mjs:27-50; R/worker.mjs:64-66,89-128; R/src/worker-runner.js:14-39,42-72; R/src/codex-review-runner.js:2335-2337.

Model missing, rejected, timed out or malformed

Wrong provider despite dashboard selection

Node worker.mjs does not use the generic resolver. Set GILF_CODEX_PROVIDER / GILF_CODEX_MODEL for that runner, and the current naming pair if you also want the dashboard status to agree. Native uses current provider/model names, but only forwarded keys reach its container. GILF_MODEL_SHADOW alone does not activate either runner's shadow-model call.

Sources: R/worker.mjs:43-54; R/src/codex-review-runner.js:2320-2337; R/cloudflare-native/container/entrypoint.mjs:314-348; R/cloudflare-native/src/container-env.js:15-86.

Missing key

The container can throw review provider "..." selected but no model API key in env (set OPENROUTER_API_KEY). Its guard checks for any supported key, not the selected provider's exact key; a later provider-specific missing-key error is still possible. Supply the matching bare secret to native Cloudflare. Prefixed aliases work in the provider abstraction but are not forwarded by native projection.

Source: R/cloudflare-native/container/entrypoint.mjs:319-324; R/src/codex-review-runner.js:2952-2960.

OpenRouter catalog guard

Source-defined failures include:

• OpenRouter model is required.
• OpenRouter model list failed (...), including response status.
• OpenRouter model list returned invalid JSON.
• OpenRouter model ... was not found.
• OpenRouter model ... is not free (prompt=..., completion=...).

Check exact model id, configured base URL and current catalog pricing. Keep free-only mode when that is your spending policy; choose a valid zero-priced model. If paid inference is intended, explicitly set GILF_OPENROUTER_REQUIRE_FREE=0. Do not set fake zero prices in GILF_MODEL_PRICES: it changes accounting, not this guard.

Sources: R/src/codex-review-runner.js:2022-2057,2982-2997. OpenRouter is recommended; OpenAI/Anthropic HTTP or Codex CLI are alternatives, not automatic outage failovers.

Timeout or unusable review

HTTP semantic calls can report ... chat completion timed out after ...ms or ... succeeded but did not return a usable ... review JSON payload. Check the actual timeout and provider response shape. isUsableCodexReviewOutput is a shallow gate, not full schema validation. A successful provider HTTP call alone does not mean a usable review was produced.

Do not enable GILF_PUBLISH_ON_CODEX_FAILURE=1 merely to turn this into apparent success. It permits a degraded fallback only when hypothesis recovery is unavailable, and is not forwarded natively. Partial Codex CLI output can be retained with degraded coverage and a missing-validation warning.

Sources: R/src/codex-review-runner.js:1912-1939,2692-2707,3037-3039,3074-3077,3169-3175. See Review output schema.

Validation unavailable or skipped

Sources: R/src/codex-review-runner.js:941-955,1268-1284,1337-1346; R/src/validation-executor.js:524-579.

A docs-only or generated-only validation skip can be intentional. GILF_VALIDATION_CODE_CHANGES_ONLY=false disables that gate; 0 does not. Policy execution.mode=never produces NOT RUN repo-native validation: execution_disabled. rather than a sandbox failure (R/src/validation-config.js:11-14; R/src/review-policy-runtime.js:74-81).

Timeout distinction: E2B collector fallback is 10 minutes, repo defaults can supply 45 minutes, and the native manifest explicitly selects 10 minutes. Set an explicit timeout appropriate to the plan. Shadow lane config is capped by GILF_VALIDATION_SHADOW_TIMEOUT_MS. A shadow failure is not evidence that primary validation failed (R/src/config.js:18-28; R/src/codex-review-runner.js:1321; R/src/validation-executor.js:269-282).

V5 enabled in Worker vars but nothing happens

This is an implementation boundary, not a provider outage. FORWARDED_KEYS excludes swarm, planner, Prime, inversion and evidence flags. No environment-only recovery exists on native Cloudflare; the integration owner must change the projection and supply the dependencies.

On Node, hypothesis_planner_skipped can name no_hypothesis_worker or codex_provider_unsupported. With no enabling worker flags, swarm returns null. GILF_DISABLE_HYPOTHESIS_WORKERS only changes a harness descriptor, not execution.

If artifact processing runs but cannot persist, evidence_artifacts_persisted can report no_artifact_storage; a processing exception emits evidence_artifacts_failed and returns original findings. GILF_EVIDENCE_REQUIRE_ARTIFACTS does not create storage or act when artifact processing is off.

Sources: R/cloudflare-native/src/container-env.js:15-86; R/src/codex-review-runner.js:3248-3267,3342-3353,3517-3569; R/src/harness-evolution.js:56-59.

E2B shadow suddenly stops

Native dispatch sums validation.e2b cost rows and logs e2b_budget_exhausted at GILF_E2B_BUDGET_USD, default 50. It then withholds the exact managed-e2b shadow selector. Primary review and publication remain unchanged. e2b_budget_read_failed means the read failed and the selector was left enabled, not safely suspended.

Review the recorded costs and intended budget before increasing the ceiling. Do not delete cost rows or substitute zero prices to resume spend. This is a cumulative ledger gate, not a documented monthly reset or primary-E2B cap.

Source: R/cloudflare-native/src/container-env.js:100-141.

Analysis succeeded, publication missing or unsafe to approve

1. Inspect actual native log omp-container: publish mode. Code default is shadow; checked-in manifest is live. Node ignores this env gate and selects live publication when App credentials are provided.
2. Check publish.status, errors and receipts separately from analysis.status.
3. Live native startup requires broker transport. Container publish mode does not match its dispatch lease is a consistency failure, not a reason to bypass the lease.
4. Approval requires pinned head, confidence 5, complete coverage, no missing validation, allowed risk and matching filters. A high confidence score alone is not merge readiness.
5. A positive confidence threshold with incomplete coverage yields a neutral check, not a success. Missing/below-threshold confidence yields failure.

Sources: R/cloudflare-native/container/entrypoint.mjs:273-302; R/worker.mjs:35-41; R/src/operator-review-policy.js:227-238; R/src/review-format.js:138-149; R/src/github-publisher.js:50-65.

The native command bridge supports retry-publish only for an existing run with analysis.status === 'succeeded' and an installation id. Its actual errors include Review run is not analyzed: ... and Cannot retry publish for ...: missing installation id. It queues recovery work; it does not directly certify publication. Use Operator API only on the branch/deployment that owns that command store. The native public ingress does not expose parity command endpoints.

Sources: R/cloudflare-native/src/cron-command-bridge.js:225-247; R/cloudflare-native/src/main.js:26-29.

Dashboard 401, 403, 404 or settings conflict

There are three authentication surfaces, not one universal token:

• Node operator handler: GILF_OPERATOR_UI_ENABLED=1 enables routing, but a missing or wrong GILF_OPERATOR_UI_TOKEN returns 401 operator_auth_required. It is fail-closed.
• Original anton-ui proxy: its separate OPERATOR_REQUIRE_AUTH gate is off by default; that does not remove downstream backend auth requirements.
• anton-ui-trace-parity: /api/* authenticates a session before forwarding. It uses backend read configuration and SON_OF_ANTON_ADMIN_TOKEN for privileged operations. Setting the original UI's bearer gate is not equivalent to configuring this session system.

Sources: R/src/operator-dashboard.js:601-620; anton-ui/worker/proxy-utils.js:46-64; anton-ui-trace-parity/worker/index.js:20-47.

Sources: anton-ui-trace-parity/worker/index.js:34-75; anton-ui-trace-parity/worker/proxy-utils.js:95-100.

For direct parity Worker policy reads, with the origin and existing token securely configured:

curl -sS -i "$ANTON_OPERATOR/operator/api/settings" \
  -H "authorization: Bearer $SON_OF_ANTON_OPERATOR_READ_TOKEN"

Settings PATCH uses the administrative token, a JSON body with the current revision and a settings patch. Generated anton_ keys do not grant settings-write access. 409 Review policy revision conflict means reload and compare before reapplying; do not overwrite a colleague's saved changes. The parity editor preserves edits and offers comparison/reapply/discard controls.

503 Review policy schema is not initialized means the workspace settings row is absent. 500 Review policy storage is unavailable is the handler's generic storage failure. Confirm the correct D1 binding and reviewed schema initialization rather than treating either as a policy validation error. 413 Request body is too large is the 256 KiB body limit. Usage can also return 413 for more than 50000 source rows; use a narrower date range.

Sources: P/cloudflare/src/worker.js:65-80,115-146; P/cloudflare/src/review-policy-api.js:20-63; P/src/review-policy-store.js:40-42,69-89,143-146; anton-ui-trace-parity/src/views/Settings.tsx:235-245.

Empty traces or schema errors

First check whether tracing was disabled, whether Node has GILF_DB_PATH, and whether the UI targets the correct backend. Node trace-store initialization errors log trace-store-init-failed. A model with no known rate legitimately has null cost; that is not the same as missing traces.

Older native databases can have a narrow review_trace_spans.kind constraint and no seq column. Source records the failure no such column: seq at offset 87: SQLITE_ERROR. Re-running CREATE TABLE IF NOT EXISTS does not upgrade an existing table.

Do not blindly apply 001-widen-trace-span-kind.sql. It drops both trace tables. Its written precondition requires both counts to be zero. Preserve/export existing data and use a reviewed copy-forward migration if either table contains rows. The relevant diagnostic query is read-only:

wrangler d1 execute son-of-anton-review-state \
  --config cloudflare-native/wrangler.jsonc --remote \
  --command 'SELECT (SELECT COUNT(*) FROM review_traces) AS traces, (SELECT COUNT(*) FROM review_trace_spans) AS spans'

If the tables themselves are absent, the query failing is also evidence; it is not authorization to destroy or recreate unrelated tables. See Upgrading for deployment-owned migration work.

Sources: R/server.mjs:27-35; R/src/codex-review-runner.js:2297,2348-2350; R/cloudflare-native/schema.sql:159-177; R/cloudflare-native/migrations/001-widen-trace-span-kind.sql:1-32.

Replay or cron recovery does not complete

The native command bridge operates on D1 command_events. CRON_AUTHORITY=shadow plans commands without claiming or executing them; dryRun: true and planned are not completion receipts. With live authority, commands transition queued to running to succeeded/failed. A transition conflict skips execution rather than racing another bridge.

Replay reads D1 webhook_deliveries, then an R2 delivery object fallback. An unknown-delivery error still contains stale wording saying ingress does not record deliveries; current ingress does record accepted deliveries. Investigate missing historical rows, the wrong database/bucket or unreplayable payload instead of assuming recording is unimplemented. Already processed/succeeded deliveries require explicit force; do not force blindly.

Sources: R/cloudflare-native/src/cron-command-bridge.js:87-113,129-180,250-269,333-374; R/cloudflare-native/src/ingress.js:227-259,311-340.

Watchdog messages such as loop artifact generatedAt is missing or invalid, loop artifact is stale: ..., or ...: no cron heartbeat recorded identify missing/old evidence. Restore the loop artifact, schedule or dependency that failed; do not manufacture a current timestamp or hide historical dead letters. The native schedule watches loop, watchdog, command bridge and nightly audit; Prime-watch is deliberately unscheduled.

Sources: R/cloudflare-native/src/cron-watchdog.js:49-94; R/cloudflare-native/src/cron.js:53-88; R/cloudflare-native/src/consumer.js:256-284.

Recovery completion criteria

After an authorized recovery, inspect the new attempt's pinned head, actual phase result, missing validations and publication receipt. A requeue command only proves enqueue; a policy write only proves that revision was saved. Preserve the original failure evidence and record unresolved external prerequisites, including provider availability, deployment wiring and source/license access. Never turn a missing check into a claimed pass.