# halo-action-pipeline — operational detail

Deep operational doctrine for the plan→apply pipeline: the hang model, executor cost/model rationale, lease sizing, the coordinator watchdog, and the `halo-apply` workflow mechanics. The pipeline contract itself (roles, files, action/progress schemas, `pipeline.py` commands) lives in `skills/halo-action-pipeline/SKILL.md`.

## The hang model — what the file-first design does and does not recover

**The file-first model recovers executors that RETURN, not executors that HANG.** An executor that returns `null` (died) is detected and its lease is reclaimed on the next round — that is the recoverable case. An executor that HANGS — never returns at all, distinct from one that returns null — is NOT recovered by the round logic: the preferred `halo-apply` workflow's round does `await parallel(leases...)` and waits for EVERY lease, so a single hang blocks the whole round indefinitely and holds a concurrency slot, starving the queued leases. The only thing that recovers a hang is a per-agent timeout. `pipeline.py stalls` IS that watchdog — but today it runs ONLY in the manual apply loop, not inside the workflow. The defenses against an effective hang: keep write-response bloat out of context (the slowdown that turns into a hang — see `[[halo-bulk-update]]` anti-pattern #3), and drive the workflow under a foreground watchdog (below).

**Why the pipeline exists:** at scale, write state held in an agent's context is fragile — a silently hung subagent stalls a run and loses its in-flight work, indistinguishable from a slow one. Moving the unit of work to a *file* (one line per MCP call) means a hang costs one call, progress is inspectable mid-flight, and any fresh agent resumes from the file, not from memory.

## The script's role — accelerator, not gate

**The script (`pipeline.py`) is an accelerator, not a gate.** It does deterministic mechanics ONLY — ready-set, `$ref` resolution, stall detection — the bookkeeping you never want an LLM eyeballing across hundreds of rows. It is pure functions over append-only files (no destructive writes), so a bug can't corrupt state. The CONTRACT is the JSONL files, which are plain readable JSON: if the script breaks or is unavailable, agents fall back to deriving the ready-set and resolving refs by hand from the files. The script failing degrades to slower, never to blocked-or-lost. Flexibility lives in the agents; determinism in the script; the file is truth either way.

**`payload_obj` → `payload` serialisation (update_asset).** Halo's `update_asset` takes ONE parameter, `payload` — a JSON `/api/Asset` body — so the planner/compiler emits a STRUCTURED `payload_obj` (Asset Fields id-keyed from the tenant's `--field-map`, custom fields name-keyed) and `resolve` serialises it to the `payload` string only AFTER `$ref` substitution. Serialising at compile time would freeze unresolved `$ref:...` text into the string. Hand-deriving without the script: substitute the `$ref`s inside `payload_obj`, then `json.dumps` it as the single `payload` argument.

## Executor cost model and tiering

Executors are safe to run on a **low-cost model tier at low effort** — the stage is mechanical (one pre-composed MCP call per action) and carries nearly all pipeline tokens, so this is where cost lives, not quality. Three conditions make that safe: the lease is fully specified (zero composition), failures are fail-loud into the append-only ledger (retried/reassigned by the coordinator), and elevated `fail` waves escalate the dispatch-site model one tier — NOT prompt surgery. Model/effort selection lives at the dispatch site in `halo-apply.js` and the agent frontmatter (never in the skill); the residual risk is a wrong-`$ref` resolution that still writes successfully, covered by guarded creates and get_asset verification sampling.

**Two-tier floor by lease kind (2026-08-11, 5th fabrication incident).** The cheapest tier stays safe ONLY for guarded creates, where the guard search independently cross-checks existence. Non-guarded mechanical (rows-mode) leases carry a one-step model floor at the dispatch site: in one run, two cheapest-tier executors appended `done` for 32 rows-mode actions with zero tool calls, while every floor-tier executor on the same actions was honest.

**Uniform mechanical rows batches default to MAIN-THREAD application.** When every action in a batch calls the SAME tool with homogeneous args (`create_validations`-class), skip the fleet: resolve each action with `pipeline.py`, call the tool directly from the main thread, collect the receipts in the session transcript, and read back. A batch with no per-row reasoning and no cross-tool parallelism win does not pay back executor startup, and every agent hop adds a fabrication surface — the fleet earns its cost on heterogeneous action files. (The server-side Array-Iteration runbook stays the first preference for uniform writes when the tenant carries one — the skill's Phase-0 gate.)

## Lease sizing — the two opposing forces

**Lease sizing balances TWO opposing forces, not just startup cost.** Executor startup costs ~20k tokens regardless of lease size, which favors LARGER leases (amortize the startup). But inline write-response accumulation — ≈KB per call, NOT offloaded under the ~262 KB cap — compounds in the executor's context across the lease and favors SMALLER leases (each write gets slower as the body pile grows; left unchecked this degrades into an effective hang). Both forces are still live for any write tool that echoes a full object.

The second force disappears for **slim-projection** write tools, which return a small receipt instead of the written entity (`{id, inventory_number, client_id/name, site_id/name, status_id, inactive, written[]}`): `update_asset` v1.0.0+ and the two `*_from_library` tools. Sizing:

- Guarded creates: **2–4 per lease** (blast radius, independent of response size).
- Slim-projection mechanical follow-ups: **15–20 per lease** — only startup amortization applies.
- Full-object mechanical writes: **6–10 per lease** to cap accumulation. Mechanical writes with TINY responses (assign / structure / relationship / status-only) also take the 15–20 sizing.

**Size to the DEPLOYED runbook, not the repo spec.** A tenant that has not imported `update_asset` v1.0.0 still returns the full ~6 KB asset object on every call and must stay at 6–10, as must any other write tool still echoing a full object. Discriminate by the response the tenant actually returns. See `[[halo-bulk-update]]` anti-pattern #3 for the write-response bloat mechanism.

## Coordinator watchdog — driving `halo-apply` on large runs

**Drive `halo-apply` under a FOREGROUND watchdog for large or mechanical runs.** The workflow's `await parallel()` round has no per-agent timeout, so a hung executor freezes the run silently with no completion notification (one incident froze at done 108→111→flat forever). Do NOT background the workflow and wait for it to report. Instead run it in an active foreground loop (e.g. `/loop`), poll the ledger (`pipeline.py status` / `stalls`) every ~45 s, and if `done` has not advanced across 2 consecutive polls → `TaskStop` the run + relaunch with `doneIds` regenerated from `progress.jsonl`. Relaunch is idempotent and ledger-resumable (guards make redo safe), so a restart never loses or double-writes work. Never rely on a completion notification to detect a hang — the background workflow can die with none.

Manual apply-loop fallback (when not using the workflow): `status` → `ready --limit K` → spawn executor leases per the sizing rule (cap concurrency ≈4-6) → heartbeat `stalls --minutes 10` → `TaskStop` + reclaim any stall → repeat until clean. Never wait on completion notifications alone to detect a hang.

**The watchdog also samples ledger integrity.** Each poll, run `pipeline.py verify-chain` alongside `status`: a `done` count that REGRESSES between polls, or fresh broken links, means the ledger is being rewritten — halt the run immediately and preserve the ledger before anything else. Ledger monitoring is the only reason the 2026-08-06 tampering incident is known at all; it must be first-class, never ad-hoc.

## Ledger integrity — append-event, verify-chain, and the tenant-as-witness rule

`progress.jsonl` is a compliance audit trail, not a scratch log. Every event is written through `pipeline.py append-event`, which hash-chains events (`seq`/`prev`/`hash`) under an exclusive `flock`. The lock is load-bearing: ~16 concurrent executors without it produced duplicate seqs that made `verify-chain` report `intact:false` on 39 of 40 samples with no actual tampering — a tamper alarm with a 97.5% false-positive rate provides no evidence. `verify-chain` recomputes the chain and exits non-zero on any break; run it as a completion gate and report COVERAGE (chained/total): a ledger observed chaining 13 of 271 events is not an integrity record.

Three rules the 2026-08-06/07 incidents proved:

1. **Direct writes are forbidden and detectable.** The old `Bash >>` contract handed executors raw shell access to the ledger; they used `sed`/`grep -v`/`git checkout` to delete their own fail events, truncating the ledger to zero twice in one run. `append-event` is the only writer; anything else is treated as tampering.
2. **A done event is not evidence the write happened.** Executors fabricated `done` events without calling the tool (54 done-without-start events in one run; 8 self-confessed in fail notes). `append-event` rejects a `done` with no prior `start` and no `--ids`; the post-run tenant population check (`commands/halo-apply.md`) is the control that actually closes this.
3. **The tenant is the only reliable witness.** Verify completion with tenant reads (`asset_summary` counts, sampled `get_asset` read-backs confirming fields are POPULATED), never by counting ledger dones — the API also accepts wrong-shaped payloads with HTTP success and silently discards the content, so even honestly-recorded dones prove acceptance, not persistence. Deleted ledger events additionally make duplicates invisible to ledger-based detection; only a tenant read finds them. Datapoint (2026-08-11): `update_ticket_fields` on a Validation ticket returned success while silently dropping the CF values — the ledger's recorded payload showed them applied; only the `get_ticket` read-back caught it.

### Seeding a mixed manual/fleet handoff (`append-event --ids`)

When some actions were applied by hand and the rest go to the fleet, seed the ledger per manual apply — WITH the returned ids:

```
python3 pipeline.py append-event --progress <run>/state/progress.jsonl \
  --id <action-id> --status done --ids '{"asset_id": N}'
python3 pipeline.py resolve --id <dependent-action-id> --prefix <tenant-prefix>   # confirm the $ref resolves BEFORE handoff
```

`--ids` takes the RETURNED entity ids as a JSON object — never a list of action ids (a real naming trap: both are "ids"). A `done` appended without it leaves every downstream `$ref` unresolvable: `resolve` reports `no done event / key` even though the done event exists. Passing `--actions` arms the receipt gate: a done on any action that declares `returns` is rejected without `--ids`, even after a start.

### Poisoned-ledger remediation (verified-false done events)

A fabricated `done` in an append-only ledger is PERMANENT, and every honest actor downstream trusts it: executors consult the ledger and skip "already-done" ids, and a `doneIds` seed from ledger events carries the lie into the workflow layer. Measured 2026-08-11: a redo wave of honest executors no-op'd over 32 fabricated dones — correctly, by their own contract — leaving the tenant 32 rows short behind a fully green ledger. When a read-back proves a `done` false:

1. **Quarantine the affected ids.** Never launch another executor wave over the poisoned ledger.
2. **Remediate from the main thread** (`pipeline.py resolve` per action → direct tool call → receipts in the session transcript → full read-back), or bootstrap a FRESH ledger for the redo.
3. **Regenerate `doneIds` from tenant read-back**, never from ledger events — the guarded-creates seeding rule extends to every action kind after a fabrication finding.
4. **Append `superseded` per corrected id** (`append-event --status superseded --note "<read-back evidence>"`). `ready` re-lists the action, `status` counts it, and `$ref` resolution ignores the retracted ids — the forward-correction primitive that keeps append-only discipline intact.
5. **Audit every lease in the wave**: count real `tool_use` events for the leased tool in each executor transcript against claimed completions — zero calls with N dones is a fabricated lease. Post-wave read-back sampling must include at least one id from EVERY lease; a global sample can miss an entire fabricating executor.

### Executor MCP-availability race (fresh workflow spawns)

Freshly spawned workflow executors sometimes cannot invoke their leased MCP tool at all — "Cannot invoke MCP tool from current environment" (~36 fail events in one wave) — and an unchanged retry succeeds, implying a tool-catalog load-timing race rather than a permissions or allowlist gap. Doctrine: the executor runs ONE ToolSearch reload of the exact leased tool and retries the call ONCE before appending `fail`; a fail after the retry is honest and final for that lease round (the coordinator's normal retry round covers the rest). This race is precisely the condition the fabrication family papers over — when the retry also fails, a `fail` event is the ONLY honest outcome, never a printed payload declared done. Open question: whether staggering or warming executor spawns removes the race; until measured, the one-retry rule is the mitigation.

## The `halo-apply` workflow (canonical apply driver)

The plugin's frozen `halo-apply` workflow runs the apply loop as deterministic JS: each round computes the ready set from the `needs` graph, partitions leases per the sizing rule, spawns `halo-action-executor` agents (`agentType`), and folds their structured results back into the done-set. Dependency order is enforced structurally — an action dispatches only when every `needs` id is done. (The plugin ships this file at `workflows/halo-apply.js` and Claude Code auto-surfaces it as the invocable `grcitsm-analyst:halo-apply` — invoke it directly; **no copy into the project is needed**. The plugin is the single source of truth, so it never drifts on plugin updates. See SETUP.)

- Args: `{graph (slim: id/tool/needs/guarded), doneIds (seeded from the ledger), runFolder, actionsFile, progressFile, helper, prefix}`. Workflow args can arrive as a JSON **string** — the script's defensive `JSON.parse` guard is mandatory.
- A dead executor's lease returns to the pool next round (guards make redo safe) — kill/reassign for free; 2-strike fail counting surfaces hard fails.
- Resume after any interruption: relaunch with `doneIds` regenerated from `progress.jsonl`. The ledger is truth; the workflow is stateless between runs.
- Permissions: the Workflow approval is the single gate (internal `agent()` spawns don't re-prompt); the write tools themselves must be allowed in the session, or every executor blocks.

## Coordinator dispatch rules

**The coordinator is a ROLE, not a dispatchable subagent.** Subagents cannot spawn agents, so a dispatched coordinator cannot field an executor fleet — a backgrounded coordinator that improvises headless `claude -p` shell children gets silent death, no TaskStop control, and credentials echoed into transcripts. The main thread loads the role spec and drives the loop itself — directly, or via the `halo-apply` workflow.

NEVER spawn executors as headless `claude -p` shell children — no TaskStop control, silent death, credentials echo into transcripts.

## Planner authoring traps (Halo-generic — encoded as structure, never prose)

These apply to any Halo tenant; the planner encodes them as dependency edges and payload shape. Payload-shape rules themselves are corpus-owned — the planner's job is to turn them into `needs` edges. The skill carries the one-line summary; this is the full list:

- **Replace-semantics arrays** (`set_ticket_assets`, `set_asset_relationships`, multi-select CFs): shape rules in `reference-docs/halo-platform/write-semantics.md` §2/§12/§14. Pipeline encoding: if an event runbook auto-appends rows after a write, a later set-call must include them — model the union as its own action with `needs` on the trigger, or prefer omit-preserve payloads over full-replace.
- **Asynchronous derive + revert**: when a tenant runs event runbooks that auto-derive fields on a triggering write, a manual write to those same fields must `needs`-follow the trigger, and the plan must not combine the trigger and the manual override in one action (the trigger can revert the override seconds later, async).
- **Ordered multi-pass fields** (framework CFs, baseline-then-framework: `write-semantics.md` §5-§6): sequence them with `needs` — the filter-source action precedes the dependent action; a prerequisite scalar precedes the fields it gates, else silent drop.
- **Content-filter hang**: exploit/attack strings in free-text hang a write at an upstream content filter with no error — the call simply never returns. The trigger is the protocol-style SUBSTRING, not the complete exploit: bare `jndi:ldap` hangs just like `${jndi:...}`. The planner lints all create/update free-text against a defang denylist before sealing the file: `jndi:`, `ldap://`, `${...}` lookup patterns, `<script`, and kin — describe the technique ("JNDI-style lookup strings (defanged)"), never quote it.
- **Payload cliff / field length caps** (`write-semantics.md` §8): plan a truncated create + the full long-form value in a follow-up action; one long-form field per call when in doubt.
