# demo-scaffold — the stored demo dataset

The demo system's content, stored as flat CSV in the repo so it can be read, diffed, reviewed
in a spreadsheet, and **replayed into any tenant**. Before this existed, the demo dataset was
re-improvised on every run — prose variety contracts in the `demo-smoke-test` skill plus one-off
generators like `outputs/ssp-spine-rebuild-2026-07-08/gen_phase3_actions.py`. The rows themselves
only existed in whichever tenant last ran the build, so "reset the tickets" meant "rebuild them
from scratch and hope they come out the same."

The scaffold is the durable answer: **the CSVs are the demo data**, and the tenant is a
projection of them.

## The contract

| Layer | What it is |
|---|---|
| `run-config.csv` | Instance-agnostic settings by NAME (client, site, agent, requester, baseline, SLAs) |
| `system/system-information.csv` | The System Information panel — the system definition everything else derives from |
| `system/ports-protocols.csv` | Ports, Protocols & Services (SSP §10), upserted per row |
| `system/client-details.csv` | Client/Organization values — **no MCP write path**, applied by hand |
| `assets/cloud-assets.csv` | Cloud Assets (type 188), one row per asset, `AF*` columns |
| `ssp/components.csv` | Components (173), **library rows only** — a library key, not authored content |
| `ssp/is-manifest.csv` | Implementation Statements (191), **library rows only** |
| `tickets/*.csv` | Issues, VDs, CRs, UARs, Incidents, Validations — one row per ticket, `CF*` columns |
| `tickets/validation-runs.csv` | Validation run rows (the child table), upserted by Validation Key |
| `narratives/*.md` | Long prose. Any cell may hold `@file.md` to pull its value from here |
| `lookups/*.csv` | The only tenant-specific files — regenerated per instance, never hand-edited |

Three rules make this portable:

1. **Symbolic keys, never tenant ids.** Every row has a `key`; `parent_key`, `asset_keys`, and
   `target_issue_key` reference other rows by that key. No ticket or asset id appears in any CSV.
   The same scaffold applies to Dev, UAT, Gov, and training clones.
2. **Names, not ids, for tenant objects.** Agent, status, SLA, priority, category, requester are
   written as the strings a human reads. `lookups/id-map.csv` resolves them at compile time.
3. **Structural data in cells, prose in files.** A cell holds a short value; anything
   multi-paragraph lives in `narratives/` behind an `@file.md` reference.

`run-config.csv` has three columns — `setting`, `value`, `notes` — and **only `value` is
machine-read.** `notes` is provenance and context for the human editing the row: it may cite the
Dev instance's ids and history, and those citations are non-binding and never resolved. An id that
appears in `value` is a portability defect; an id that appears in `notes` is a footnote.

Column values for every lookup field are validated against the verbatim allowed-value lists in
`scripts/demo_scaffold_schema.py` (transcribed from the live-verified golden field catalog).
This matters more than it looks: Halo does **not** error on an unknown lookup value, it silently
drops it. `check` is the only thing standing between a typo and a field that looks written but
isn't.

## Running it

`demo_compile.py` (in `scripts/` beside this scaffold in the `demo-build` skill) is pure and offline — it
never calls MCP. Tenant-derived inputs arrive as files an agent produced upstream (same discipline
as `ssp_compile.py`).

Work on a COPY of this directory. Plugin dirs under `~/.claude/plugins/` are re-cloned by
`/plugin` operations, so edits made in place are disposable.

```bash
SK="$CLAUDE_PLUGIN_ROOT/skills/demo-build"
WORK=outputs/demo-rebuild-<date>

# validate the scaffold — row-level errors, non-zero exit, emits nothing
python3 "$SK/scripts/demo_compile.py" check --scaffold "$WORK/scaffold"

# compile to halo-action-pipeline step files
python3 "$SK/scripts/demo_compile.py" compile --scaffold "$WORK/scaffold" --out "$WORK/state"
```

Lookup paths default to `<scaffold>/lookups/*`. The copy shipped in the plugin has headers only —
regenerate it against the target tenant first (see **Regenerating the lookups** below).

Compile emits ordered step files, then `/grcitsm-analyst:halo-apply` runs each one:

| Step | File | Notes |
|---|---|---|
| 00 | `actions-00-system.jsonl` | `update_system_information` + `update_ports_protocols` — the system definition, written first |
| 01 | `actions-01-cloud-assets.jsonl` | create → fields → rel 9 to Component |
| 02 | `actions-02-tickets.jsonl` | the operational tickets: create → site → agent → **category alone** → CFs → SLA → assets → parent |
| 03 | `actions-03-validations.jsonl` | the 226 per-check Validation tickets — **optional per build** |
| 04 | `actions-04-validation-runs.jsonl` | child-table run rows, upserted by key 627 — **optional per build** |

Steps 03 and 04 are emitted only when the build opted into the KSI validation population.
`--skip-validations` on `check` and `compile` leaves both out; the set references nothing in
the operational tickets, so it can be added by a later run against the same build.

Two ordering facts are baked into the emission and are not stylistic:

- **Category is written in its own call, before the CF payload.** Framework CFs auto-derive only
  on category change, and the CF call deliberately omits category so the derive is not re-fired.
- **Asset links resolve to concrete ids from `lookups/snapshot.csv`.** `asset_ids_json` and
  `relationships_json` are serialized strings, so a `$ref` cannot resolve inside them. Apply step
  01, refresh the snapshot with the new asset ids, then recompile step 02. Compile fails loud
  rather than emitting an unresolvable link.

## Every ticket lands on New

No ticket CSV has a `status` column. **Every** ticket is created on New
(`run-config.new_status`) and a human drives it from there — UAR approval, the
Evaluation → IIR → OIR → FIR → AAR incident workflow, deviation adjudication and the cascade it
fires, issue remediation and POA&M close-out, change approval.

This is the point, not a limitation. The process *is* the demo and the approval is the evidence; a
seeded end-state would be a record of work that never happened. It also means the scaffold carries
no stage spreads, which is most of why it's small.

Baked into the schema rather than left to discipline:

- **`check` errors on any `status` value in a ticket CSV**, and compile always writes
  `run-config.new_status`. (Asset status is unaffected — `Active` on a Cloud Asset is a lifecycle
  state, not a workflow stage. `validation-runs.status` is the run result, Pass/Fail.)
- **Incidents declare a `ticket_type` of `Alert` or `Incident`.** Alert → Incident is an in-place
  *type flip*, not two entities, so a row says which type it starts life as and the human escalates
  it. Correlation still works through `parent_key`.
- **Fields the human produces are flagged.** Pre-filling `CFFinalReportSentAt`,
  `CFLessonsLearned`, `CFCorrectiveActions`, `CFrootcauses`, or recovery/reporting timestamps
  contradicts a New ticket, so `check` warns. Evaluation-tier fields (PAIN, detected/started at,
  detection source) are fine to seed — they exist before the response begins.

## The system definition (step 00)

The system is authored here and written, not treated as pre-existing ground truth — a rebuild into
a wiped or fresh tenant has to be able to stand the system up. Everything else in the scaffold
derives from these narratives, so author them first.

`update_system_information` has an unusual write contract, and the compiler resolves it from
`lookups/site-field-map.csv` + `lookups/site-lookup-values.csv`. `customfields[]` is keyed by
**integer field id**, and the value shape varies per CF type:

| CF type | Shape | Note |
|---|---|---|
| 0 / 1 free-form | `{id, value: "string"}` | |
| 2 single lookup | `{id, value: <fcode int>}` | a bare scalar — not an array, not `{id,name}` |
| 3 multi-select | `{id, value: [{value, label}]}` | never a CSV string |
| 4 date | `{id, value: "YYYY-MM-DDT00:00:00"}` | |

A lookup write needs the **fcode**, not just the label; a label-only write echoes `display=""` and
does not persist. Compile fails loud on a missing fcode or an unmapped field id rather than
emitting a write that would silently no-op.

**HTML comments in narrative files are stripped before writing**, and a companion file containing
only comments counts as *unauthored* — it is skipped with a warning rather than publishing
placeholder scaffolding into a live field.

**System POCs are out of scope.** System Owner / ISSO / Incident POC are established by Halo
**Agent Role assignment**, not by any Site-level write, so the scaffold neither authors nor applies
them. Security Contact and Sales Contact are different — they are Site contact *fields*, so they
live in `system-information.csv` and are written normally. Read POC state with `system_pocs`.

**No MCP write path exists for the Client record.** `client_details` is read-only and there is no
client-update tool, so compile emits `manual-setup.md` for a human to set in the UI and verify on
read-back. A client-update MCP tool is a genuine gap worth filing.

The SSP library rows (`ssp/`) conform to the existing `ssp-csv-compile` manifest contract and are
compiled by **that** skill's `ssp_compile.py` — deliberately not reimplemented here, so the
library-first chain and the three-phase framework-CF write stay in one place.

## Resetting the tickets

MCP cannot hard-delete a ticket, so reset is two moves:

1. Delete the target site's tickets in the Halo UI (the destructive step stays human and manual).
2. Re-apply the step files. Every create carries a guard (`search` on the ticket summary /
   asset inventory number), so a **partial** wipe is safe: surviving rows are skipped, missing
   ones are recreated. That guard is also what makes an interrupted apply resumable.

Because the guard matches on summary, a guard hit against data you *meant* to wipe means the wipe
was incomplete — not that the row should be skipped. Verify the site is empty before a full rebuild.

## Regenerating the lookups

`lookups/` is the only per-instance surface. Regenerate it against the target tenant (read-only
MCP calls) whenever you point the scaffold at a new instance or the tenant's config changes:

| File | Source |
|---|---|
| `id-map.csv` | `client_details`, `agents`, `statuses` (TICKET statuses), `asset_type_allowed_statuses` (ASSET statuses under their own `kind=asset_status` — the names collide: `Rejected` = ticket 19, asset 10), `slas_and_priorities`, `users` |
| `asset-field-map.csv` | `asset_type_fields` (unparameterized — filter locally). CANONICAL format: `assettype_id,field_name,field_id,prefix`, field_name UNPREFIXED, covering type 188 AND the spine types; the system-package driver derives each compiler's variant |
| `snapshot.csv` | `asset_summary` / `ticket_summary` — existing keys → ids |
| `pps-snapshot.csv` | `get_ports_protocols` — `(service,port,protocol,purpose) → port_protocol_id`. Without it EVERY PPS row INSERTs, which duplicates a populated CTPortsProtocols table silently (36-from-17, measured 2026-08-10; MCP has no PPS delete) |

The `users` rows carry one trap the others don't: **scope the lookup to the run's client and
site.** `run-config.default_requester` names the site's `Is General User: Yes` account, and that
display name repeats across sites and clients, so an unscoped name→id lookup resolves to some
other site's account without erroring. Filter `users` by the run-config `client_name` +
`site_name`, take the `Is General User = Yes` row, and write it to `id-map.csv` as `kind=user`.

### Precondition: the KSI validation L2 parents

**Applies only to a build that opted into the validation population.** With
`--skip-validations`, the `validations` family is never read, the external parent resolution
below is never evaluated, and no sweep or snapshot ticket row is required.

`validations.csv` rows carry a `parent_key` naming a KSI family (`KSI-IAM-AAM`), which is an
**external parent** — a ticket the scaffold does not author. Those per-KSI L2 validation tickets
are created on the Site by the tenant's own `create_validation_tickets` MCP sweep (zero-param and
idempotent; re-running it creates nothing new). Before compiling step 03:

1. Run `create_validation_tickets` against the target Site.
2. Add each resulting L2 ticket to `lookups/snapshot.csv` as `ticket,<KSI id>,<ticket id>` —
   for example `ticket,KSI-IAM-AAM,41822`.

Compile hard-fails without them, one named error per unresolved row (226 of them on the shipped
set). That is the designed fail-loud, not a bug: a validation child silently created without its
KSI parent is invisible in the family rollup the demo is built to show.

## Intended size

Small on purpose for the hand-authored ticket types — the scaffold is a demonstration corpus, not a
load test, and every hand-written row costs a human read when it's reviewed and a walk-through when
it's demoed. The two Validation files are the deliberate exception: they are machine-generated from
the compliance mod and reviewed as a generator, not row by row.

| Entity | Rows | Covers |
|---|---|---|
| Cloud Assets | 6 | AWS + Azure Gov, public/private, VM / managed DB / container |
| Components (library) | 4 | enough to hang ISs and asset rel-9 from |
| Implementation Statements (library) | 4 | one per Component |
| Change Requests | 4 | one per change type that feeds a distinct OCR section |
| Issues | 8 | umbrella + 2 children, standalone CVE, misconfig, web-app, both SLA families |
| Deviations | 4 | all four DR types (FP / RA / OR / OR RA) |
| UARs | 3 | account creation, privileged access, recertification |
| Alerts + Incidents | 3 + 2 | 1:1 (inc-unauth-access ← alert-ssh-brute) AND N:1 (inc-tenant-enum gathers alert-inference-anomaly + alert-waf-probe, alerts carry parent_key) |
| Validations *(optional per build)* | 226 | one per steampipe-compliance-mod check, 43 KSI families |
| Validation runs *(optional per build)* | 1,582 | 7 daily runs per check, improving arc, seed-reproducible |

Every stage spread the `demo-smoke-test` variety contract used to ask for (UAR
2 New · 2 Awaiting · 1 Approved · 1 Rejected · 1 Closed; deviation outcomes; closed Issues;
a complete AAR) **no longer applies to the scaffold** — those states are produced by the human
working the process. That is why the row counts dropped roughly four-fold: the scaffold seeds the
starting position, and the walk-through creates the spread.

### Validation provenance + regeneration

The Validation set is vendored, not authored. Its source of truth for **what is checked** is the
`Stratus-Cyber/steampipe-compliance-mod` repo at commit `81074b2` — one Validation row per control
check in the mod (136 AWS, 72 Azure, 18 Entra across 43 KSI families). When the mod gains or
retires a check, regenerate rather than hand-editing the CSV:

```bash
SC=skills/demo-build/scripts

python3 "$SC/vendor_ksi_checks.py" \
  --mod   /path/to/steampipe-compliance-mod \
  --out   skills/demo-build/scaffold/tickets/validations.csv

python3 "$SC/generate_validation_runs.py" \
  --validations  skills/demo-build/scaffold/tickets/validations.csv \
  --out          skills/demo-build/scaffold/tickets/validation-runs.csv \
  --week-ending  2026-08-08
```

`vendor_ksi_checks.py` fails loud on a check whose title carries no `KSI-XXX-YYY.NN` prefix and on
any `CFValidationID` collision — both would otherwise land as a silently mis-parented or duplicated
row. `generate_validation_runs.py` is deterministic under `--seed` (default `20260808`): same
inputs, byte-identical output.

The stored run week ends **2026-08-08**. Re-stamp it for a fresh demo so the runs read as
"this week" rather than as history:

```bash
python3 "$SC/generate_validation_runs.py" \
  --validations skills/demo-build/scaffold/tickets/validations.csv \
  --out         skills/demo-build/scaffold/tickets/validation-runs.csv \
  --week-ending <a-recent-friday>
```

The arc is generated, not sampled: 40 checks failing on day 1 fall to 6 by day 7, with the
remediations clumped by theme (identity, tagging, encryption, logging, backup) across the week and
the surviving failures written to match the demo's open Issues.

## Demo-data quality standards (the authoring contract)

The dataset is BOTH living demo data and the training exercise corpus: every ticket must read like
a practitioner wrote it, and every type-specific surface must be populated. These standards govern
the CSVs themselves — a row that misses them is a scaffold defect, not a run-time one.

**Completeness contract (every ticket row):**

- `agent` + `category` + `requester` set (+ `team` on the ConMon types); an SLA on the SLA-based
  types (Issues, Data Requests)
- every type-specific `CF*` column populated per `ticket_type_fields` — no blank POA&M / DR / CR /
  UAR-detail cells
- `summary` + `description` practitioner-real and defanged; UAR summaries are MACHINE-COMPOSED from
  the detail CFs, so set the CFs and never author summary text to fight them
- category is authored so the 6-dimension attribution derives on apply, with Requirements landing
  NON-EMPTY (if a read-back shows them empty, re-send `category_1` to re-fire the derive — the
  fix is at apply time, the authoring obligation is to set the category)
- `asset_keys` populated wherever the type supports assets; `parent_key` wired per the reference
  hierarchy

**Variety contract (across each type's set):**

| Type | Vary across the set | Cover at least once |
|---|---|---|
| UARs (3) | requesting user, target system (Entra/AWS/Azure Gov), AM category | account creation / privileged access / recertification. No stage spread — UARs land on New and the human works the approval, which is the demo |
| Issues (8) | CVE vs misconfig vs web-app weakness; scanner variety; CVSS severity spread; target dates spread over weeks; umbrella vs standalone | both SLAs — ONE SLA per theme family, never mixed within a family; chains parented by a deviation and by the patch-window CR. All land on New |
| Deviations (4) | all four DR types FP / RA / OR / OR-RA; distinct rationales | No outcome spread — all land on New; the human adjudicates, which is what exercises the cascade (incl. the Rejected no-cascade path) |
| CRs (4) | patch / transformative / maintenance / misc; outage boolean both ways | cross-type parenting (patch window parents the vuln umbrellas). All land on New |
| Alerts + Incidents (3 + 2) | alert sources, correlation patterns (1:1 escalation: incident.parent_key = alert; N:1 correlation: each alert.parent_key = the gathering incident) | Alert vs Incident as a `ticket_type` on the row (in-place type flip). All land on New with evaluation-tier fields only; the human drives Evaluation → IIR → OIR → FIR → AAR, so the AAR is produced by the walk-through, not seeded |

A gap against these standards is a scaffold edit — fix the CSV and recompile — never a per-run
improvisation.

## Scope

In scope: Cloud Assets, a handful of library-sourced Components and Implementation Statements,
and the operational ticket set (Issues, VDs, CRs, UARs, Incidents, Validations + runs).

Out of scope by design: the reconciler-owned SSP spine (Capabilities 178, Control Implementations
192, Implemented Requirements 190, Set Parameters 193) and the relationship reconcilers. Those are
materialized by `reconcile_ssp_assets` / `reconcile_ssp_relationships_*`. Hand-creating them
produces AFUUID-mismatched duplicates.

`system_information` is **in** scope and is written first, as compile step 00 — see **The system
definition (step 00)** above. (Until 2026-08-06 it was treated as read-only ground truth; that rule
was reversed so a rebuild into a wiped or fresh tenant can stand the system up.)
