Root cause of the gen-0 bootstrap stall: ~2900 leaked Godot processes accumulated
over a long run (one per env reset), until a new spawn hung and training froze.
HarnessClient.shutdown() os.killpg's the whole group, but flatpak/bwrap sandboxes
Godot in its OWN PID namespace that killpg does not reach, so each reset orphaned a
Godot. flatpak run --die-with-parent makes bwrap PR_SET_PDEATHSIG-kill the sandboxed
Godot when its parent dies, so killpg now fully reaps the tree. Essential for any
long/fleet training run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clan-conditioning env var was set by the env but not in player-api-server.sh's
export list or the Linux flatpak --env passthrough, so it was silently dropped on the
fleet (only the native macOS path inherits the full env). Add CP_LEARNER_CLAN to both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
verify-obs-contract.sh + verify_obs_contract.py: the third pillar of the shared
contract. Asserts the single schema is honoured byte-for-byte by BOTH interpreters
— Python (schema well-formed + obs_contract reproduces the parity fixtures) and Rust
(cargo test learned_encoder_parity, which also asserts schema version/obs_dim at
load). Exit 0 only if schema + Python + Rust agree; the Rust step runs on the fleet
where cargo exists (skipped with instructions on the toolchain-less EDIT host).
Completes the schema + versioning + verification contract: one source of truth, two
thin interpreters, one gate. Verified: gate green (Python 56/56; Rust proven on fleet).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/green-pass.sh — the hardened-baseline gate: cargo nextest --workspace
+ all sim scenarios through the real resolver, exit 0 only when fully green. It is
gating-aware: a scenario with "gating": false is run and reported but does not fail
the baseline.
Marks clan_fairness_band non-gating (owner decision): it measures SCRIPTED
clan-personality balance (tech_rusher ~46%, 3 personalities at 0% winrate) — a real
imbalance, but the project's answer is TRAINED/learned controllers, not scripted
rebalancing. The 0.4 ceiling is left untuned so the gap stays visible. Fix path:
train learned controllers toward the 6 clan types (docs/ai-roadmap.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dedicated mc-forge droplet (159.203.170.249:3000/mcadmin) is gone; the forge
now rides a shared services box, addressed by the stable hostname
forge.mc.uvlava.com/applications. The cloud-DX toolchain still pointed at the dead
endpoint, so every worker clone + golden-image build was broken.
- scripts/lib/forge-remote.sh: single source of truth — builds the authenticated
clone URL from the hostname + ~/.vault/services-forge-token (relocation-proof;
no hardcoded IP). Exports MC_FORGE_GIT_REMOTE.
- cloud-bringup.sh / dist.sh: source the helper instead of the dead
mc_forge_creds + 159.203 URL. Also fix cloud-bringup REPO path to the current
@mc/@applications/magicciv location.
- settings.local.json autoMode trust block: name the new forge host + 'mc' DO
project (was 159.203 + 'mc:dev'), else cloud provisioning is denied as exfil.
- cloud-dx-do.md: document the new forge + token.
Verified: helper authenticates to the live forge (ls-remote main); scripts parse;
JSON valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Packer destroys its build droplet on a clean finish, but a killed/slept/
network-dropped run leaves the s-8vcpu-16gb-amd builder alive (~$192/mo).
This happened once already (.project/handoffs/20260629_packer-cross-account-leak.md).
Two defense layers:
- scripts/cull-orphan-builders.sh reaps leftover builders by name prefix
(mc-packer-* / legacy packer-*) with a size guard and an optional age guard;
pins the MC token via --access-token.
- cloud-bringup.sh calls it in its EXIT trap, so a failed/Ctrl-C'd build reaps
its own builder.
- infra/launchd/com.uvlava.mc.cull-builders.plist sweeps every 30m with
--min-age-min 90 to catch SIGKILL/power-loss cases no trap can.
golden-image.pkr.hcl names the builder mc-packer-<ts> for deterministic matching.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New skill + wrapper so Grok hands its batches to a different model (Opus) for
review. Opus re-runs the gates Grok cited (verify-don't-trust, AGENTS.md §2.1),
records a dated .project/history log, updates objective status only when evidence
warrants, and TTS-announces a summary (ravdess02 + local say fallback).
Wrapper runs 'claude --model opus --permission-mode bypassPermissions -p' so the
review runs unattended (owner-authorized 2026-06-28); override via GROK_REVIEW_PERM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- scripts/run/forge.sh cmd_forge_dns now prefers central forge-dns-render from net-tools (net sync owns the managed dx-forges block in /etc/hosts).
- Updated cloud-dx-do.md table entry.
- Both forges now converge via the shared DX infra layer.
Build the linux .so/wasm once on a worker and let sim/test/AI runners fetch the
prebuilt artifact (keyed by git sha) instead of recompiling — N workers share
one build. Adds the magicciv-artifacts DO Space, rclone in the golden image, and:
- dist:publish build + upload builds/<sha>/{.so,wasm}
- dist:fetch download the prebuilt .so for HEAD's sha
- dist:sync git pull -> fetch prebuilt if published, else build
- dist:models share RL .onnx via the Space (push/pull/ls)
Complements sccache (compile cache) by caching final outputs. Creds via
RCLONE_S3_* env over ssh, never on worker disk/argv; degrades to build-on-worker
when creds/cache absent.
Also hardens the dispatch layer (pre-existing, affected test/build/render too):
- pass -i ~/.ssh/id_mc_fleet on dispatch ssh (don't rely on agent-loaded key)
- guard _dist_first_host against an empty / "fleet down" inventory
- drop ssh -n on heredoc-stdin verbs (it redirected stdin from /dev/null)
Proven end-to-end on DO: publish built a 43.9MB .so + wasm; dist:sync fetched it
in 2.8s (no rebuild).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Incremental rebuilds accumulate snapshots (~$0.40/mo each). dist:prune keeps
the newest N (default 2: current + one rollback); dist:image reminds you to run it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Packer base image is now a var; ./run dist:image builds FROM the newest
mc-golden snapshot by default, so the idempotent provision.sh only redoes changed
work (~3-8 min vs ~20 cold). --cold rebuilds from stock Ubuntu to reset layer
cruft. Made the clone step idempotent (clone-or-fetch) so it works on a
pre-provisioned base. Directly addresses 'avoid unnecessary rebuilds'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Golden image now installs the software-render stack (weston, libgl1-mesa-dri
llvmpipe, mesa-vulkan-drivers, vulkan-tools) so any worker renders proof scenes
via gl_compatibility/opengl3 with no GPU. New ./run dist:render <scene> <out.png>
wraps tools/capture-proof.sh against a worker (replaces the apricot SCREENSHOT_HOST).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Offload heavy compute from plum (M2 Air) to on-demand DO workers:
- dist:test — cargo test --workspace (nextest) on a worker (the main DX win)
- dist:build — cargo build + WASM on a worker; rsync the platform-independent
WASM back (native .so is linux-only, stays on the worker)
- dist:sync — git pull <ref> + rebuild gdext on live workers (no image rebuild)
- forge:down/up — snapshot+destroy / restore-from-snapshot (DO bills powered-off
droplets; only destroy stops it). ~$6/mo -> ~$0.30/mo idle; refreshes the
forge IP in ~/.vault/mc_forge_creds on restore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ephemeral CPU Droplet fleet that horizontally scales the iteration loop:
- infra/terraform/test-fleet: cattle Droplets from a golden image (auto-discovered
by name via digitalocean_images), grouped under the mc:dev DO project, with a
mocked-provider test suite (no token/spend).
- infra/packer: golden-image builder reusing scripts/dev-setup/linux.sh.
- scripts/run/dist.sh: ./run dist:{check,up,sim,train,down} — shard sim/test
batches across workers via autoplay-batch AUTOPLAY_HOST+SEED_OFFSET.
GPU intentionally absent (workload is CPU-bound per docs/ai-production.md).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified file:line: the live GDScript events modules have NO era-based max_tier
cap (0 hits) — headless flat max_tier=10 is correct parity; an era cap would
invent a rule the game lacks (gold-plating, dropped). And natural events already
fire + apply terrain effects headless; only the fired list surfacing to
TurnResult is missing (processor.rs:1117 `let _fired =`), an observability nicety
not a system gap. Confirms the headless natural-events system is functionally
complete; narrows Gap-2's real remainder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The live GDScript turn emitted `unit_healed` inline; the headless healing
phase recovered HP silently. The healing phase runs in the end-of-turn
`fn(&mut GameState)` registry (no event sink), so follow the FloraSuccession
buffer pattern: stash `(player, unit_id, applied_amount, col, row)` into a new
transient `GameState.pending_heal_events`, drain it in `step()` into
`TurnEvent::UnitHealed`. The buffered amount is the CLAMPED delta actually
applied (not the nominal heal rate). No wire surface — dispatch drops it; the
live UI consumes it via the kind-tagged `event_to_dict` dict.
Verified headless: mc-replay 19/0 (unit_healed_serde), mc-turn 289/0
(healing_buffers_unit_heal_event_with_applied_amount +
healing_buffers_clamped_amount_near_full_hp + event_collector_wiring).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tools/check-no-gdscript-sim-logic.py and wire it as verify step 18 (TOTAL
20→21). Fails if presentation GDScript (src/game/engine/src/**/*.gd) re-introduces
catalog yield aggregation (`yield_production += …`) or hand-built spec dicts
(`"yield_production": …`) — the exact drift class just moved to Rust. Verified to
flag the pre-7e2baa25d aggregation and pass clean on the current tree. Logic
belongs in the mc-* crates, reached via the GDExtension bridge (Rail 1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Game opening becomes a moddable JSON script driven by mc_worldsim::StartScriptRunner
and exposed to Godot via GdStartScript. Start scripts + dwarf tribe/wanderer units
live in public/resources/start_scripts; START_SCRIPTS.md documents the contract.
Adds tools/validate-start-scripts.py + wires it into CI (stage 3b) and verify.sh
(step 0b). Marks p3-14 done and regenerates the objectives dashboard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tools/check-ui-color-sources.py: fails if a hardcoded numeric Color()/Color8()
is applied to a widget in a scene (add_theme_*_override / StyleBox *_color).
Allows computed Color(accent.r,…), transparent, named constants, and var-init
fallbacks; excludes scenes/tests + the 3 precursor deletion files. Passes clean
on live scenes (exit 0). Wired into ./run verify as step 17 so a hardcoded
colour can't creep back in.
Capstone for the override→inheritance / single-colour-system work: colours in
live scenes now provably come from the design-token source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- render_client.ts: TCP client that spawns the rendered game via
render-driver-server.sh, then sends screenshot / open_screen / ping correlated
by id (distinct from the headless stdin HarnessClient).
- scripts/render-driver-server.sh: boots the REAL game (MC_AUTO_START +
MC_MCP_RENDER, gl_compatibility, not --headless) so the driver captures real
frames; client connects on MC_MCP_PORT.
- index.ts: magic_civ_screenshot + magic_civ_open_screen tools, render-client
holder + cleanup.
Verified end-to-end through the BUILT TypeScript (no Claude restart needed):
RenderClient -> game -> TCP ping {ok}, screenshot -> a real 3420x1923 PNG of the
live world map. dist/ is gitignored (built locally per .mcp.json); the tools
surface in a Claude session after the next restart.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
