RustNotMC/docs/release-flow.md

# Release flow

How a change gets from a pull request to players: the checks a pull request passes, how it merges, and the
automatic releases that follow. Releases are automatic and frequent so the latest Horizon is always playable.
There is one channel ([One channel](#one-channel)): each release is published as the release and GitHub's *latest*
once its smoke test passed and no `release-blocker` is open, and launchers and servers install it at once.

## Pull request checks

The `checks` workflow (`.github/workflows/ci.yml`) runs two jobs on every pull request (opened, pushed to or
reopened; labels never start it):

| Job | Runs on | What | Time |
| --- | --- | --- | --- |
| `verify` | GitHub `ubuntu-latest` | Public-tree check, Python tests, rustfmt, clippy with warnings denied, Rust tests. Dependencies and the workspace build come from a cache saved by main's runs | ~4 min cold, less warm |
| `parity / parity` | A runner with the label `horizon-game`: the Mac mini's and playhorizon's (below) | Only for a change to a harness, an oracle, a probe or the runner (or the `run-parity` label): the vanilla harnesses the change can affect (`tools/parity.py --changed <base>`). Other pull requests skip it: [parity-main](#parity-on-main) checks them on main | seconds when nothing is affected; minutes per affected harness |

Pushes to main run `verify` only; the nightly audit covers main's harnesses.

`verify` is the check main requires, so it has no condition and runs in every run of `checks`: GitHub counts a
*skipped* required check as passed, and labels used to start runs whose `verify` was skipped, which let
auto-merge merge before the real `verify` finished. A pull request that changes no Rust (docs, tools) runs only its
quick steps and finishes in seconds.

### The parity job

The job is `.github/workflows/parity.yml`, a reusable workflow: `checks` calls it for pull requests from this
repository, `run-parity.yml` for a fork's pull request a maintainer labels `run-parity`.

A pull request every harness reaches (a crate split moves files every probe links: #2160 ran all 352 harnesses, 71
minutes on a Mac mini runner, while five other pull requests' jobs queued behind it) carries the `full-parity` label:
its job runs on playhorizon's runner only (4 harnesses at a time), and waits for it while that runner is busy
instead of taking the Mac mini's. Labels don't start `checks`: add it when opening the pull request
(`gh pr create --label full-parity`), or before the next push.

Horizon never downloads the game, so harnesses that compare with vanilla run only where the maintainer's own
copy is: the Mac mini and playhorizon (below). Their runners run as the maintainer's account and keep their checkout
between runs (`clean: false`), so the probes build incrementally into `.local/parity-target` and earlier passes
stand.

The job tests the pull request's head merged into main's current tip, which is what merging it would land: harness
and game fixes that reached main after the branch was made apply to it too, and an old branch runs main's runner
scripts and selection unless it changes them. The base for the selection is that main commit, so only the pull
request's own changes count. The job summary names the main commit. A head that conflicts with main is tested
alone, with a warning. The check is still reported on the head.

1. `tools/ci/runner-env.sh` puts the machine's Python 3.13 and the JDK `research-lock.json` pins on the PATH
   (Homebrew and `java_home` on the Mac; on Linux what `linux-runner.sh` pinned, else nixpkgs), and sccache when
   the machine has it.
2. `tools/ci/vanilla-data.sh` links `.local/` to the runner's copy of the vanilla data, cloned once from
   `~/code/horizon-parity/shared-local` into `~/.actions-runners/horizon/_work/horizon/shared-local`, and
   prepares `.local/horizon-client` with the pull request's own `horizon-prepare` once per preparer version
   (its sources, the exporters in `mods/horizon/src/test/java` and `research-lock.json`). The game's sounds, which
   depend only on the Minecraft version, are kept once in `shared-local/sounds/<version>/` (copied from a folder
   `horizon-prepare sounds` made from the maintainer's own game; the Mac mini has no official launcher of its own)
   and cloned into each preparation (not linked: vanilla's packs refuse symbolic links), for the audio harnesses.
   `tools/prepare_output.py check` then compares what was prepared with `crates/horizon-prepare/output-versions.txt`:
   launchers keep a player's prepared resources across releases whose preparer declares the same `OUTPUT_VERSION`
   (`crates/horizon-prepare/src/lib.rs`), so a change to the prepared files fails here until it bumps the number
   and appends the new digest (the error gives both lines).
3. `tools/parity.py --changed <base sha>` picks the harnesses whose traced dependencies (their script, Python
   modules, Java oracles and every Rust source their probes compile) changed since the merge base, plus any
   harness nothing is recorded about yet, and runs those that have no pass for their current inputs. A change to
   `Cargo.lock` or `research-lock.json` affects every harness. Each harness has 25 minutes, the job 45.

Failed harnesses' logs are uploaded as the `parity-logs` artifact.

**Platform-specific harnesses.** A harness listed under `linux_only` in `tools/parity_hosts.json` gives
different results on macOS for a reason that is not a Horizon bug (the reason is printed). `parity.py` skips it on
other platforms unless it is named on the command line. Today it holds only `save_roundtrip` (it needs `vscreen`):
`options`, `chunk_generation`, `world_border_render` and `mob_ai`, which once differed on macOS through the
platform's libm, pass on the Mac mini since `horizon_core::jmath` computes what HotSpot computes (checked
2026-10-14).

**Pull requests from forks** never run code on the Mac mini by themselves. A maintainer who has read the change
adds the `run-parity` label: that runs the job once, on the commit the pull request had then. To run it again
after new pushes, remove the label and add it again.

**Two runners.** The Mac mini has two runners for this repository, `ludovics-mac-mini`
(`~/.actions-runners/horizon`, `CARGO_BUILD_JOBS=10`) and `ludovics-mac-mini-2` (`~/.actions-runners/horizon-2`,
`CARGO_BUILD_JOBS=6`), with the same labels, so the 30-minute nightly audit or a long parity job never holds a
release's macOS build back. Each keeps its own checkout, `target/` and Rust install; the second's
`_work/horizon/shared-local` links to the first's, so they share the vanilla data, the preparations (one at a
time, under a lock) and the parity passes and slots.

**The desktop's runner (retired).** On 2026-10-26 the maintainer asked that CI stop using his desktop: its services are
stopped and `ludodesktop-horizon` has no label a job asks for (only `self-hosted`, `Linux`, `X64`), so nothing waits for
it. What follows describes it as it ran. It ran on the maintainer's Linux desktop while nobody used it: its
screen locked, or untouched for 15 minutes (his rule, 2026-10-26). `tools/ci/linux-runner.sh install`, run there as
him, registers it and installs two systemd user services: `horizon-runner` (the runner, in
`~/.actions-runners/horizon`, nice 19, in `horizon-ci.slice` with CPU and I/O weight 20, half the threads (on all of
them the CPU package reached 92–96 °C) and memory capped at two thirds of the RAM; `CARGO_BUILD_JOBS` and the harness slots at 8, after the machine froze under ~7 cold cargo
builds on 2026-10-14; its own Rust install; the JDK, sccache and the build libraries from nixpkgs; its own
`vscreen` so no game it starts reaches his screen) and `horizon-runner-pause` (`tools/ci/runner_pause.py`), the
only thing that starts the runner. The watcher stops it at once on keyboard, mouse or gamepad input (read from
`/dev/input`: only whether bytes came, never what), while an unlocked session has had input in the last 15 minutes,
while the CPU package is at 90 °C or more, while a game runs (Steam, gamescope, Wine, Proton or Minecraft
processes, fleet-load's own detection, or a process outside the CI slice using the GPU at 50 % or more), or while
`~/.actions-runners/horizon/pause` exists, and starts it again 5 minutes after the last such sign. A job already
under way is throttled to 2 threads at the lowest weight and the runner stops after it: freezing the job would make
GitHub drop it as lost and the harnesses time out. Old pull requests' `runner-env.sh` know only the Mac: the
runner's `.env` and `.path` give their jobs Python, the JDK and the vanilla data (`HORIZON_SHARED_SOURCE`) all the
same. The nightly audit and parity-main stay on the Mac mini's runners: the audit compares with the previous night,
so it stays on one platform, and a parity-main job slowed down on the desktop because he came back could time out,
and its failures bisect and comment on merges.

**playhorizon's runner.** `playhorizon-horizon` runs on a server a hosting partner lends the project (Ubuntu, 8
AMD EPYC threads, 30 GB, in Detroit; reachable only through the maintainer's tailnet). Nobody sits at it, so the
watcher never pauses it except by hand. It gets half the machine: `RUNNER_MEMORY=24`, `CARGO_BUILD_JOBS` and the
harness slots at 4, and a drop-in (`~/.config/systemd/user/horizon-runner.service.d/cpus.conf`) that starts it under
`taskset -c 4-7` (the user slice can't set `AllowedCPUs`: systemd doesn't delegate the cpuset controller to it).
Agents' `tools/remote.sh` runs and benchmarks share CPUs 0-3, so they never slow a pull request's job.
Without Nix, `linux-runner.sh` takes the build libraries, the JDK, sccache, Xvfb and Mesa's software Vulkan
(lavapipe; there is no GPU) from apt, and Python 3.13 from uv. The partner can reach the machine, so gh is not
logged in there: the registration token comes from another machine (`RUNNER_TOKEN`), and no job with secrets may
run there (parity has none). It is also a parity host (`playhorizon` in `tools/parity_hosts.json`, 3 jobs on CPUs 0-3). Since
2026-10-26 it has the `horizon-game` label: pull requests' parity jobs run there or on the Mac mini, and `full-parity`
ones only there.

## Merging

Pull requests merge with a merge commit, by hand: never `gh pr merge --auto`. Branch protection on main
requires `verify` for everyone, and Ludovic's review for the files `.github/CODEOWNERS` lists (what builds, signs
and ships releases). Branches don't have to be up to date with main: main's own `checks` run catches the rare
semantic conflict.

**Who merges.** The Horizon Team (GitHub team `horizon-team`: kacpherek, levzzz5154, rdbtCVS, Online5277) and
Ludovic. The orchestrator's agents prove only the risky pull requests (below), so a contributor's ordinary pull
request doesn't wait for them.

**An ordinary pull request merges on CI.** A Team member:

1. reads the whole diff (for a fork, before adding `run-parity`: that runs the fork's code on a maintainer's machine);
2. adds `run-parity` to a fork's pull request; our own branches run `parity` by themselves. Since #2448 it tests
   the pull request merged into main;
3. merges once `verify` and `parity` pass, nobody's change request is open, and every comment has been read (a
   "don't merge" from anyone holds it).

Never merge your own pull request: another Team member reviews and merges it. Never force-push or rebase a
contributor's branch; push a fixup commit, or merge main into it.

**A risky pull request gets the `needs-proof` label** and waits for an agent's local proof (both single-player
paths, `lan_e2e`, a screenshot for anything visible) before it merges. Risky means any of:

- it changes what is saved or loaded (`save.rs`, `entity_nbt.rs`, `block_entity_nbt.rs`, `region.rs`, `datafix/`):
  a bug there damages players' worlds;
- the network or the server thread: `horizon-protocol`, `net.rs`, `remote*.rs`, `lan/`, `lan*.rs`,
  `integrated.rs`, `levels.rs`, `dedicated*`;
- world generation (`worldgen/`): seeds must keep generating the same terrain;
- the launcher's installs and updates, or the mod runtime;
- the harnesses themselves (`tools/test_*_parity.py`, the Java oracles): a wrong harness hides regressions;
- over ~500 lines, or `parity` failed, was skipped or ran no harness for gameplay code.

Unsure? Add `needs-proof`: it costs a few hours, a broken release costs more.

`parity` becomes required once it has run green on pull requests for a while (a week without a false failure):

```sh
gh api -X POST repos/horizon-rust/horizon/branches/main/protection/required_status_checks/contexts -f 'contexts[]=parity / parity'
```

A skipped `parity` (a fork's pull request without the label) counts as passed, so requiring it never blocks a
fork's pull request: the Team member decides whether to run it, and a skipped run makes the pull request risky.

## Parity on main

Most pull requests merge on `verify` alone and leave the vanilla harnesses to `.github/workflows/parity-main.yml`,
which runs them on main a batch of merges at a time, on the Mac mini's runner:

- It starts on every push to main, one run at a time (`concurrency: parity-main`). Merges that land while a run
  goes on wait in one pending run: GitHub keeps only the newest and cancels the others, so the next run takes them
  all.
- `tools/parity_main.py base` gives the newest commit main contains whose parity-main passed (else the last
  merge's parent), and the run is `tools/parity.py --changed <that commit>`: the union of the harnesses every merge
  since then can affect, once, with every earlier pass reused.
- On a failure, `tools/parity_main.py bisect` checks out the batch's first-parent merges by halves and runs only
  the failing harnesses, names the first merge they fail on in the run's summary and comments on its pull request
  (`🤖 Agent: parity-main found that this merge breaks ...`). Fix it with a pull request as usual; the next
  run checks from the last green commit again.
- Game releases ship only a commit both `checks` and `parity-main` passed on (`tools/release_train.py`, see
  below), so a broken merge holds releases instead of reaching players. `release.yml` also starts when a
  parity-main run finishes. A forced release (`force=true`) doesn't wait for parity-main.

A pull request still runs its own parity job when it changes a harness script, an oracle
(`mods/horizon/src/test/`), a probe (`crates/*/examples/`, `game/*_fixture.rs`, `game/*_harness.rs`), the runner
(`tools/parity*.py`, `tools/lab.py`, `tools/parity_hosts.json`, `parity*.yml`), or carries the `run-parity` label
(`run-parity.yml` runs it when the label is added).

## Game releases

`.github/workflows/release.yml` releases the game by itself. It starts after every green `checks` or `parity-main`
run of a push to main (it releases a commit only once both passed on it), every 10 minutes (so a release waiting for merges to settle goes out without another merge), and by hand.

```mermaid
flowchart LR
    pr(["pull request"]) --> checks{"verify<br/>parity"}
    checks -- green --> automerge["auto-merge<br/>(merge commit)"]
    automerge --> main(["main"])
    main --> mainchecks{"checks on main"}
    mainchecks -- green --> decide{"decide<br/>tools/release_train.py"}
    cron(["every hour :17"]) --> decide
    decide -- "player-facing PRs settled 5 min<br/>(or waiting 20 min), or release-now" --> build
    decide -- otherwise --> wait(["wait"])
    subgraph build["build, in parallel"]
        linux["Linux x86_64<br/>GitHub ubuntu-22.04"]
        windows["Windows x64<br/>GitHub windows-latest"]
        mac["macOS arm64<br/>Mac mini runner<br/>+ offscreen game run"]
    end
    build --> gate{"gate: the smoke test drew a world,<br/>no release-blocker is open"}
    gate -- no --> nothing(["nothing is published"])
    gate -- yes --> publish["publish vX.Y.Z as the release and latest<br/>notes from the merged PRs"]
    publish --> launcher(["every launcher and server installs it"])
    publish --> kira(["Kira posts it; the first release<br/>after 20:00 UTC pings, once a day"])
```

### When it releases

`tools/release_train.py decide --kind game` looks at the pull requests merged into main (first-parent merge
commits) since the last **published** game release:

- A pull request counts when it changed what players run: `crates/` (not `examples/`, `tests/`, `benches/`),
  `xtask/`, `packaging/`, `Cargo.toml`, `Cargo.lock`, `research-lock.json`, or an exporter `horizon-prepare`
  compiles (the `mods/horizon/src/test/java/dev/horizonlab/parity/*.java` files its sources name). Markdown never
  counts.
- It releases in batches, so players get a few releases a day rather than one per merge: once 5 pull requests count
  and the newest of them merged at least 5 minutes ago (a wave of merges becomes one release), or once the oldest has
  waited 6 hours, however few they are.
- A pull request labelled `release-now` (a crash or a blocker) releases at once.
- A pull request labelled `skip-release` (work players don't see) doesn't count and isn't listed in the notes.
- An open issue labelled `release-blocker` holds every game release, forced ones too (`held by release-blocker #N`).
- One release at a time (concurrency group `game-release`); a run that finds nothing new does nothing.
- A scheduled or hand-started run releases main's newest commit whose `checks` passed; after `checks`, the commit
  they passed on.
- The first release of a new release line (below) goes out at once, without waiting for a pull request.

### Tags

How releases are numbered, and how installs from before 0.2 reach it: [versioning.md](versioning.md).

- With a release line in `.github/release-line.txt` (`MAJOR.MINOR`, e.g. `0.2`), a game release is `vMAJOR.MINOR.PATCH`,
  the patch after the newest release tagged on that line (`v0.2.0` first). A `feature` label doesn't change the line:
  only that file does, by pull request.
- Without the file (every release before 0.2), `vYYYY.MM.DD`, then `.01`, `.02`… the same UTC day.
- Every reader (`release_train.py`, the launcher, `horizon-server update`, the website) ranks
  every `vX.Y.Z` tag above every dated one.

### Labels

| Label | Effect |
| --- | --- |
| `feature` | Listed under **New** (Kira pings the releases role); makes the next launcher release a minor one |
| `release-now` | Releases as soon as main's checks pass, without waiting for merges to settle |
| `skip-release` | Doesn't trigger a release and isn't listed in the notes |
| `run-parity` | Runs the parity job once on a fork's pull request |
| `full-parity` | Runs the pull request's parity job on playhorizon's runner only (a change every harness reaches) |
| `parity-regression` | Opened by the nightly audit for a harness that fails on main |
| `release-blocker` | On an issue: no game release is published while it is open ([One channel](#one-channel)) |

Pull requests without `feature` are listed under **Fixed**. Titles are written for players, since they become
the notes.

### Builds

| Archive | Runs on | Checks |
| --- | --- | --- |
| `horizon-linux-x86_64.tar.gz` | GitHub `ubuntu-22.04` (an older glibc, so it runs on any recent distribution) | the standard loader and libraries only (`tools/check_imports.py`) |
| `horizon-server-linux-x86_64.tar.gz` | the same Linux job, after the client archive: `horizon-server`, `horizon-prepare` and the game built with `--no-default-features` as `horizon-engine` (docs/server.md) | as above, and neither `libasound.so.2` nor `libudev.so.1` (`tools/check_imports.py --server`) |
| `horizon-windows-x64.zip` | GitHub `windows-latest`, C runtime linked in | system DLLs only (`tools/check_imports.py`) |
| `horizon-macos-arm64.zip` | The Mac mini's runner (`vars.MAC_RUNNER=macos-latest` sends it to GitHub) | system libraries only (`otool -L`); a 1500-frame offscreen run of a new world (about 12 s; 300 frames end before the world has loaded) with resources prepared from the runner's own Minecraft (`--offscreen --world smoke --frames 1500 --smoke-report`: frames drawn, chunks generated and meshed, a player, no error) |

Each builds `horizon-client`, `horizon-ctl` and `horizon-prepare` in release mode, checks all three exist, starts
each one (`--help`, `--version`), packages the folder with `cargo run -p xtask -- package` and archives it. The
GitHub jobs keep their Cargo caches (`Swatinem/rust-cache`); the Mac mini keeps its `target/`. Only the Mac
runs the game: GitHub's runners have neither a GPU worth the name nor the game files.

### Notes

`tools/release_train.py notes` writes them from the same pull requests:

- the title, `Horizon 0.2.3: <the first pull request titles>` (before 0.2, `Horizon 2026.10.15.01: …`, a day's later releases counting .01, .02…, two digits so GitHub's releases page, which sorts them as text, kept them in order), features first, newest first;
- a line's first release (`X.Y.0`) opens with a paragraph that says so; 0.2.0's says releases are numbered from now
  on and how a server that says it is up to date takes it;
- `### New` (label `feature`) and `### Fixed` (the rest), one line per pull request: its title, its number, the
  issues it closes;
- `### Known issues`: the bullets of [docs/known-issues.md](known-issues.md);
- thanks to outside contributors, how to install, the disclaimer.

Kira posts every release once in Discord's #releases. It pings the releases role at most once a day: the first game
release published after 20:00 UTC, with a short list of what came out since its last ping. Every other post is
silent (no ping and Discord's @silent flag). Launcher releases post without a ping.

### One channel

Since 2026-10-23 there is one release channel, and it is every launcher's default. Since 2026-10-27 (Ludovic's
decision, revising 2026-10-23's) players may also turn on [Nightly](#nightly-channel), an opt-in pre-release of main
plus pull requests the trusted reviewers picked, not all reviewed yet; nothing about releases changes for anyone who
doesn't. Every game release is published
as the release, never a pre-release, and marked GitHub's *latest*: the launcher, `horizon-server update` and `install.sh` take it at once. Because nothing
promotes it later, the checks that used to gate a promotion run in the release itself, and the publish job
(`tools/release_train.py gate`) publishes nothing unless both pass:

- the release smoke test: the macOS build runs the game offscreen on the Mac mini and keeps its report,
  `smoke-test.json`, only when the game drew a world. The gate reads the report again before publishing, and the
  release carries it. A run whose macOS build went to GitHub (`MAC_RUNNER=macos-latest`) has no report, so it
  publishes nothing;
- no open issue labelled `release-blocker`. `decide` checks first, so a blocked train builds nothing; the gate checks
  again right before publishing, so an issue labelled while the builds ran still holds the release. Only the label
  and the issue's state count, never its text: whoever opened the issue can edit the text after a maintainer
  labels it. Close the issue (or remove the label) and the next run releases everything waiting, with notes
  covering all of it.

A failed gate fails the run; the next run (10 minutes later) decides again. Nothing older than the latest release is
ever published: tags only count up.

The launcher has no channel setting any more and looks for releases every 30 minutes. The game pre-releases of the
old Alpha channel (v0.2.1 on, published before this change) stay listed for anyone who wants one by hand, but no
launcher installs them as an update, and `release_train.py` ignores them: the first release of the single channel
lists everything since the last release that was not a pre-release (v0.2.0). Launchers from before this change keep
updating either way: with Alpha (or 0.17's "Beta builds") on they take every release, pre-release or not; with it
off, every release that is not a pre-release, which is now every new one.

`promote.yml` and `tools/release_channels.py` (the daily promotion at 20:00 UTC) are gone.

### Signed checksums

Every game and launcher release carries `SHA256SUMS` (the archives' SHA-256, `sha256sum`'s format) and
`SHA256SUMS.sig`: one base64 Ed25519 signature per line over `horizon release <tag>\n` followed by SHA256SUMS,
so a signed list can't be reused under another tag.

- A `sign` job of its own makes them: `tools/ci/sign_release.sh` runs the runner's `sha256sum` and OpenSSL and
  nothing else. It is the only job that holds the private key, the `RELEASE_SIGNING_KEY` secret of the
  repository's `release` environment, which only real releases from main enter (`environment:` is empty in dry
  runs, which sign with a throwaway key). No code from the repository runs beside the key, so no pull request
  can read it by changing what a build compiles; the key is in no file, no log and no password vault, and nobody
  needs to read it.
- The publish job then checks the signature the way launchers do (`horizon-release-sign verify --publishing`,
  crates/horizon-release-sign), against the public keys built in from
  `crates/horizon-release-sign/release-keys.txt`. While that file lists no key, releases go out with SHA256SUMS
  alone and a warning (today's releases keep working). Once it lists one, a missing secret or a key it doesn't
  list fails the release before anything is published, instead of publishing something launchers would refuse.
- Making the key: `tools/ci/new_release_key.sh` pipes a new key from `horizon-release-sign keygen` straight into
  `gh secret set RELEASE_SIGNING_KEY --env release` and prints only the public key, which then goes into
  release-keys.txt by pull request.

**Releases from before signing.** The launcher and `horizon-server update` install only signed releases, with
one exception, so a player or a server can still go back to an older version: every release published before
signing began has its archives' SHA-256 pinned in `crates/horizon-release-sign/legacy-releases.txt`, built into
both. Those releases install only with exactly those files, whatever GitHub serves for them later. The file was
written once by `tools/ci/legacy_releases.py` from the SHA-256s GitHub computed at upload; it lists no signed
release and never changes again (`--check` says whether it is complete).

**Rotating the key** (a new key every year, or at once if the old one may have leaked). The publish job signs
with every key the environment holds, `RELEASE_SIGNING_KEY` and `RELEASE_SIGNING_KEY_NEXT`, and the two names take
turns:

1. `tools/ci/new_release_key.sh <the empty name>` makes the next key; add its public key to release-keys.txt
   beside the old one. From then on every release is signed with both keys (one line each).
2. Wait until launchers and servers built with both keys are in use (a few weeks: launchers update themselves).
3. Delete the old key's secret (`gh secret delete <its name> --env release`) and its line in release-keys.txt.
   The next rotation makes its new key under the name just freed.

If the old key may have leaked, skip the wait: remove its public key from release-keys.txt at once and delete
its secret. Launchers built before the new key cannot check releases signed only with it: they refuse to install
them with a clear message, and players download the launcher again from the website once.

### Starting one by hand

```sh
gh workflow run release --ref <branch>                         # dry run: builds everything, writes the notes, publishes nothing
gh workflow run release --ref main -f dry_run=false            # the usual decision, now
gh workflow run release --ref main -f dry_run=false -f force=true   # release now, whatever the rules (even while paused)
```

A hand-started run on any branch but main is always a dry run. The notes are in the run's summary and its
`release-notes` artifact.

### Pausing

```sh
gh variable set RELEASES_PAUSED --body 1     # no automatic game or launcher releases
gh variable delete RELEASES_PAUSED           # back on; the next hourly run catches up
```

While paused, runs still decide and say why in their summary; `force=true` still releases, but never past an open
`release-blocker`.

### Rolling back

A bad release is withdrawn by turning it back into a draft:

```sh
gh release edit v0.2.3 --draft
```

To stop the next releases while a problem is fixed, label an issue about it `release-blocker`. For the bad
release itself: launchers list only published releases, so the previous version becomes the newest one again, launchers that
didn't install the bad one won't, and players who did can pick the previous version in the launcher's version
list (`launcher/src/backend.rs` tests `withdrawn_release_is_never_offered`). The tag stays, so the next release
that day takes the next number, and its notes start from the last *published* release, so they list the
withdrawn release's changes again with the fix. Usually the better way out is forward: revert the pull request,
label the revert `release-now`, and a fixed release is out as soon as main's checks pass.

## Launcher releases

`.github/workflows/launcher.yml` releases the launcher the same way. It starts on a push to main that changes
`launcher/`, `crates/horizon-prepare/` or `crates/horizon-theme/` (what the launcher builds in), every hour at :47,
and by hand; pushes to `launcher` and `launcher-*` branches only build.

- `tools/release_train.py decide --kind launcher` counts the pull requests merged since the last published
  launcher release that changed those paths (not `launcher/links.json`, which the launcher reads from main, nor
  tests or Markdown), with the same batching rule (5 pull requests settled 5 minutes, or the oldest 6 hours), `release-now`, `skip-release` and `RELEASES_PAUSED`.
- The tag is `launcher@X.Y.Z` on the release line (`launcher@0.2.0` first; the game and the launcher count their
  patches separately). Without `.github/release-line.txt`, `launcher-vX.Y.Z`: the patch after the last launcher tag,
  or the next minor (`X.Y+1.0`) when one of those pull requests is labelled `feature`. `launcher-v0.2.0` was taken
  long before 0.2, hence the new prefix.
- No version-bump commits: the build gets the version in `HORIZON_LAUNCHER_VERSION` and the tag in
  `HORIZON_LAUNCHER_TAG`, which `launcher/src/backend.rs` reads with `option_env!` (falling back to `Cargo.toml`'s),
  so `--version`, the window, the User-Agent and the self-update comparison all use them. Each build checks
  `--version` says it.
- Linux on GitHub `ubuntu-22.04`, Windows on `windows-latest`, macOS on the Mac mini's runner; each runs the
  launcher's tests, checks its libraries, and installs the latest game release with the new launcher.
- Published with `--latest=false`: the repository's latest release stays a game release. Notes list the pull
  requests under New and Fixed; Kira posts launcher releases without pinging.

Making a launcher release GitHub's latest instead would break servers: every `horizon-server` already installed,
`install.sh` and `install.ps1` ask for `releases/latest` and expect the game's archives there, and a server that
can't update can't fetch the fix either. So the game release stays the latest, and it sends people to the launcher
(#1647): its notes open with a "Want to play? Download the Horizon launcher" block (`PLAYERS` in
`tools/release_train.py`: raw HTML, which GitHub shows and launchers' notes skip; the site's Releases page leaves
it out too), and its archives carry labels ("Game files the
launcher installs: Linux (…)") that GitHub shows in place of the file names. Everything that downloads matches
the asset's name, which is unchanged.

Starting by hand, pausing and rolling back work as for the game (`gh workflow run launcher …`,
`RELEASES_PAUSED`, `gh release edit launcher@X.Y.Z --draft`). Launchers older than the withdrawn release then
see the previous one as the newest; launchers that already updated keep running.

With several game releases a day, the newest launcher release soon falls out of the newest 100 releases the
launcher and the website read: both then look the newest launcher tags up one by one
(`git/matching-refs/tags/launcher@`, then `launcher-v` if there is none, then `releases/tags/<tag>`).

## Nightly channel

`.github/workflows/nightly-build.yml` publishes an opt-in build once a night (20:17 UTC) and when a trusted
reviewer asks, for players who want fixes before they are released (#2681). It builds main plus the pull requests the
trusted reviewers picked, merged in their order. With none picked, it builds the newest `union/<agent>-<name>` branch
on origin (main plus the pull requests the contribution gate proved together in one wave) when that union is newer
than main's head and holds something main lacks, otherwise main (`tools/ci/nightly.py pick`). A night with nothing
new publishes nothing. A Nightly can include pull requests not reviewed yet: that is what it is for, playing them
before the gate.

Every Nightly ships its exact source. Its notes name the commit; `horizon-source.tar.gz` on the release is that
commit's `git archive`, signed with the builds in `SHA256SUMS`; and `nightly.py keep-source` tags the commit
`nightly-source/<date>-<short>`. That tag is never pruned, unlike the `nightly-<date>` release and the
`nightly/build-*` branch (7 days each), because native mods pin a Nightly's commit and the launcher downloads that
commit's source to build them ([native-mods.md](native-mods.md)).

### Trusted reviewers

One list, `.github/trusted-reviewers.txt`, governs two things for the people on it (kacpherek, rdbtCVS and
Online5277 since 2026-10-27, levzzz5154 since 2026-10-28) and for Ludovic, with no repository role: `/nightly` (below) and triage by comment.

```text
/close                    closes an issue as completed; a pull request is closed without merging
/close not planned        closes an issue as not planned
/close duplicate of #123  comments "Duplicate of #123", closes as a duplicate
/reopen
/label bug, good first issue    existing labels only; never creates one
/unlabel bug
```

- **Where.** On the first line of a comment on the issue or pull request (`.github/workflows/triage-command.yml`,
  `tools/ci/triage_command.py`). The workflow answers with one comment, `🤖 Triage: <action>, on behalf of @login`,
  linking the comment that asked; a refusal answers too.
- **Refused.** Any label with `release` in its name (`release-now`, `release-blocker`, `skip-release`) and
  `full-parity`; anything on Ludovic's own issues and pull requests; a merged pull request; a locked thread; more
  than 60 actions per reviewer in 24 hours (counted from the workflow's own comments).
- **Safety.** Same as `/nightly`: main's copy of the workflow runs, no pull request's code is checked out or run,
  the comment reaches the script as env vars, and the token has `issues` and `pull-requests` write only.
- **Ludovic stays in charge.** A one-line pull request removing a login from the list revokes it as soon as it merges
  (the file is his under CODEOWNERS). A line `paused` stops `/nightly` and the triage commands for every reviewer;
  his own still work.

### Trusted reviewers pick the Nightly's pull requests

The Nightly is where contributors test pull requests before the gate. The trusted reviewers listed in
`.github/trusted-reviewers.txt` (kacpherek, rdbtCVS and Online5277 since 2026-10-27, levzzz5154 since 2026-10-28) and Ludovic pick them with a
comment on any issue or pull request (`.github/workflows/nightly-command.yml`, `tools/ci/nightly_set.py`):

```text
/nightly 2640 2642     the set becomes these pull requests, merged onto main in this order; builds now
/nightly add 2650      adds one, or takes its newest head again; builds now
/nightly remove 2640   drops one; builds now
/nightly clear         empties the set: back to the proven union or main; builds now
/nightly status        what the set holds
/nightly dry-run 2640  builds main + these and signs with a throwaway key; publishes nothing, the set stays
```

- **The answer.** The workflow answers in the thread with one comment starting `🤖 Nightly:`: the set, the build's
  run, and later the published Nightly (or why it failed).
- **The set** is `set.json` on the `nightly-state` branch: who asked, when, and each pull request at the head it had
  then. A head is taken again only when someone asks again or at the scheduled run, never silently in between. The
  scheduled run takes every pull request's newest head and drops the merged and closed ones; the notes say so.
  `builds.jsonl` there records each build's set (for `nightly_stats.py`).
- **Assembly.** `pick` (main's code) fetches each head and merges them onto main with `git merge-tree
  --write-tree`: git objects only, no working tree, so no file, hook or filter of theirs reaches the runner. The
  result is pushed as `nightly/build-<date>-<n>` and `build` compiles that, without secrets, as for a release.
  Equal inputs give the same commit, so an unchanged set and main publish nothing new. On a conflict the build
  stops and answers which two pull requests conflict and where; a reviewer can then open an integration pull
  request from their fork that combines them, and pick that one instead.
- **Old builds.** The scheduled run deletes the `nightly/build-*` branches older than 7 days (Ludovic,
  2026-10-27), and lists each in its summary. It never deletes the newest build, a build a published `nightly` or
  `nightly-<date>` release points to, or any other branch (`nightly.py prune-builds`).
- **Refused.** A pull request that changes a path CODEOWNERS gives to Ludovic (`.github/`, `tools/ci/`, the
  signing and update code, `launcher/src/backend.rs`, `launcher/links.json`…), more than 30 in a set, and more than
  12 builds a day (the scheduled one always runs; 6 until 2026-10-28).
- **No repository role.** Reviewers need no permission on the repository: the list file and the workflow are the
  whole of it.
- **Ludovic stays in charge.**
  - Ludovic's `/nightly clear` or `/nightly remove` always applies. A pull request he removed can only come back
    when he adds it.
  - A one-line pull request removing a login from `.github/trusted-reviewers.txt` revokes it as soon as it merges
    (the file is his under CODEOWNERS).
  - A line `paused` in that file stops every reviewer's `/nightly` (and triage commands); his own still work.
  - Main, the gate and releases don't change: picks reach only players who turned Nightly on.

- **Jobs.** `pick` chooses or assembles the commit and writes the notes (each pull request in it with its title,
  author and who picked it, or for a union the open pull requests it holds beyond main). `build` is
  `game-build.yml`, the same builds and smoke test as a release, with the version `nightly-<date>` and
  `HORIZON_NIGHTLY="<date> <short sha>"` built in, macOS signed ad hoc. `sign` signs SHA256SUMS for the channel
  (the message says `horizon release nightly`, so a nightly's signature never passes for a release's). `publish`
  checks the signature and the smoke test (not `release-blocker`: a nightly is how blockers get tested), renames the
  previous `nightly` pre-release to `nightly-<its date>`, publishes the new one as the `nightly` pre-release (never
  GitHub's latest), and deletes the `nightly-<date>` pre-releases and tags older than 7 days, nothing else.
- **The signing key never meets the pull requests' code.** Every workflow file and script runs from main's copy (a
  schedule and `issue_comment` run the default branch; `uses: ./…` takes the caller's commit), never a pull
  request's `.github/`, and a pull request that changes `.github/` or `tools/ci/` is refused. `build` compiles the
  merge with a read-only token that isn't kept in the checkout and no secrets; the runner's Developer ID keychain
  stays locked. `sign` is the only job with the key, and checks out only main's `tools/ci/sign_release.sh`, which
  hashes the archives with the runner's `sha256sum` and signs with its OpenSSL: it never runs what `build` made.
  `publish` runs main's `horizon-release-sign`, never the archives. Nothing a nightly's build leaves reaches a
  release either: `game-build.yml`'s `trusted` input is true only for `release.yml`. An untrusted build runs its
  macOS jobs on the runner that runs pull requests' parity jobs (`horizon-game`), never the release runner
  (`MAC_RUNNER`), and its Rust cache has its own key and is never saved. A trusted build doesn't use the machine's
  shared sccache, which pull requests' jobs write (`HORIZON_NO_SCCACHE`, `tools/ci/runner-env.sh`).
- **Players.** Releases stay the default and the only channel any launcher follows unless its player turns Nightly
  on. `release_train.py`, `horizon-server update`, `install.sh` and the launcher's release list skip the tags: none
  of them is `vX.Y.Z`.

```sh
gh workflow run "nightly build"                  # now; skips when the pick is the commit already published
gh workflow run "nightly build" -f force=true    # build and publish anyway
gh workflow run "nightly build" -f dry_run=true  # build and sign with a throwaway key, publish nothing
```

## Nightly audit

`.github/workflows/nightly.yml` runs every harness from scratch (`tools/parity.py --all`) on main at 03:00 Paris
time, on the Mac mini's runner, after preparing the resources again if the preparer or its exporters changed.
`tools/ci/parity_issues.py` then compares the results with the previous audit
(`.local/parity-runner/audit.json` in the runner's checkout):

- a harness that passed last night and fails now gets an issue labelled `parity-regression`, titled
  `Parity regression: <harness>`, with the commit, the run and the end of its log (one issue per harness: an open
  one gets a comment instead);
- a harness with an open issue that passes again gets its issue closed.

The bugs session picks the issues up. The audit's logs are the run's `nightly-parity-logs` artifact; harnesses
listed `linux_only` are skipped there (see above). By hand: `gh workflow run "nightly audit"` (add
`-f dry_run=true` to leave issues alone).

## Disk

Releases ship often, so nothing they leave behind may grow without bound.

- **Artifacts:** every upload keeps 3 days (nightly parity logs 7); a release's assets live on the release itself.
- **Mac runners:** `tools/ci/runner-gc.sh` runs every 6 hours from launchd on the Mac mini, as a copy taken from main
  (`~/.actions-runners/runner-gc.sh`, plist `dev.horizon.runner-gc`), never from a pull request's checkout. It skips a
  runner in a job, removes a checkout's cargo build folders past 12 GB (`MAX_TARGET_GB`) and harness work folders
  under `.local/` untouched for 3 days (`STALE_DAYS`), and keeps the 2 newest shared preparations (`KEEP_PREPARED`)
  while no runner works. It also stops harness JVMs (vanilla servers, oracles) left orphaned for 3 hours
  (`ORPHAN_HOURS`): parent gone, command line under a runner's checkout or a worktree's `.local/`. `tools/parity.py`
  stops its own harnesses on SIGTERM and SIGHUP (a cancelled job), so these come only from a SIGKILL or a harness run
  by hand. Runner builds go through the machine's sccache (`tools/ci/runner-env.sh`), so what it removes
  comes back from the cache. Update the copy after changing the script:
  `ssh macmini 'curl -fsSL https://raw.githubusercontent.com/horizon-rust/horizon/main/tools/ci/runner-gc.sh -o ~/.actions-runners/runner-gc.sh'`.