RustNotMC/docs/benchmark-report.md

Benchmark report format (version 1)

Every benchmark Horizon writes uses this one JSON format: horizon-client --benchmark, the launcher's own measurements, and tools/perf_bench.py (which also writes it for vanilla and vanilla with mods). The same file is the local log a player can attach to an issue and, only with the player's consent, what the telemetry upload sends (minus the local-only fields below). The types, the statistics and the hardware detection are in crates/horizon-bench; nothing else builds a report by hand.

A report holds summaries and histogram counts, never raw frame streams: a 10-minute run is a few kilobytes whatever the frame rate. Measured numbers are rounded to 3 decimals.

Example

{
  "format": "horizon-benchmark-report",
  "version": 1,
  "kind": "game",
  "created": "2026-10-15T14:03:27Z",
  "app": { "name": "horizon-client", "version": "0.21.0", "commit": "e96e1571" },
  "hardware": {
    "os": "macos", "os_version": "27.0", "arch": "aarch64",
    "cpu": "Apple M4 Pro", "cpu_threads": 14, "cpu_cores": 14, "ram_mb": 65536, "storage": "nvme", "form": "desktop",
    "gpu": { "name": "Apple M4 Pro", "vendor": "apple", "driver": "", "backend": "metal", "kind": "integrated" },
    "display": { "width": 2560, "height": 1440, "scale": 2.0, "refresh_hz": 60 }
  },
  "conditions": {
    "scenario": "flight", "seed": "1234", "render_distance": 16, "simulation_distance": 12,
    "window": [1280, 720], "vsync": false, "repeats": 3, "warmup_s": 2.0, "power": "ac", "load_avg_1m": 1.2
  },
  "metrics": {
    "startup.title_ms": { "n": 3, "mean": 812.0, "stdev": 21.5, "min": 790.0, "max": 833.0, "ci95": 53.4 },
    "memory.rss_peak_mb": { "n": 3, "mean": 1210.0, "stdev": 4.0, "min": 1206.0, "max": 1214.0, "ci95": 9.9 }
  },
  "frames": [
    {
      "phase": "village", "repeat": 0, "frames": 4920, "seconds": 12.0,
      "avg_fps": 410.0, "low_1pct_fps": 120.3, "low_01pct_fps": 61.0,
      "p50_ms": 2.31, "p95_ms": 3.8, "p99_ms": 6.9, "p999_ms": 14.2, "max_ms": 21.0,
      "cpu_p50_ms": 1.4, "gpu_p50_ms": 1.9,
      "histogram": [0, 0, 3011, 1520, 301, 62, 15, 7, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    }
  ]
}

Fields

format, version. Always "horizon-benchmark-report" and 1. A reader rejects other formats and newer versions. Adding an optional field keeps the version; renaming, removing or changing the meaning of one (histogram edges included) makes it 2.

kind. "game" (a game run: Horizon, or vanilla from tools/perf_bench.py) or "launcher".

created. When the run ended, UTC, YYYY-MM-DDTHH:MM:SSZ.

app. What was measured: name (horizon-client, horizon-launcher, minecraft), version, optionally commit (Horizon builds) and mods ([{ "name", "version" }], vanilla with mods).

hardware. os (macos, linux, windows), os_version, arch, cpu (brand string), cpu_threads, cpu_cores (physical), cpu_base_mhz and cpu_boost_mhz (where the OS gives them; not on Apple silicon), ram_mb, storage (the drive under the game folder: nvme, ssd, hdd), form (laptop, desktop); gpu (name, vendor, driver, backend: metal, vulkan, dx12, gl; kind: integrated, discrete, virtual, cpu, other; vram_mb, a discrete GPU's own memory) and display (the screen the window was on: width, height in physical pixels, scale, refresh_hz) when known. Any field can be missing. On mains or battery is conditions.power.

conditions. What has to match for two reports to be compared, all optional: scenario, seed, render_distance, simulation_distance, window ([width, height] actually rendered), vsync, fps_cap (absent: unlimited), offscreen (rendered without presenting: not a player number), repeats, warmup_s, power (ac, battery), load_avg_1m (before the run; above about 1 the machine was busy), backend (when forced). Readers compare reports only with equal scenario, distances, window, vsync, fps_cap and offscreen.

metrics. Scalar measurements, by name; each is a summary over the run's repeats: n, mean, stdev (sample standard deviation, null when n is 1), min, max, ci95 (half-width of the 95% confidence interval of the mean, Student's t, null when n is 1). The unit is the name's suffix: _ms, _s, _mb, _bytes, _fps, _pct, _mspt, _w, _c (degrees Celsius), _count. Names in use:

Name	What
startup.title_ms	process start to the first frame of the title screen
startup.world_ms	process start to in-world, every chunk around the spawn loaded and drawn
chunks.<phase>.load_ms	teleport to every chunk in view distance loaded and drawn
chunks.<phase>.generated_count, chunks.<phase>.meshed_count, chunks.<phase>.generated_per_s	chunks generated and section meshes built while loading, and the rate
fps.<phase>.avg_fps, fps.<phase>.low_1pct_fps, fps.<phase>.low_01pct_fps	per-phase figures across repeats
tick.mspt, tick.tps	integrated server tick time and rate (planned)
memory.rss_peak_mb, memory.rss_steady_mb	resident memory: peak over the run, median while flying
memory.gpu_mb, memory.<phase>.gpu_mb	video memory the game allocated (or NVIDIA attributes to it), where readable
power.package_w, power.battery_drain_pct	power draw, battery used per hour (laptops; planned)
thermal.cpu_c, thermal.throttled_pct	temperature, share of the run spent throttled (planned)
launcher.start_ms	launcher process start to its first frame (a session)
launcher.cold_start_ms, launcher.warm_start_ms	the same in horizon-launcher benchmark: its first start, then the others
launcher.rss_mb, launcher.cpu_idle_pct	launcher resident memory (sampled; max is the peak), CPU used while idle
launcher.release_list_ms	fetching the release list
launcher.install_ms, launcher.update_ms	downloading and unpacking a version: a new one, an update
launcher.prepare_ms	preparing a version's resources
launcher.play_to_game_frame_ms	Play pressed to the game's first frame
launcher.binary_bytes, game.binary_bytes	size of the executables

New names can be added without a new version; the table lists the ones the tools write.

frames. One entry per measured phase and repeat (a scene's flight, the launcher's page switching): phase, repeat, frames, seconds, avg_fps, low_1pct_fps and low_01pct_fps (the average frame time of the slowest 1% or 0.1% of frames, as FPS), p50_ms, p95_ms, p99_ms, p999_ms, max_ms, cpu_p50_ms and gpu_p50_ms when measured, and histogram: frame counts per bucket. The bucket edges are fixed by the version, so histograms from any number of reports add up:

0, 1, 2, 3, 4.17, 5, 6, 7, 8.33, 10, 12, 14, 16.67, 20, 25, 33.33, 40, 50, 66.67, 100, 150, 250, 500, 1000 ms

Bucket i counts frames from edge i (included) to edge i + 1; the last bucket is 1000 ms and over (24 buckets). 4.17, 8.33, 16.67 and 33.33 ms are 240, 120, 60 and 30 FPS.

Local only

label (a name the person running it chose) and local (paths of the logs the run wrote next to the report) stay on the machine: the upload removes them and dates the report to the day only, as every telemetry item is (Report::for_upload). The upload is horizon-telemetry's benchmark item: this report, as it is, only with the player's consent to benchmark uploads (docs/telemetry.md). Nothing in a report identifies the player, the machine's name or the world: no host name, user name, path or address.

Where reports go

• The game: benchmarks/<kind>-<created>-<scenario>.json in the game folder.
• The launcher: benchmarks/ in the launcher's data folder.
• tools/perf_bench.py: one report per game and render distance, benchmarks/perf/<label>-rd<N>.json in the repository checkout.