takana/scripts/rust-frontier/README.md

rust-frontier — self-hosting the builder's rust toolchain

The last frontier of selfhost-verify variante (b): replace the Alpine rustc/cargo 1.91.1 in /toolchain with a hammer-built rust, via the purist mrustc chain (no prebuilt rustc binary). Path:

mrustc → rustc 1.90.0 (builds bundled LLVM) → 1.91.0 → 1.91.1

These scripts + patch are the working artifacts. The mrustc tree itself lives in .scratch/mrustc (gitignored — it's a large third-party clone, upstream master be69c74); only our deltas are tracked here.

Files

• run_rustc.patch — patch against mrustc be69c74 for run_rustc/Makefile and run_rustc/rustc_proxy.sh. Apply from the mrustc root: git -C .scratch/mrustc apply /path/to/run_rustc.patch
• build-stage0-musl.sh — rebuild the mrustc stage0 rustc+cargo with a musl host triple (output-1.90.0-x86_64-unknown-linux-musl/). Run inside the Alpine devfs sandbox (gcc, MRUSTC_TARGET_VER=1.90). Reuses the already-built LLVM and bin/mrustc.
• run-runrustc.sh — run a run_rustc make target inside the devfs sandbox, pointed at the musl-host stage0.
• bootstrap.toml — rust's own bootstrap config for hop 1 (1.90.0 → 1.91.0): build/host/target = x86_64-unknown-linux-musl, stage0 rustc/cargo = the run_rustc musl-host 1.90.0 prefix, external LLVM 20.1.8 reused via llvm-config (clears bootstrap's >=19 gate, no LLVM rebuild), crt-static = false (dynamic link against the devfs system musl, like Alpine and the run_rustc toolchain), rpath = true. Copy it into the rust source root.
• bootstrap-1.91.1.toml — config for hop 2 (1.91.0 → 1.91.1). Same as hop 1 except: stage0 rustc = /stage0/bin/rustc (the installed hop-1 toolchain, bound read-only via STAGE0), stage0 cargo = the run_rustc 1.90.0 cargo (1.90 satisfies bootstrap's "minor-1" check for a 1.91.x source), extended = true + tools = ["cargo"] so the install yields a hammer-built cargo, and cargo-native-static = true — the hermetic devfs has no system OpenSSL/curl/libgit2, so this makes bootstrap build cargo with the all-static feature set (vendored openssl/libgit2 + static curl/libz, compiled from the vendored C sources via gcc). Without it the cargo tool fails on openssl-sys.
• swap-rust-into-toolchain.sh — overlay the hammer-built rust 1.91.1 (.scratch/rust-1.91.1-prefix) onto a toolchain dir (default .dev-fs/alpine) so hammer build / bootstrap stage1 compile the 4/4 rust inputs (arje-zero, hammerd) with hammer-rust. Additive + reversible: hammer's x86_64-unknown-linux-musl rustlib and hash-suffixed .sos don't collide with Alpine's x86_64-alpine-linux-musl, so only /usr/bin/{rustc,cargo} are replaced (Alpine backed up to *.alpine). --restore undoes it. Wired into selfhost-verify.sh as SWAP_RUST=1 (applies before the stage1 build, restores on exit via trap). This swap changes of_tree (rustc emits the binaries), so the criterion is auto-consistency against a NEW RUST_EXPECT_REF, not the Alpine 9adefb82.
• run-xpy.sh — run rust's x.py inside the devfs sandbox, binding the source at /src, the install prefix at /out, and the mrustc tree at /mrustc (so the stage0 + llvm-config paths in bootstrap.toml resolve). Set the STAGE0 env var to also bind a prior hop's installed prefix read-only at /stage0. Usage: run-xpy.sh <SRCDIR> <PREFIXDIR> -- build --stage 2 compiler/rustc library/std.

The runners hardcode /home/sergio/hammer and assume the prepared devfs at .dev-fs/alpine (256-TLS-key musl loader + g++; see the bootstrap-devfs notes).

Why the patches exist (the snags)

The lab is musl-only (Alpine devfs); the 4/4 inputs (arje-zero/hammerd) are built --target x86_64-unknown-linux-musl. Three coupled problems had to be fixed to bootstrap a usable rust there:

1. stage-1 std built for gnu (mmap64/open64 undefined at link). minicargo was invoked without --target, so it used mrustc's default gnu triple — and gnu std references glibc's LFS64 symbols that musl 1.2.5 doesn't export. Fix: pass --target $(RUSTC_TARGET) to minicargo (Makefile:162) and to the stage-1 smoke-test (Makefile:172). A correct musl std uses the base libc symbols (open/mmap/…), matching Alpine's reference std.

2. self-contained CRT absent. musl's default link is -static-pie against a lib/rustlib/<t>/lib/self-contained/ CRT dir that the mrustc sysroot never populates (also needs static -lunwind). Fix: -C target-feature=-crt-static -C link-self-contained=no so the Alpine gcc supplies crt1/crti/crtn + crtbeginS/crtendS and links dynamically against the system musl (RUSTFLAGS in Makefile:111, plus both hello_world smoke-test rules: the stage-1 one at Makefile:172 and the final fully-bootstrapped one at Makefile:262 — the latter only surfaces once stage-2/3 actually completes, i.e. after snag #3 is fixed).

3. host-triple = gnu on a musl system (the deep one). cargo compiles build-scripts and proc-macros for the host; mrustc's stage0 rustc has its host triple baked as x86_64-unknown-linux-gnu (not runtime-overridable). A proxy --target musl inject fixes build-script bins, but proc-macros are dlopen'd and must match the host triple structurally — injection makes them target artifacts that rustc rejects. Aliasing musl rlibs as gnu also fails: rustc reads the triple from rlib metadata (E0461), not the path.

   The only fix is a stage0 rustc whose host triple is musl — that's what build-stage0-musl.sh produces (CFG_COMPILER_HOST_TRIPLE via RUSTC_TARGET, --target via MRUSTC_TARGET). Then host==target==musl: cargo doesn't cross-compile, proc-macros load natively, and rustc_proxy.sh only needs the CRT flags (no --target inject).

   Gotcha: do not pass OUTDIR_SUF on make's command line — a command-line var can't be reassigned by the makefile, so the -musl suffix append is skipped and the build contaminates the gnu output dir. Let it default.

4. final cargo needs vendored OpenSSL. The fully-bootstrapped prefix/bin/cargo rule (Makefile:235) builds cargo with default features, so openssl-sys looks for a system OpenSSL via pkg-config and fails (the hermetic musl devfs has none, and no openssl.pc). Fix: add --features vendored-openssl to that cargo build so openssl-src compiles OpenSSL from source — same approach the stage0 cargo build uses. Real cargo (unlike minicargo) handles the OUT_DIR fine, so snag #4 doesn't recur here. Only surfaces after the final hello_world (snag #2/262) passes.

5. minicargo OUT_DIR host/target split (only when building the musl stage0's cargo). With RUSTC_TARGET set, minicargo runs each cargo build-script under cargo-build/host/build_X/, but the target crate's include! / include_bytes!(concat!(env!("OUT_DIR"), "/..")) resolves OUT_DIR to cargo-build/build_X/ (no host/, empty) → mrustc aborts with signal 6. Hit libsqlite3-sys (bindgen.rs) and the cargo binary (man.tgz). The gnu build never split (single triple → single dir). Since host==target the script outputs are bit-identical, so build-stage0-musl.sh symlinks the target dir to the host dir (symlink_outdirs). Late sys-crates (cargo, curl, openssl, libgit2-sys, libssh2-sys) run their scripts near the end, so it's applied in a retry loop around the incremental cargo build.

Workflow

# 1. apply the run_rustc patch into the mrustc clone
git -C .scratch/mrustc apply scripts/rust-frontier/run_rustc.patch

# 2. rebuild stage0 rustc+cargo with a musl host triple (~hours, reuses LLVM)
.scratch/build-stage0-musl.sh            # -> output-1.90.0-x86_64-unknown-linux-musl/

# 3. build the optimised rustc + cargo via run_rustc (stage 1/2/3)
.scratch/run-runrustc.sh all

# 4. climb hop 1 — 1.90.0 -> 1.91.0 (stage0 = the run_rustc musl-host 1.90.0)
cp scripts/rust-frontier/bootstrap.toml .scratch/rustc-1.91.0-src/bootstrap.toml
.scratch/run-xpy.sh .scratch/rustc-1.91.0-src .scratch/rust-1.91.0-prefix -- \
    build --stage 2 compiler/rustc library/std
.scratch/run-xpy.sh .scratch/rustc-1.91.0-src .scratch/rust-1.91.0-prefix -- \
    install --stage 2 rustc library/std        # -> relocatable 1.91.0 (rpath)

# 5. climb hop 2 — 1.91.0 -> 1.91.1 (stage0 rustc = the installed 1.91.0)
cp scripts/rust-frontier/bootstrap-1.91.1.toml .scratch/rustc-1.91.1-src/bootstrap.toml
STAGE0=.scratch/rust-1.91.0-prefix .scratch/run-xpy.sh \
    .scratch/rustc-1.91.1-src .scratch/rust-1.91.1-prefix -- \
    build --stage 2 compiler/rustc library/std src/tools/cargo
STAGE0=.scratch/rust-1.91.0-prefix .scratch/run-xpy.sh \
    .scratch/rustc-1.91.1-src .scratch/rust-1.91.1-prefix -- \
    install --stage 2 rustc library/std cargo  # -> hammer rustc+cargo 1.91.1

# 6. swap rust 1.91.1 into /toolchain + auto-consistency verify (in-VM)
#    PRESEED=hammerd es OBLIGATORIO con SWAP_RUST: arje-zero (monorepo tawasuyu) vendorea
#    ~1973 crates; en un rebuild in-VM completo (PRESEED=all) ese vendor desborda el
#    rootfs en RAM de la VM → ENOSPC. Con PRESEED=hammerd se preseedea el arje-zero
#    host-built (takana-rust, locked) y sólo hammerd se reconstruye in-VM.
SWAP_RUST=1 KVM=1 MEM=16384 PRESEED=hammerd ./scripts/selfhost-verify.sh

DONE (2026-06-16): both hops built. rust-1.91.1-prefix/bin/{rustc,cargo} run standalone (rpath), rustc 1.91.1 (ed61e7d7e) host x86_64-unknown-linux-musl LLVM 20.1.8, cargo 1.91.1. Alpine's exact rust version, fully self-hosted from the mrustc chain (mrustc → 1.90.0 → 1.91.0 → 1.91.1). Each hop ≈45 min compile (LLVM reused throughout); hop 2's cargo needed the cargo-native-static fix.

✓ AUTO-CONSISTENCY VERIFICADA IN-VM (2026-06-18): SWAP_RUST=1 … PRESEED=hammerd → el host construye stage1 con hammer-rust (of_tree 7fa6cb4e), la VM lo reconstruye con hammer-rust y reproduce 7fa6cb4e bit a bit (stage1' == stage1, DRIVER_RC=0). RUST_EXPECT_REF=7fa6cb4e (≠ el 9adefb82 de Alpine — rustc emite los bytes). El drift de arje-zero quedó cerrado fijando su Cargo.lock en tawasuyu (recipes/arje-zero.toml → commit 9967b02c): build --locked ⇒ bit-reproducible (lock1 == lock2).

Full diagnostic history is in the selfhost-verify-rust-frontier memory.