Sergio
476168bb07
250 líneas de comentario en 151 scripts. Control verificado: el diff no toca NI UNA línea que no empiece por #, y la sintaxis de los 151 pasa. El barrido saltea heredocs y cadenas triples, y el guardián DISPARÓ 3 veces: las tres eran el MOTD que el script escribe DENTRO de la imagen construida — texto del producto, no comentario del script. Se cambiaron aparte y a propósito, que es rebranding, no limpieza. Y el hallazgo caro: casaba contra , que es el TARGET de tracing — o sea el module_path!, o sea el nombre del crate. La etapa 4 lo movió a y el script quedó casando NADA. No fallaba: imprimía cero atribuciones, indistinguible de un log sin problemas. Comprobado con el binario (RUST_LOG=info sobre zlib), no deducido. Ahora acepta las dos, y tiene que seguir aceptándolas porque los logs viejos en disco dicen la vieja. Además 14 rutas de módulo en docs, que el barrido anterior no tocó porque no es frontera de palabra.
169 lines
10 KiB
Markdown
169 lines
10 KiB
Markdown
# rust-frontier — self-hosting the builder's rust toolchain
|
|
|
|
The last frontier of `selfhost-verify` variante (b): replace the Alpine
|
|
`rustc/cargo 1.91.1` in `/toolchain` with a hammer-built rust, via the **purist
|
|
mrustc chain** (no prebuilt rustc binary). Path:
|
|
|
|
mrustc → rustc 1.90.0 (builds bundled LLVM) → 1.91.0 → 1.91.1
|
|
|
|
These scripts + patch are the working artifacts. The mrustc tree itself lives in
|
|
`.scratch/mrustc` (gitignored — it's a large third-party clone, upstream master
|
|
`be69c74`); only our deltas are tracked here.
|
|
|
|
## Files
|
|
|
|
- **`run_rustc.patch`** — patch against mrustc `be69c74` for `run_rustc/Makefile`
|
|
and `run_rustc/rustc_proxy.sh`. Apply from the mrustc root:
|
|
`git -C .scratch/mrustc apply /path/to/run_rustc.patch`
|
|
- **`build-stage0-musl.sh`** — rebuild the mrustc stage0 `rustc`+`cargo` with a
|
|
**musl host triple** (`output-1.90.0-x86_64-unknown-linux-musl/`). Run inside
|
|
the Alpine devfs sandbox (gcc, MRUSTC_TARGET_VER=1.90). Reuses the already-built
|
|
LLVM and `bin/mrustc`.
|
|
- **`run-runrustc.sh`** — run a `run_rustc` make target inside the devfs sandbox,
|
|
pointed at the musl-host stage0.
|
|
- **`bootstrap.toml`** — rust's own bootstrap config for hop 1 (1.90.0 → 1.91.0):
|
|
`build`/`host`/`target = x86_64-unknown-linux-musl`, stage0 `rustc`/`cargo` =
|
|
the run_rustc musl-host 1.90.0 prefix, external LLVM 20.1.8 reused via
|
|
`llvm-config` (clears bootstrap's `>=19` gate, no LLVM rebuild), `crt-static =
|
|
false` (dynamic link against the devfs system musl, like Alpine and the
|
|
run_rustc toolchain), `rpath = true`. Copy it into the rust source root.
|
|
- **`bootstrap-1.91.1.toml`** — config for hop 2 (1.91.0 → 1.91.1). Same as hop 1
|
|
except: stage0 `rustc = /stage0/bin/rustc` (the installed hop-1 toolchain, bound
|
|
read-only via `STAGE0`), stage0 `cargo` = the run_rustc 1.90.0 cargo (1.90
|
|
satisfies bootstrap's "minor-1" check for a 1.91.x source), `extended = true` +
|
|
`tools = ["cargo"]` so the install yields a hammer-built cargo, and
|
|
**`cargo-native-static = true`** — the hermetic devfs has no system
|
|
OpenSSL/curl/libgit2, so this makes bootstrap build cargo with the `all-static`
|
|
feature set (vendored openssl/libgit2 + static curl/libz, compiled from the
|
|
vendored C sources via gcc). Without it the cargo tool fails on `openssl-sys`.
|
|
- **`swap-rust-into-toolchain.sh`** — overlay the hammer-built rust 1.91.1
|
|
(`.scratch/rust-1.91.1-prefix`) onto a toolchain dir (default `.dev-fs/alpine`)
|
|
so `hammer build` / `bootstrap stage1` compile the 4/4 rust inputs (arje-zero,
|
|
hammerd) with hammer-rust. Additive + reversible: hammer's `x86_64-unknown-linux-musl`
|
|
rustlib and hash-suffixed `.so`s don't collide with Alpine's
|
|
`x86_64-alpine-linux-musl`, so only `/usr/bin/{rustc,cargo}` are replaced (Alpine
|
|
backed up to `*.alpine`). `--restore` undoes it. Wired into `selfhost-verify.sh`
|
|
as `SWAP_RUST=1` (applies before the stage1 build, restores on exit via trap).
|
|
**This swap changes `of_tree`** (rustc emits the binaries), so the criterion is
|
|
auto-consistency against a NEW `RUST_EXPECT_REF`, not the Alpine `9adefb82`.
|
|
- **`run-xpy.sh`** — run rust's `x.py` inside the devfs sandbox, binding the
|
|
source at `/src`, the install prefix at `/out`, and the mrustc tree at `/mrustc`
|
|
(so the stage0 + `llvm-config` paths in `bootstrap.toml` resolve). Set the
|
|
`STAGE0` env var to also bind a prior hop's installed prefix read-only at
|
|
`/stage0`. Usage:
|
|
`run-xpy.sh <SRCDIR> <PREFIXDIR> -- build --stage 2 compiler/rustc library/std`.
|
|
|
|
> The runners hardcode `/home/sergio/hammer` and assume the prepared devfs at
|
|
> `.dev-fs/alpine` (256-TLS-key musl loader + g++; see the bootstrap-devfs notes).
|
|
|
|
## Why the patches exist (the snags)
|
|
|
|
The lab is **musl-only** (Alpine devfs); the 4/4 inputs (arje-zero/hammerd) are
|
|
built `--target x86_64-unknown-linux-musl`. Three coupled problems had to be fixed
|
|
to bootstrap a usable rust there:
|
|
|
|
1. **stage-1 std built for gnu** (`mmap64`/`open64` undefined at link).
|
|
`minicargo` was invoked without `--target`, so it used mrustc's default **gnu**
|
|
triple — and gnu std references glibc's LFS64 symbols that musl 1.2.5 doesn't
|
|
export. Fix: pass `--target $(RUSTC_TARGET)` to minicargo (Makefile:162) and to
|
|
the stage-1 smoke-test (Makefile:172). A correct musl std uses the **base** libc
|
|
symbols (`open`/`mmap`/…), matching Alpine's reference std.
|
|
|
|
2. **self-contained CRT absent.** musl's default link is `-static-pie` against a
|
|
`lib/rustlib/<t>/lib/self-contained/` CRT dir that the mrustc sysroot never
|
|
populates (also needs static `-lunwind`). Fix: `-C target-feature=-crt-static
|
|
-C link-self-contained=no` so the Alpine gcc supplies crt1/crti/crtn +
|
|
crtbeginS/crtendS and links dynamically against the system musl (RUSTFLAGS in
|
|
Makefile:111, plus **both** hello_world smoke-test rules: the stage-1 one at
|
|
Makefile:172 *and* the final fully-bootstrapped one at Makefile:262 — the latter
|
|
only surfaces once stage-2/3 actually completes, i.e. after snag #3 is fixed).
|
|
|
|
3. **host-triple = gnu on a musl system** (the deep one). cargo compiles
|
|
build-scripts and **proc-macros** for the *host*; mrustc's stage0 rustc has its
|
|
host triple baked as `x86_64-unknown-linux-gnu` (not runtime-overridable). A
|
|
proxy `--target musl` inject fixes build-script *bins*, but **proc-macros are
|
|
dlopen'd and must match the host triple structurally** — injection makes them
|
|
target artifacts that rustc rejects. Aliasing musl rlibs as gnu also fails:
|
|
rustc reads the triple from rlib *metadata* (`E0461`), not the path.
|
|
|
|
**The only fix is a stage0 rustc whose host triple is musl** — that's what
|
|
`build-stage0-musl.sh` produces (`CFG_COMPILER_HOST_TRIPLE` via `RUSTC_TARGET`,
|
|
`--target` via `MRUSTC_TARGET`). Then host==target==musl: cargo doesn't
|
|
cross-compile, proc-macros load natively, and `rustc_proxy.sh` only needs the
|
|
CRT flags (no `--target` inject).
|
|
|
|
Gotcha: do **not** pass `OUTDIR_SUF` on make's command line — a command-line var
|
|
can't be reassigned by the makefile, so the `-musl` suffix append is skipped and
|
|
the build contaminates the gnu output dir. Let it default.
|
|
|
|
4. **final cargo needs vendored OpenSSL.** The fully-bootstrapped `prefix/bin/cargo`
|
|
rule (Makefile:235) builds cargo with default features, so `openssl-sys` looks
|
|
for a system OpenSSL via pkg-config and fails (the hermetic musl devfs has none,
|
|
and no `openssl.pc`). Fix: add `--features vendored-openssl` to that cargo build
|
|
so `openssl-src` compiles OpenSSL from source — same approach the stage0 cargo
|
|
build uses. Real cargo (unlike minicargo) handles the OUT_DIR fine, so snag #4
|
|
doesn't recur here. Only surfaces after the final hello_world (snag #2/262) passes.
|
|
|
|
5. **minicargo OUT_DIR host/target split** (only when building the musl stage0's
|
|
cargo). With `RUSTC_TARGET` set, minicargo runs each cargo build-script under
|
|
`cargo-build/host/build_X/`, but the *target* crate's `include!` /
|
|
`include_bytes!(concat!(env!("OUT_DIR"), "/.."))` resolves `OUT_DIR` to
|
|
`cargo-build/build_X/` (no `host/`, empty) → mrustc aborts with signal 6. Hit
|
|
`libsqlite3-sys` (`bindgen.rs`) and the `cargo` binary (`man.tgz`). The gnu
|
|
build never split (single triple → single dir). Since host==target the script
|
|
outputs are bit-identical, so `build-stage0-musl.sh` symlinks the target dir to
|
|
the host dir (`symlink_outdirs`). Late sys-crates (cargo, curl, openssl,
|
|
libgit2-sys, libssh2-sys) run their scripts near the end, so it's applied in a
|
|
retry loop around the incremental cargo build.
|
|
|
|
## Workflow
|
|
|
|
```sh
|
|
# 1. apply the run_rustc patch into the mrustc clone
|
|
git -C .scratch/mrustc apply scripts/rust-frontier/run_rustc.patch
|
|
|
|
# 2. rebuild stage0 rustc+cargo with a musl host triple (~hours, reuses LLVM)
|
|
.scratch/build-stage0-musl.sh # -> output-1.90.0-x86_64-unknown-linux-musl/
|
|
|
|
# 3. build the optimised rustc + cargo via run_rustc (stage 1/2/3)
|
|
.scratch/run-runrustc.sh all
|
|
|
|
# 4. climb hop 1 — 1.90.0 -> 1.91.0 (stage0 = the run_rustc musl-host 1.90.0)
|
|
cp scripts/rust-frontier/bootstrap.toml .scratch/rustc-1.91.0-src/bootstrap.toml
|
|
.scratch/run-xpy.sh .scratch/rustc-1.91.0-src .scratch/rust-1.91.0-prefix -- \
|
|
build --stage 2 compiler/rustc library/std
|
|
.scratch/run-xpy.sh .scratch/rustc-1.91.0-src .scratch/rust-1.91.0-prefix -- \
|
|
install --stage 2 rustc library/std # -> relocatable 1.91.0 (rpath)
|
|
|
|
# 5. climb hop 2 — 1.91.0 -> 1.91.1 (stage0 rustc = the installed 1.91.0)
|
|
cp scripts/rust-frontier/bootstrap-1.91.1.toml .scratch/rustc-1.91.1-src/bootstrap.toml
|
|
STAGE0=.scratch/rust-1.91.0-prefix .scratch/run-xpy.sh \
|
|
.scratch/rustc-1.91.1-src .scratch/rust-1.91.1-prefix -- \
|
|
build --stage 2 compiler/rustc library/std src/tools/cargo
|
|
STAGE0=.scratch/rust-1.91.0-prefix .scratch/run-xpy.sh \
|
|
.scratch/rustc-1.91.1-src .scratch/rust-1.91.1-prefix -- \
|
|
install --stage 2 rustc library/std cargo # -> hammer rustc+cargo 1.91.1
|
|
|
|
# 6. swap rust 1.91.1 into /toolchain + auto-consistency verify (in-VM)
|
|
# PRESEED=hammerd es OBLIGATORIO con SWAP_RUST: arje-zero (monorepo tawasuyu) vendorea
|
|
# ~1973 crates; en un rebuild in-VM completo (PRESEED=all) ese vendor desborda el
|
|
# rootfs en RAM de la VM → ENOSPC. Con PRESEED=hammerd se preseedea el arje-zero
|
|
# host-built (takana-rust, locked) y sólo hammerd se reconstruye in-VM.
|
|
SWAP_RUST=1 KVM=1 MEM=16384 PRESEED=hammerd ./scripts/selfhost-verify.sh
|
|
```
|
|
|
|
**DONE (2026-06-16):** both hops built. `rust-1.91.1-prefix/bin/{rustc,cargo}` run
|
|
standalone (rpath), `rustc 1.91.1 (ed61e7d7e)` host `x86_64-unknown-linux-musl`
|
|
LLVM 20.1.8, `cargo 1.91.1`. Alpine's exact rust version, fully self-hosted from
|
|
the mrustc chain (mrustc → 1.90.0 → 1.91.0 → 1.91.1). Each hop ≈45 min compile
|
|
(LLVM reused throughout); hop 2's cargo needed the `cargo-native-static` fix.
|
|
|
|
**✓ AUTO-CONSISTENCY VERIFICADA IN-VM (2026-06-18):** `SWAP_RUST=1 … PRESEED=hammerd`
|
|
→ el host construye stage1 con hammer-rust (of_tree `7fa6cb4e`), la VM lo reconstruye
|
|
con hammer-rust y reproduce `7fa6cb4e` **bit a bit** (`stage1' == stage1`, DRIVER_RC=0).
|
|
`RUST_EXPECT_REF=7fa6cb4e` (≠ el `9adefb82` de Alpine — rustc emite los bytes). El drift
|
|
de arje-zero quedó cerrado fijando su `Cargo.lock` en tawasuyu (`recipes/arje-zero.toml`
|
|
→ commit `9967b02c`): build `--locked` ⇒ bit-reproducible (`lock1 == lock2`).
|
|
|
|
Full diagnostic history is in the `selfhost-verify-rust-frontier` memory.
|