# Talos CM5 Builder Custom Talos Linux images for Raspberry Pi 5 / CM5 on Compute Blade hardware. Gitea Releases Docker Hub Docker Pulls Docker Image Size The official Talos Image Factory does not support CM5 — the mainline kernel lacks CM5 device trees and RP1 driver support. This builder uses the RPi downstream kernel (via [talos-rpi5/talos-builder](https://github.com/talos-rpi5/talos-builder) patches) to produce working CM5 images with our extensions and overclock config. ## Current versions | Component | Version | |-----------|---------| | Talos Linux | Talos version | | RPi Kernel | Kernel version | | iscsi-tools | iscsi-tools version | | util-linux-tools | util-linux-tools version | ## Image tags Release images are published to docker.io/svrnty/talos-rpi5 with the format: ``` v-k- ``` For example: `v1.12.4-k6.12.47-4` | Segment | Meaning | |---------|---------| | `v1.12.4` | Upstream Talos Linux version | | `k6.12.47` | RPi downstream kernel version | | `3` | Build revision (bumped for config/patch changes on the same upstream versions) | ## Usage ### Install from raw disk image Download `metal-arm64.raw.zst` from the [latest release](../../releases/latest) and flash to eMMC: ```bash zstd -d metal-arm64.raw.zst -o metal-arm64.raw # Flash to eMMC/SD via your preferred tool (dd, balenaEtcher, etc.) ``` ### Upgrade an existing node ```bash talosctl upgrade --image docker.io/svrnty/talos-rpi5:v1.12.4-k6.12.47-4 --nodes ``` In-place upgrades are fully supported. The image includes patches to force GRUB with `--no-nvram` on arm64 (working around the RPi5/CM5 `SetVariableRT` firmware limitation) and to handle the SBC EFI-only disk layout (no separate BOOT partition). ### What's included - RPi downstream kernel with CM5/RP1 support (4K page size, aligned with upstream Talos) - GRUB bootloader with `--no-nvram` for reliable `talosctl upgrade` on RPi5/CM5 - SBC EFI-only boot layout support (probe, install, revert all fall back to EFI partition when BOOT partition is absent) - Fallback to classic bind mounts on kernels without `open_tree` support (Linux <6.15) - Overclock: 2.6GHz (`arm_freq=2600`, `over_voltage_delta=50000`, `arm_boost=1`) - PCIe Gen 3 enabled for NVMe (~800 MB/s, via `dtparam=pciex1_gen=3` in `config.txt`) - Serial console on GPIO UART0 (`dtoverlay=uart0-pi5`, 115200 baud, even parity) - Extensions: `iscsi-tools`, `util-linux-tools` ## Known issues ### Serial console (read-only) The serial console outputs kernel logs, boot messages, and panic traces over GPIO UART0 (`ttyAMA0`) at 115200 baud with **even parity**. Talos Linux has no interactive shell — the console is **read-only** and cannot accept input. On Pi5/CM5, GPIO 14/15 are not mapped to UART by default (unlike Pi4). The `dtoverlay=uart0-pi5` device tree overlay is required and is applied via `configTxtAppend` (after `disable-bt` and `disable-wifi` — ordering matters). #### Wiring (Compute Blade) Connect a **3.3V** USB-to-UART adapter to the front UART header (3-pin: GND, RX, TX). Only two wires are needed for read-only monitoring: ``` USB-UART Adapter Compute Blade (Front UART) ┌──────────────┐ ┌──────────────┐ │ GND ├──────────┤ GND │ │ RX ├──────────┤ TX │ │ TX │ │ RX │ └──────────────┘ └──────────────┘ 3.3V logic 3.3V GPIO ``` - **Adapter RX** connects to **Blade TX** (adapter receives data from the blade) - **GND** to **GND** (common ground reference) - Adapter TX is not needed for read-only monitoring (Talos has no shell to send input to) > **Warning:** CM5 GPIO is **3.3V only**. A 5V logic adapter will crash or damage the board. Tested with: [USB to TTL Serial Cable (FT232RNL)](https://www.pishop.ca/product/usb-to-ttl-serial-cable-for-raspberry-pi-5-debugging-ft232rnl-chip/) set to 3.3V mode. #### Reading the console ```bash # macOS — use screen with even parity screen /dev/cu.usbserial-XXXXXXXX 115200,,cse # Linux — configure with stty, then read with cat stty -F /dev/ttyUSB0 115200 cs8 parenb -parodd cat /dev/ttyUSB0 ``` *Upstream: talos-builder#4* ### Install disk config ignored on SBCs Talos ignores the `machine.install.disk` config field on SBC platforms. You **must flash the disk image directly** to your target disk (eMMC, SD, NVMe). For NVMe boot, `dd` the metal image to the NVMe drive and configure the EEPROM boot order (`BOOT_ORDER=0xf416`, `PCIE_PROBE=1`). *Upstream: talos-builder#22* ## Patches | Patch | Target | Description | |-------|--------|-------------| | `0001` (pkgs) | Kernel | RPi downstream kernel 6.12.x with CM5/RP1 device tree and driver support | | `0001` (talos) | Modules | arm64 kernel module list for RPi downstream kernel | | `0002` (talos) | GRUB | `--no-nvram` for `grub-install` on arm64 (U-Boot lacks EFI `SetVariable`) | | `0003` (talos) | Bootloader | Force GRUB over sd-boot on arm64 (sd-boot crashes without EFI runtime) | | `0004` (talos) | Runtime | Fallback to classic bind mounts on kernels without `open_tree` (Linux <6.15) | | `0005` (talos) | GRUB | Handle missing BOOT partition for SBC EFI-only disk layouts | | `0001` (overlay) | Toolchain | Bump Go to 1.24.13 (CVE fix) | | `0002` (overlay) | Upgrade | Detect EFI mount path for SBC layouts (no BOOT partition) | ## Roadmap This project targets production-ready Talos clusters on RPi5/CM5 hardware. | Status | Milestone | Description | |--------|-----------|-------------| | Tested | **4K page size** | Aligned with upstream Talos kernel config. Reduces memory overhead and improves workload compatibility (Longhorn, jemalloc, F2FS, etc.). | | Tested | **Reliable in-place upgrades** | Force GRUB bootloader with `--no-nvram` on arm64, handle SBC EFI-only disk layout. Verified end-to-end with `talosctl upgrade`. | | Tested | **Kernel <6.15 compatibility** | Unconditional `open_tree` capability check — falls back to classic bind mounts on RPi downstream kernel 6.12.x. | | Tested | **Serial console** | GPIO UART0 (`ttyAMA0`) via `dtoverlay=uart0-pi5`. Read-only output at 115200 baud, even parity. Verified on Compute Blade with FT232RNL adapter. | | Tested | **NVMe boot support** | `dd` image to NVMe + set EEPROM `BOOT_ORDER=0xf416` and `PCIE_PROBE=1`. Verified on 1TB Kingston NVMe on Compute Blade. | ## NVMe boot The kernel has NVMe built-in (`CONFIG_BLK_DEV_NVME=y`), so booting from NVMe should work by flashing the disk image directly and configuring the RPi5/CM5 EEPROM. ### 1. Flash the image to NVMe Connect the NVMe drive via a USB adapter and flash: ```bash zstd -d metal-arm64.raw.zst | sudo dd of=/dev/ bs=4M status=progress sync ``` ### 2. Configure EEPROM boot order Use `rpiboot` to update the CM5 EEPROM. Clone the usbboot repo and edit the boot config: ```bash git clone --depth=1 https://github.com/raspberrypi/usbboot cd usbboot && make # Edit the EEPROM config for CM5 cp recovery/boot.conf recovery/boot.conf.bak ``` Add or update these values in `recovery/boot.conf`: ```ini BOOT_ORDER=0xf416 PCIE_PROBE=1 ``` Then flash via USB with the CM5 in USB boot mode (hold nRPIBOOT or disable eMMC boot on your carrier board): ```bash sudo ./rpiboot -d recovery ``` `BOOT_ORDER` is read right-to-left: try NVMe (`6`) first, then SD (`1`), then USB (`4`), then restart (`f`). `PCIE_PROBE=1` is required for non-HAT+ NVMe adapters (Compute Blade, most M.2 carrier boards). ### 3. Boot from NVMe Power on. The RPi firmware should find the boot partition on NVMe, load U-Boot, and boot Talos. ### Optional: enable PCIe Gen 3 PCIe Gen 3 doubles NVMe throughput (~400 MB/s → ~800 MB/s). Not officially certified by Raspberry Pi but stable on most NVMe drives. **New installs** — PCIe Gen 3 is enabled by default in images built from this repo (`config.txt.append` includes `dtparam=pciex1_gen=3`). **Existing nodes** — After a `talosctl upgrade`, the overlay rewrites `config.txt` with the baked-in settings (including PCIe Gen 3). If you need to enable it manually on an older image: 1. Power off the node and remove the NVMe drive 2. Connect via USB adapter and mount the first (EFI) partition 3. Add to `config.txt` under the `[pi5]` section: ```ini dtparam=pciex1_gen=3 ``` 4. Unmount, reinstall the drive, and power on To verify after boot: ```bash talosctl -n dmesg | grep -i pcie # Look for "Gen 3" in the PCIe link speed output ``` ## Reference architecture Recommended storage layout for production Talos clusters on CM5 Compute Blade hardware. Each CM5 has a 32GB eMMC and an M.2 NVMe slot. | Role | Boot disk | NVMe | EEPROM `BOOT_ORDER` | Rationale | |------|-----------|------|---------------------|-----------| | **Control plane** | NVMe | Talos OS | `0xf416` (NVMe first) | etcd lives on the STATE partition — NVMe gives ~800 MB/s sequential and ~50K random IOPS for fast consensus and snapshot I/O. | | **Postgres / storage** | eMMC | Data (`/var/mnt/data`) | `0xf214` (eMMC first) | Talos OS on eMMC keeps `talosctl upgrade` away from the data drive. The full 1TB NVMe is dedicated to database storage via `machine.disks`. | | **Compute workers** | eMMC | None | `0xf214` (eMMC first) | Stateless workloads — no local storage needed. eMMC is more than enough for Talos OS (~2.2GB used). | ### Storage configuration **Control planes** — no extra config needed. Talos installs to the NVMe automatically when it's the boot disk. **Postgres / storage nodes** — add `machine.disks` to mount the NVMe as a data volume: ```yaml machine: disks: - device: /dev/nvme0n1 partitions: - mountpoint: /var/mnt/data ``` Talos automatically partitions, formats (XFS), and mounts the NVMe on first boot. The data persists across `talosctl upgrade` since upgrades only touch the boot disk (eMMC). **Compute workers** — no storage config needed. ### EEPROM setup All CM5 modules require `PCIE_PROBE=1` for NVMe detection on Compute Blade (non-HAT+ carrier). Set via `rpiboot`: ```ini # recovery/boot.conf BOOT_ORDER=0xf416 # NVMe-first (control planes) # or BOOT_ORDER=0xf214 # eMMC-first (storage/compute workers) PCIE_PROBE=1 # always required ``` ### Overclock All node types share the same overclock config (baked into the image via `config.txt.append`): ```ini dtparam=pciex1_gen=3 dtoverlay=uart0-pi5 arm_freq=2600 over_voltage_delta=50000 arm_boost=1 ``` Verified stable at 44.6°C max under full CPU + memory + disk stress across 10 nodes with Compute Blade heatsinks. ## Building For local builds, CI/CD setup, runner configuration, and project structure, see [TECHNICAL.md](TECHNICAL.md). ## License This project is licensed under the [Mozilla Public License 2.0](LICENSE). It builds upon the following MPL 2.0 licensed upstream projects: - [siderolabs/talos](https://github.com/siderolabs/talos) — Talos Linux OS - [siderolabs/pkgs](https://github.com/siderolabs/pkgs) — Talos package definitions - [talos-rpi5/sbc-raspberrypi5](https://github.com/talos-rpi5/sbc-raspberrypi5) — Raspberry Pi 5 SBC overlay Our patches to these projects are in the `patches/` directory and are distributed under the same MPL 2.0 terms.