Components

Boot-wrapper-aarch64

The Armv8-R AArch64 application level programmers’ model differs from the Armv8-A AArch64 profile in the following ways:

  • Armv8-R AArch64 supports only a single Security state, Secure.

  • EL2 is mandatory.

  • EL3 is not supported.

  • Armv8-R AArch64 supports the A64 ISA instruction set with some modifications.

See the section B1.1.1 of Arm Architecture Reference Manual Supplement for more details.

In this software stack, boot-wrapper-aarch64 works as an alternative Trusted-Firmware solution, to solve the difference in the startup process due to the above differences.

The original general boot-wrapper is a fairly simple implementation of a boot loader intended to run under an ARM Fast Model and boot Linux. In this software stack, boot-wrapper-aarch64 implements the following functions for the Armv8-R AArch64 architecture:

  • S-EL2 booting for Armv8-R AArch64

  • Supports to boot Linux or U-Boot from S-EL1

  • Supports to boot Xen hypervisor or U-Boot from S-EL2

  • Provides PSCI services (CPU_ON / CPU_OFF) for booting SMP Linux on baremetal

  • Provides PSCI services (CPU_ON / CPU_OFF) to support Xen SMP boot by introducing libfdt to manipulate Flattened Device Trees dynamically and reserve /memreserve to prevent from overriding PSCI services

Boot-wrapper-aarch64 is implemented in https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git/, with additional patches applied in the meta-arm-bsp layer.

Two new compilation options have been added in these patches: --enable-psci and --enable-keep-el.

Flag --enable-psci can be used to choose PSCI method between Secure Monitor Call (SMC) and Hypervisor Call (HVC). To use smc, select --enable-psci or --enable-psci=smc. To use hvc, select --enable-psci=hvc.

Armv8-R AArch64 does not support smc, thus hvc is selected in this stack.

Flag --enable-keep-el can be used to enable boot-wrapper-aarch64 to boot next stage at S-EL2. As the Armv8-R AArch64 architecture boots from S-EL2, if the next stage requires executing from S-EL2, the boot-wrapper-aarch64 shall not drop to S-EL1.

Configuration of these two options in this stack can be found here.

Bootloader (U-Boot)

The stack’s bootloader is implemented by U-Boot (version 2022.01), with additional patches applied in the meta-arm-bsp layer.

Additional Patches

The patches are based on top of U-Boot’s “vexpress64” board family (which includes the Juno Versatile Express development board and the FVP_Base_RevC-2xAEMvA model) because the FVP_Base_AEMv8R model has a similar memory layout. The board is known in U-Boot as BASER_FVP and is implemented through additions to the vexpress_aemv8.h header file and the vexpress_aemv8r_defconfig default configuration file.

As well as supporting the BASER_FVP memory map, to allow running Xen at S-EL2 it is required to run U-Boot at S-EL2 as well, which has the following implications:

  • It is required to initialize the S-EL2 Memory Protection Unit (MPU) to support unaligned memory accesses.

  • Boot-wrapper-aarch64’s PSCI handler uses the exception vector at S-EL2 (VBAR_EL2). By default U-Boot overwrites this for its panic handler.

  • We cannot disable hypercalls in HCR_EL2, as HVC instructions are used for PSCI services.

  • A mechanism is required to decide at runtime whether to boot the next stage at S-EL2 (for Xen) or S-EL1 (for Linux).

Additional patches have therefore been added to:

  • Configure Memory Protection Units (MPU). Additionally, add logic to detect whether a Memory Management Unit (MMU) is present at the current exception level and if not, trigger the MPU initialization and deinitialization. It is only possible to determine which memory system architecture is active at S-EL1 from S-EL2 (system register VTCR_EL2), so at S-EL1 assume the MMU is configured for backwards compatibility.

  • Disable setting the exception vectors (VBAR_EL2). There is already logic to disable exception vectors for Secondary Program Loader (SPL) builds, so this has been extended to allow disabling the vectors for non-SPL builds too.

  • Make disabling hypercalls when switching to S-EL1 configurable.

  • Extend the ARMV8_SWITCH_TO_EL1 functionality:

    • By adding support for switching to S-EL1 for EFI (Extensible Firmware Interface) booting (it was previously only implemented for booti and bootm).

    • The environment variable armv8_switch_to_el1 has been added to allow the boot exception level to be configured at runtime, which overrides the compile-time option.

To support amending the device tree at runtime, there is also a patch to enable CONFIG_LIBFDT_OVERLAY for the BASER_FVP.

Boot Sequence

U-Boot’s “distro” feature provides a standard bootcmd which automatically attempts a pre-configured list of boot methods. For the BASER_FVP, these include:

  1. Load a boot script from memory.

  2. Load a boot script stored at /boot.scr on any supported partition on any attached block device.

  3. Load a “removable media” EFI payload stored at /EFI/boot/bootaa64.efi on any supported partition on any attached block device.

A boot script is a file (compiled using U-Boot’s mkimage command) which contains commands that are executed by U-Boot’s interpreter.

For baremetal Linux, option 3 above is used. Grub2 (in EFI mode) is installed at the required path in the boot partition of the disk image, which is attached to the FVP virtio block interface. Grub2 is configured with a single menu item to boot the Linux kernel, the image for which is stored in the same partition.

For virtualization, option 2 above is used (because Xen’s integration/mpu branch does not support EFI). A boot script is added to the boot partition of the disk image, which:

  • Loads a device tree overlay file from the same partition and applies it to the firmware’s device tree.

  • Loads the Xen module binaries and passthrough device trees to the required memory addresses.

  • Loads the Xen binary itself.

  • Sets the armv8_switch_to_el1 environment variable to n, so that Xen will boot at S-EL2.

  • Boots Xen using the booti boot method.

Option 1 above is not used.

Hypervisor (Xen)

This software stack uses Xen as the Type-1 hypervisor for hardware virtualization, which makes it possible to run multiple operating systems (Linux as the Rich OS and Zephyr as RTOS) in parallel on a single Armv8-R AEM FVP model (AArch64 mode). Xen in this software stack is implemented by version 4.17 at commit 3f5d614663 and the patches in integration/mpu branch to support the Armv8-R AArch64 architecture.

In addition to the differences mentioned in the Boot-wrapper-aarch64 section, the Armv8-R AArch64 system level architecture differs from the Armv8-A AArch64 profiles in the following ways:

  • Armv8-R AArch64 provides a Protected Memory System Architecture (PMSA) based virtualization model.

  • The Armv8-R AArch64 implementation supports PMSA at S-EL1 and S-EL2, based on Memory Protection Unit (MPU).

  • Armv8-R AArch64 supports Virtual Memory System Architecture (VMSA), in which provides a Memory Management Unit (MMU), as an optional memory system architecture at S-EL1. In other words: optional S-EL1 MMU is supported in the Armv8-R AArch64 implementation.

These patches are mainly based on the above differences to provide support for Xen on the Armv8-R AArch64 architecture, enabling the virtualization use case described in Use Cases Overview in the following ways:

  • Enable MPU at S-EL2 to introduce virtualization at S-EL2

  • Support MPU at S-EL1 to host a Zephyr RTOS

  • Support MMU at S-EL1 to host a Linux Rich OS

And have the following functions to complete the entire virtualization use case:

  • The RTOS (Zephyr) domain with S-EL1 MPU and the Rich OS (Linux) domain with S-EL1 MMU can share the same physical cores, to make these two types of OSes can run in parallel (New feature for Xen supported by Armv8-R AArch64 only)

  • Xen with S-EL2 MPU isolates the RTOS workload and the Rich OS workload

  • Xen shares device tree nodes with boot-wrapper-aarch64 to implement SMP (Symmetric Multi-Processing)

  • Xen co-exists with boot-wrapper-aarch64 providing PSCI services at S-EL2

  • Xen runs in secure state only, thus OSes in Xen domains run in secure state too

Note

The Armv8-R AEM FVP model supports 32 MPU regions by default, which is the typical setting configured by the parameters of cluster0.num_protection_regions_s1 and cluster0.num_protection_regions_s2. Any other settings about the number of MPU regions may cause unexpected malfunctionality or low performance.

In this implementation, Xen utilizes the “dom0less” feature to create 2 DomUs at boot time. Information about the DomUs to be created by Xen is passed to the hypervisor via device tree. The resources for DomUs, including memory, heap, number of CPU cores and supported devices, etc are all fully static allocated at compile time.

Linux Kernel

The Linux kernel in this stack is version 5.15 at https://git.yoctoproject.org/linux-yocto/.

The kernel configuration for Armv8-R AArch64 in this stack is defined in the cfg and scc files in the meta-arm-bsp layer in the meta-arm repository. The device tree can also be found in the meta-arm-bsp layer.

Devices supported in the kernel:

  • serial

  • virtio 9p

  • virtio disk

  • virtio network

  • virtio rng

  • watchdog

  • rtc

Zephyr

The Zephyr OS is an open source real-time operating system based on a small-footprint kernel designed for user on resource-constrained and embedded systems.

The stack supports to run Zephyr (version 3.1.0) either on baremetal, or as a Xen domain in the virtualization solution.

Zephyr is implemented in https://github.com/zephyrproject-rtos/zephyr, and supports the Armv8-R AArch64 architecture using Arm FVP BaseR AEMv8-R board listed in Zephyr supported boards.

Zephyr provides many demos and sample programs. This software stack supports the following 3 classic samples: