Components

Boot-wrapper-aarch64

The Armv8-R AArch64 application level programmers’ model differs from the Armv8-A AArch64 profile in the following ways:

  • Armv8-R AArch64 supports only a single Security state, Secure.

  • EL2 is mandatory.

  • EL3 is not supported.

  • Armv8-R AArch64 supports the A64 ISA instruction set with some modifications.

See the section B1.1.1 of Arm Architecture Reference Manual Supplement for more details.

In this Software Stack, boot-wrapper-aarch64 works as an alternative Trusted-Firmware solution, to solve the difference in the startup process due to the above differences.

The original general boot-wrapper is a fairly simple implementation of a boot loader intended to run under an ARM Fast Model and boot Linux. In this Software Stack, boot-wrapper-aarch64 implements the following functions for the Armv8-R AArch64 architecture:

  • S-EL2 booting for Armv8-R AArch64

  • Supports to boot Linux or U-Boot from S-EL1

  • Supports to boot Xen hypervisor or U-Boot from S-EL2

  • Provides PSCI services (CPU_ON / CPU_OFF) for booting SMP Linux on baremetal

  • Provides PSCI services (CPU_ON / CPU_OFF) to support Xen SMP boot by introducing libfdt to manipulate Flattened Device Trees dynamically and reserve /memreserve to prevent from overriding PSCI services

Boot-wrapper-aarch64 is implemented in https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git/, with additional patches applied in the meta-arm-bsp layer.

Two new compilation options have been added in these patches: --enable-psci and --enable-keep-el.

Flag --enable-psci can be used to choose PSCI method between Secure Monitor Call (SMC) and Hypervisor Call (HVC). To use smc, select --enable-psci or --enable-psci=smc. To use hvc, select --enable-psci=hvc.

Armv8-R AArch64 does not support smc, thus hvc is selected in this stack.

Flag --enable-keep-el can be used to enable boot-wrapper-aarch64 to boot next stage at S-EL2. As the Armv8-R AArch64 architecture boots from S-EL2, if the next stage requires executing from S-EL2, the boot-wrapper-aarch64 shall not drop to S-EL1.

Configuration of these two options in this stack can be found here.

Bootloader (U-Boot)

The stack’s bootloader is implemented by U-Boot (version 2022.07), with additional patches applied in the meta-arm-bsp layer.

Additional Patches

The patches are based on top of U-Boot’s “vexpress64” board family (which includes the Juno Versatile Express development board and the FVP_Base_RevC-2xAEMvA model) because the FVP_Base_AEMv8R model has a similar memory layout. The board is known in U-Boot as BASER_FVP and is implemented through additions to the vexpress_aemv8.h header file and the vexpress_aemv8r_defconfig default configuration file.

As well as supporting the BASER_FVP memory map, to allow running Xen at S-EL2 it is required to run U-Boot at S-EL2 as well, which has the following implications:

  • It is required to initialize the S-EL2 Memory Protection Unit (MPU) to support unaligned memory accesses.

  • Boot-wrapper-aarch64’s PSCI handler uses the exception vector at S-EL2 (VBAR_EL2). By default U-Boot overwrites this for its panic handler.

  • We cannot disable hypercalls in HCR_EL2, as HVC instructions are used for PSCI services.

  • A mechanism is required to decide at runtime whether to boot the next stage at S-EL2 (for Xen) or S-EL1 (for Linux).

Additional patches have therefore been added to:

  • Configure Memory Protection Units (MPU). Additionally, add logic to detect whether a Memory Management Unit (MMU) is present at the current exception level and if not, trigger the MPU initialization and deinitialization. It is only possible to determine which memory system architecture is active at S-EL1 from S-EL2 (system register VTCR_EL2), so at S-EL1 assume the MMU is configured for backwards compatibility.

  • Disable setting the exception vectors (VBAR_EL2). There is already logic to disable exception vectors for Secondary Program Loader (SPL) builds, so this has been extended to allow disabling the vectors for non-SPL builds too.

  • Make disabling hypercalls when switching to S-EL1 configurable.

  • Extend the ARMV8_SWITCH_TO_EL1 functionality:

    • By adding support for switching to S-EL1 for EFI (Extensible Firmware Interface) booting (it was previously only implemented for booti and bootm).

    • The environment variable armv8_switch_to_el1 has been added to allow the boot exception level to be configured at runtime, which overrides the compile-time option.

  • Disable setting the COUNTER_FREQUENCY. The value it’s set to might be different from the value used by the first-stage bootloader.

To support amending the device tree at runtime, there is also a patch to enable CONFIG_LIBFDT_OVERLAY for the BASER_FVP.

Boot Sequence

U-Boot’s “distro” feature provides a standard bootcmd which automatically attempts a pre-configured list of boot methods. For the BASER_FVP, these include:

  1. Load a boot script from memory.

  2. Load a boot script stored at /boot.scr on any supported partition on any attached block device.

  3. Load a “removable media” EFI payload stored at /EFI/boot/bootaa64.efi on any supported partition on any attached block device.

A boot script is a file (compiled using U-Boot’s mkimage command) which contains commands that are executed by U-Boot’s interpreter.

For baremetal Linux, option 3 above is used. Grub2 (in EFI mode) is installed at the required path in the boot partition of the disk image, which is attached to the FVP virtio block interface. Grub2 is configured with a single menu item to boot the Linux kernel, the image for which is stored in the same partition.

For virtualization, option 2 above is used. A boot script is added to the boot partition of the disk image, which:

  • Loads a device tree overlay file from the same partition and applies it to the firmware’s device tree.

  • Loads the Xen module binaries and passthrough device trees to the required memory addresses.

  • Loads the Xen binary itself.

  • Sets the armv8_switch_to_el1 environment variable to n, so that Xen will boot at S-EL2.

  • Boots Xen using the booti boot method.

Option 1 above is not used.

Hypervisor (Xen)

This Software Stack uses Xen as the Type-1 hypervisor for hardware virtualization, which makes it possible to run multiple operating systems (Linux as the Rich OS and Zephyr as RTOS) in parallel on a single Armv8-R AEM FVP model (AArch64 mode). Xen in this Software Stack is implemented by version 4.17 and the additional patches in directory meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-extended/xen/files to support the Armv8-R AArch64 architecture.

In addition to the differences mentioned in the Boot-wrapper-aarch64 section, the Armv8-R AArch64 system level architecture differs from the Armv8-A AArch64 profiles in the following ways:

  • Armv8-R AArch64 provides a Protected Memory System Architecture (PMSA) based virtualization model.

  • The Armv8-R AArch64 implementation supports PMSA at S-EL1 and S-EL2, based on Memory Protection Unit (MPU).

  • Armv8-R AArch64 supports Virtual Memory System Architecture (VMSA), in which provides a Memory Management Unit (MMU), as an optional memory system architecture at S-EL1. In other words: optional S-EL1 MMU is supported in the Armv8-R AArch64 implementation.

These patches are mainly based on the above differences to provide support for Xen on the Armv8-R AArch64 architecture, enabling the virtualization use case described in Use Cases Overview in the following ways:

  • Enable MPU at S-EL2 to introduce virtualization at S-EL2

  • Support MPU at S-EL1 to host a Zephyr RTOS

  • Support MMU at S-EL1 to host a Linux Rich OS

And have the following functions to complete the entire virtualization use case:

  • The RTOS (Zephyr) domain with S-EL1 MPU and the Rich OS (Linux) domain with S-EL1 MMU can share the same physical cores, to make these two types of OSes can run in parallel (New feature for Xen supported by Armv8-R AArch64 only)

  • Xen with S-EL2 MPU isolates the RTOS workload and the Rich OS workload

  • Xen shares device tree nodes with boot-wrapper-aarch64 to implement SMP (Symmetric Multi-Processing)

  • Xen co-exists with boot-wrapper-aarch64 providing PSCI services at S-EL2

  • Xen runs in secure state only, thus OSes in Xen domains run in secure state too

Note

The Armv8-R AEM FVP model supports 32 MPU regions by default, which is the typical setting configured by the parameters of cluster0.num_protection_regions_s1 and cluster0.num_protection_regions_s2. Any other settings about the number of MPU regions may cause unexpected malfunctionality or low performance.

In this implementation, Xen utilizes the “dom0less” feature to create 2 DomUs at boot time. Information about the DomUs to be created by Xen is passed to the hypervisor via device tree. The resources for DomUs, including memory, number of vCPUs, supported devices, etc., are all fully static allocated at compile time.

The resources for DomUs also include static shared memory and static event channel. The static shared memory device tree nodes allow users to statically set up shared memory on a dom0less system, enabling domains to do shm-based communication. The static event channel communication can be established statically between two domains (not only between two domUs, also between dom0 and domU). Static event channel connection information between domains will be passed to Xen via the device tree node. The static event channel will be created and established in Xen before the domain started. Domain only needs hypercall EVTCHNOP_send to send notifications to the remote guest.

The documentation at this link provides more details about the device tree.

The static shared memory and static event channel are supported by Xen 4.17, as listed in the Xen Project 4.17 Feature List.

Linux Kernel

The Linux kernel in this stack is version 5.19 at https://git.yoctoproject.org/linux-yocto/.

The kernel configuration for Armv8-R AArch64 in this stack is defined in the cfg and scc files in the meta-arm-bsp layer in the meta-arm repository. The device tree can also be found in the meta-arm-bsp layer.

Devices supported in the kernel:

  • serial

  • virtio 9p

  • virtio disk

  • virtio network

  • virtio rng

  • watchdog

  • rtc

In addition to the above configurations, Linux running in the Xen domain as a guest OS also enables the following additional configurations to support static shared memory, static event channel, docker, etc.:

Additional Patches

The static shared memory and static event channel drivers are enabled in Linux by additional patches in directory meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-kernel/linux/files/fvp-baser-aemv8r64/kmeta-extra/features. These patches provide the following features:

  • Enable xen,dom0less to the Linux domain.

  • Add a new IOCTL type IOCTL_EVTCHN_BIND to bind statically allocated static event channel port.

  • Add a new xen,shared-info property for information sharing between Xen and domains in xen,dom0less system.

  • Add static shared memory driver.

Zephyr

The Zephyr OS is an open source real-time operating system based on a small-footprint kernel designed for user on resource-constrained and embedded systems.

The stack supports to run Zephyr (version 3.2.0) either on baremetal, or as a Xen domain in the virtualization solution.

Zephyr is implemented in https://github.com/zephyrproject-rtos/zephyr, and supports the Armv8-R AArch64 architecture using Arm FVP BaseR AEMv8-R board listed in Zephyr supported boards.

Zephyr provides many demos and sample programs. This Software Stack supports the following 3 classic samples:

A demo application is also provided in this Software Stack to showcase the communication between the Zephyr domain and the Linux domain using OpenAMP RPMsg, based on the static shared memory and static event channel provided by the Xen hypervisor.

Additional Patches

The static shared memory and static event channel drivers are enabled in Zephyr by additional patches in directory meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-kernel/zephyr-kernel/files. These patches provide the following features:

  • Add hypervisor node in device tree to run Zephyr on Xen.

  • Add MPU static shared memory region.

  • Add Kconfig item XEN and XEN_DOMAIN_SHARED_MEM to enable Xen driver and Xen domain static shared memory in Zephyr.

  • Save event comes before channel is bound, so that the event won’t be lost if the domain which receives events has not bound.