The Armv8-R AArch64 application level programmers’ model differs from the Armv8-A AArch64 profile in the following ways:
Armv8-R AArch64 supports only a single Security state, Secure.
EL2 is mandatory.
EL3 is not supported.
Armv8-R AArch64 supports the A64 ISA instruction set with some modifications.
See the section B1.1.1 of Arm Architecture Reference Manual Supplement for more details.
In this Software Stack, boot-wrapper-aarch64 works as an alternative Trusted-Firmware solution, to solve the difference in the startup process due to the above differences.
The original general boot-wrapper is a fairly simple implementation of a boot loader intended to run under an ARM Fast Model and boot Linux. In this Software Stack, boot-wrapper-aarch64 implements the following functions for the Armv8-R AArch64 architecture:
S-EL2 booting for Armv8-R AArch64
Supports to boot Linux or U-Boot from S-EL1
Supports to boot Xen hypervisor or U-Boot from S-EL2
Provides PSCI services (
CPU_OFF) for booting SMP Linux on baremetal
Provides PSCI services (
CPU_OFF) to support Xen SMP boot by introducing libfdt to manipulate Flattened Device Trees dynamically and reserve
/memreserveto prevent from overriding PSCI services
Boot-wrapper-aarch64 is implemented in https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git/, with additional patches applied in the meta-arm-bsp layer.
Two new compilation options have been added in these patches:
--enable-psci can be used to choose PSCI method between Secure Monitor
Call (SMC) and Hypervisor Call (HVC). To use
--enable-psci=smc. To use
Armv8-R AArch64 does not support
hvc is selected in this
--enable-keep-el can be used to enable boot-wrapper-aarch64 to boot
next stage at S-EL2. As the Armv8-R AArch64 architecture boots from S-EL2, if
the next stage requires executing from S-EL2, the boot-wrapper-aarch64 shall not
drop to S-EL1.
Configuration of these two options in this stack can be found here.
The patches are based on top of U-Boot’s “vexpress64” board family (which
includes the Juno Versatile Express development board and the
FVP_Base_RevC-2xAEMvA model) because the
FVP_Base_AEMv8R model has a
similar memory layout. The board is known in U-Boot as
BASER_FVP and is
implemented through additions to the
vexpress_aemv8.h header file and the
vexpress_aemv8r_defconfig default configuration file.
As well as supporting the
BASER_FVP memory map, to allow running Xen at
S-EL2 it is required to run U-Boot at
S-EL2 as well, which has the
It is required to initialize the S-EL2 Memory Protection Unit (MPU) to support unaligned memory accesses.
Boot-wrapper-aarch64’s PSCI handler uses the exception vector at S-EL2 (
VBAR_EL2). By default U-Boot overwrites this for its panic handler.
We cannot disable hypercalls in
HVCinstructions are used for PSCI services.
A mechanism is required to decide at runtime whether to boot the next stage at S-EL2 (for Xen) or S-EL1 (for Linux).
Additional patches have therefore been added to:
Configure Memory Protection Units (MPU). Additionally, add logic to detect whether a Memory Management Unit (MMU) is present at the current exception level and if not, trigger the MPU initialization and deinitialization. It is only possible to determine which memory system architecture is active at S-EL1 from S-EL2 (system register
VTCR_EL2), so at S-EL1 assume the MMU is configured for backwards compatibility.
Disable setting the exception vectors (
VBAR_EL2). There is already logic to disable exception vectors for Secondary Program Loader (SPL) builds, so this has been extended to allow disabling the vectors for non-SPL builds too.
Make disabling hypercalls when switching to S-EL1 configurable.
By adding support for switching to S-EL1 for EFI (Extensible Firmware Interface) booting (it was previously only implemented for
The environment variable
armv8_switch_to_el1has been added to allow the boot exception level to be configured at runtime, which overrides the compile-time option.
Disable setting the
COUNTER_FREQUENCY. The value it’s set to might be different from the value used by the first-stage bootloader.
To support amending the device tree at runtime, there is also a patch to enable
CONFIG_LIBFDT_OVERLAY for the
U-Boot’s “distro” feature provides a standard
bootcmd which automatically
attempts a pre-configured list of boot methods. For the
Load a boot script from memory.
Load a boot script stored at
/boot.scron any supported partition on any attached block device.
Load a “removable media” EFI payload stored at
/EFI/boot/bootaa64.efion any supported partition on any attached block device.
A boot script is a file (compiled using U-Boot’s
mkimage command) which
contains commands that are executed by U-Boot’s interpreter.
For baremetal Linux, option 3 above is used. Grub2 (in EFI mode) is installed at the required path in the boot partition of the disk image, which is attached to the FVP virtio block interface. Grub2 is configured with a single menu item to boot the Linux kernel, the image for which is stored in the same partition.
For virtualization, option 2 above is used. A boot script is added to the boot partition of the disk image, which:
Loads a device tree overlay file from the same partition and applies it to the firmware’s device tree.
Loads the Xen module binaries and passthrough device trees to the required memory addresses.
Loads the Xen binary itself.
armv8_switch_to_el1environment variable to
n, so that Xen will boot at S-EL2.
Boots Xen using the
Option 1 above is not used.
This Software Stack uses Xen as the Type-1 hypervisor for hardware virtualization, which makes it possible to run multiple operating systems (Linux as the Rich OS and Zephyr as RTOS) in parallel on a single Armv8-R AEM FVP model (AArch64 mode). Xen in this Software Stack is implemented by version 4.17 and the additional patches in directory meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-extended/xen/files to support the Armv8-R AArch64 architecture.
In addition to the differences mentioned in the Boot-wrapper-aarch64 section, the Armv8-R AArch64 system level architecture differs from the Armv8-A AArch64 profiles in the following ways:
Armv8-R AArch64 provides a Protected Memory System Architecture (PMSA) based virtualization model.
The Armv8-R AArch64 implementation supports PMSA at S-EL1 and S-EL2, based on Memory Protection Unit (MPU).
Armv8-R AArch64 supports Virtual Memory System Architecture (VMSA), in which provides a Memory Management Unit (MMU), as an optional memory system architecture at S-EL1. In other words: optional S-EL1 MMU is supported in the Armv8-R AArch64 implementation.
These patches are mainly based on the above differences to provide support for Xen on the Armv8-R AArch64 architecture, enabling the virtualization use case described in Use Cases Overview in the following ways:
Enable MPU at S-EL2 to introduce virtualization at S-EL2
Support MPU at S-EL1 to host a Zephyr RTOS
Support MMU at S-EL1 to host a Linux Rich OS
And have the following functions to complete the entire virtualization use case:
The RTOS (Zephyr) domain with S-EL1 MPU and the Rich OS (Linux) domain with S-EL1 MMU can share the same physical cores, to make these two types of OSes can run in parallel (New feature for Xen supported by Armv8-R AArch64 only)
Xen with S-EL2 MPU isolates the RTOS workload and the Rich OS workload
Xen shares device tree nodes with boot-wrapper-aarch64 to implement SMP (Symmetric Multi-Processing)
Xen co-exists with boot-wrapper-aarch64 providing PSCI services at S-EL2
Xen runs in secure state only, thus OSes in Xen domains run in secure state too
The Armv8-R AEM FVP model supports 32 MPU regions by default, which is the
typical setting configured by the parameters of
cluster0.num_protection_regions_s2. Any other settings about the number
of MPU regions may cause unexpected malfunctionality or low performance.
In this implementation, Xen utilizes the “dom0less” feature to create 2 DomUs at boot time. Information about the DomUs to be created by Xen is passed to the hypervisor via device tree. The resources for DomUs, including memory, number of vCPUs, supported devices, etc., are all fully static allocated at compile time.
The resources for DomUs also include static shared memory and static event
channel. The static shared memory device tree nodes allow users to statically
set up shared memory on a dom0less system, enabling domains to do shm-based
communication. The static event channel communication can be established
statically between two domains (not only between two domUs, also between dom0
and domU). Static event channel connection information between domains will be
passed to Xen via the device tree node. The static event channel will be created
and established in Xen before the domain started. Domain only needs hypercall
EVTCHNOP_send to send notifications to the remote guest.
The documentation at this link provides more details about the device tree.
The kernel configuration for Armv8-R AArch64 in this stack is defined in the cfg and scc files in the meta-arm-bsp layer in the meta-arm repository. The device tree can also be found in the meta-arm-bsp layer.
Devices supported in the kernel:
In addition to the above configurations, Linux running in the Xen domain as a guest OS also enables the following additional configurations to support static shared memory, static event channel, docker, etc.:
File meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-extended/xen/files/fvp-baser-aemv8r64.cfg introduces Xen support for fvp-baser-aemv8r64
File xen.cfg contains kernel features needed to run as a Xen guest OS
File docker.cfg contains kernel features needed by docker
File meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-kernel/linux/files/fvp-baser-aemv8r64/kmeta-extra/features/xen-shmem.cfg contains kernel features to support Xen static shared memory
Kernel features that support Xen static event channels are enabled by default, so no additional configuration is required
The static shared memory and static event channel drivers are enabled in Linux by additional patches in directory meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-kernel/linux/files/fvp-baser-aemv8r64/kmeta-extra/features. These patches provide the following features:
xen,dom0lessto the Linux domain.
Add a new IOCTL type
IOCTL_EVTCHN_BINDto bind statically allocated static event channel port.
Add a new
xen,shared-infoproperty for information sharing between Xen and domains in
Add static shared memory driver.
The Zephyr OS is an open source real-time operating system based on a small-footprint kernel designed for user on resource-constrained and embedded systems.
The stack supports to run Zephyr (version 3.2.0) either on baremetal, or as a Xen domain in the virtualization solution.
Zephyr provides many demos and sample programs. This Software Stack supports the following 3 classic samples:
A demo application is also provided in this Software Stack to showcase the communication between the Zephyr domain and the Linux domain using OpenAMP RPMsg, based on the static shared memory and static event channel provided by the Xen hypervisor.
The static shared memory and static event channel drivers are enabled in Zephyr by additional patches in directory meta-armv8r64-extras/dynamic-layers/virtualization-layer/recipes-kernel/zephyr-kernel/files. These patches provide the following features:
Add hypervisor node in device tree to run Zephyr on Xen.
Add MPU static shared memory region.
Add Kconfig item
XEN_DOMAIN_SHARED_MEMto enable Xen driver and Xen domain static shared memory in Zephyr.
Save event comes before channel is bound, so that the event won’t be lost if the domain which receives events has not bound.