Validation

Run-Time Integration Tests

The run-time integration tests are a mechanism for validating the Software Stack’s core functionalities.

The tests are run using the OEQA test framework. Please refer to OEQA on Arm FVPs for more information on the this framework. Some tests are also built as a Yocto Ptest (Package Test) and implemented using the Bash Automated Testing System (BATS).

The integration tests run on an image and depend on its target sub-stack (Baremetal Zephyr, Baremetal Linux or Virtualization). The tests may also change their behaviour depending on the the build parameters for target image.

To run the tests via kas, please refer to the section Validate in User Guide.

Test Suite

The 3 supported sub-stacks of the Software Stack that are tested by the framework are detailed below. The testing scripts can be found in meta-armv8r64-extras/lib/oeqa/runtime/cases.

Detailed below is a brief description of what each test does:

zephyr_v8r64

Zephyr test case which is capable of validating the output of zephyr-helloworld, zephyr-philosophers and zephyr-synchronization.
linuxboot

This case is implemented in meta-arm/lib/oeqa/runtime/cases/linuxboot.py. It waits for a Linux login prompt on the default console.
sysinfo

Test case to check system information (e.g. kernel version, hostname) is the same between Yocto build system and Yocto image.
basictests

Run basic-tests implemented using BATS. See Basic Tests for more details.
xen

Test case which is capable of validating the boot of Xen.
ptest_docker

Test case to check docker engine installation, daemon and runtime.
rpmsg

Test case to validate the Inter-VM communication using RPMsg between the Zephyr domain and the Linux domain on Xen.

The following table is a map of the test sets contained in the 3 sub-stacks:

Test Case	Baremetal Zephyr	Baremetal Linux	Virtualization
zephyr_v8r64	x		x
linuxboot		x	x
sysinfo		x	x
basictests		x	x
linuxlogin			x
xen			x
ptest_docker			x
rpmsg			x

For the Virtualization stack, more specifically, whether the test case is applicable or not depends on the parameters to the build the target image, and the corresponding relationship is as follows:

Test Case	Two DomUs (default)	Two DomUs	Single DomU (Zephyr)	Single DomU (Linux)
	XEN_DOM0LESS_DOMAINS: ZEPHYR + LINUX	XEN_DOM0LESS_DOMAINS: ZEPHYR + LINUX	XEN_DOM0LESS_DOMAINS: ZEPHYR	XEN_DOM0LESS_DOMAINS: LINUX
	ZEPHYR_APPLICATION: zephyr-rpmsg-demo	ZEPHYR_APPLICATION: zephyr-helloworld zephyr-synchronization zephyr-philosophers	ZEPHYR_APPLICATION: zephyr-helloworld zephyr-synchronization zephyr-philosophers
zephyr_v8r64	x	x	x
linuxboot	x	x		x
sysinfo	x	x		x
basictests	x	x		x
linuxlogin	x	x		x
xen	x	x	x	x
ptest_docker	x	x		x
rpmsg	x

For the build parameters XEN_DOM0LESS_DOMAINS and XEN_DOM0LESS_ZEPHYR_APPLICATION, please refer to the section Customizing Build Environment Parameters in User Guide.

Basic Tests

The Basic Tests perform the basic checks on the devices and functionalities supported in the system. These tests are available for both the Baremetal Linux and Virtualization images and consist of a series of BATS tests that can be found in meta-armv8r64-extras/recipes-refstack-tests/basic-tests/files/testcases.

Detailed below is information on which test is applicable to which stack and a brief description of what each test does:

Baremetal Linux:
- rtc
  
  The BATS script implementing the test is test-rtc.bats. Checks that the rtc (real-time clock) device and its correct driver are available and accessible via the filesystem and verifies that the hwclock command runs successfully.
- watchdog
  
  The BATS script implementing the test is test-watchdog.bats. Checks that the watchdog device and its correct driver are available and accessible via the filesystem.
- networking
  
  The BATS script implementing the test is test-net.bats. Checks that the network device and its correct driver are available and accessible via the filesystem and that outbound connections work (invoking wget).
- smp
  
  The BATS script implementing the test is test-smp.bats. Checks for CPU availability and that basic functionality works, like enabling and stopping CPUs and preventing all of them from being disabled at the same time.
- virtiorng
  
  The BATS script implementing the test is test-virtiorng.bats. Check that the virtio-rng device is available through the filesystem and that it is able to generate random numbers when required.
Virtualization:
- Linux Domain
  - Same tests as Baremetal Linux
  - shmem
    
    The BATS script implementing the test is test-shmem.bats. Check that the xen_mem device is available.
  - evtchn
    
    The BATS script implementing the test is test-evtchn.bats. Check that the xen/evtchn device is available.

The Basic Tests are built and installed in the image according to the following BitBake recipe: meta-armv8r64-extras/recipes-refstack-tests/basic-tests/basic-tests.bb.

Integration Tests Implementation

This section gives a high-level description of how the integration testing logic is implemented.

To enable the integration tests, the testimage.bbclass is used. This class supports running automated tests against images. The class handles loading the tests and starting the image.

The Writing New Tests section of the Yocto Manual explains how to write new tests when using the testimage.bbclass. These are placed under meta-armv8r64-extras/lib/oeqa/runtime/cases and will be selected by the different machines/configurations by modifying the TEST_SUITES variable. For example, the file meta-armv8r64-extras/classes/refstack-virtualization-tests.bbclass adds different combinations of test cases to the TEST_SUITES variable according to the build parameters of the virtualizaiton stack.

For the Basic Tests, the basic-tests.bb that inherits the Ptest framework/class and installs the BATS test files located under meta-armv8r64-extras/recipes-refstack-tests/basic-tests. Each image recipe (Baremetal Linux or Virtualization) includes basic-tests.bb.