.. # Copyright (c) 2022, Arm Limited. # # SPDX-License-Identifier: MIT .. _Validation: ########## Validation ########## ************************** Run-Time Integration Tests ************************** The run-time integration tests are a mechanism for validating the Software Stack's core functionalities. The tests are run using the `OEQA`_ test framework. Please refer to `OEQA on Arm FVPs`_ for more information on the this framework. Some tests are also built as a Yocto `Ptest`_ (Package Test) and implemented using the `Bash Automated Testing System`_ (BATS). The integration tests run on an image and depend on its target sub-stack (**Baremetal Zephyr**, **Baremetal Linux** or **Virtualization**). The tests may also change their behaviour depending on the the build parameters for target image. To run the tests via kas, please refer to the section :ref:`Validate` in :ref:`User Guide`. .. _OEQA: https://wiki.yoctoproject.org/wiki/QA .. _OEQA on Arm FVPs: https://git.yoctoproject.org/meta-arm/tree/documentation/oeqa-fvp.md?h=langdale .. _Ptest: https://wiki.yoctoproject.org/wiki/Ptest .. _Bash Automated Testing System: https://github.com/bats-core/bats-core ********** Test Suite ********** The 3 supported sub-stacks of the Software Stack that are tested by the framework are detailed below. The testing scripts can be found in :repo:`meta-armv8r64-extras/lib/oeqa/runtime/cases`. Detailed below is a brief description of what each test does: * zephyr_v8r64 Zephyr test case which is capable of validating the output of ``zephyr-helloworld``, ``zephyr-philosophers`` and ``zephyr-synchronization``. * linuxboot This case is implemented in :file:`meta-arm/lib/oeqa/runtime/cases/linuxboot.py`. It waits for a Linux login prompt on the default console. * sysinfo Test case to check system information (e.g. kernel version, hostname) is the same between Yocto build system and Yocto image. * basictests Run ``basic-tests`` implemented using BATS. See :ref:`Basic Tests` for more details. * xen Test case which is capable of validating the boot of Xen. * ptest_docker Test case to check docker engine installation, daemon and runtime. * rpmsg Test case to validate the Inter-VM communication using RPMsg between the Zephyr domain and the Linux domain on Xen. The following table is a map of the test sets contained in the 3 sub-stacks: +--------------+------------------+-----------------+----------------+ | Test Case | Baremetal Zephyr | Baremetal Linux | Virtualization | +==============+==================+=================+================+ | zephyr_v8r64 | x | | x | +--------------+------------------+-----------------+----------------+ | linuxboot | | x | x | +--------------+------------------+-----------------+----------------+ | sysinfo | | x | x | +--------------+------------------+-----------------+----------------+ | basictests | | x | x | +--------------+------------------+-----------------+----------------+ | linuxlogin | | | x | +--------------+------------------+-----------------+----------------+ | xen | | | x | +--------------+------------------+-----------------+----------------+ | ptest_docker | | | x | +--------------+------------------+-----------------+----------------+ | rpmsg | | | x | +--------------+------------------+-----------------+----------------+ For the Virtualization stack, more specifically, whether the test case is applicable or not depends on the parameters to the build the target image, and the corresponding relationship is as follows: +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | Test Case | Two DomUs (default) | Two DomUs | Single DomU (Zephyr) | Single DomU (Linux) | | +-----------------------+--------------------------+--------------------------+-----------------------+ | | XEN_DOM0LESS_DOMAINS: | XEN_DOM0LESS_DOMAINS: | XEN_DOM0LESS_DOMAINS: | XEN_DOM0LESS_DOMAINS: | | | | | | | | | * ZEPHYR + LINUX | * ZEPHYR + LINUX | * ZEPHYR | * LINUX | | +-----------------------+--------------------------+--------------------------+-----------------------+ | | ZEPHYR_APPLICATION: | ZEPHYR_APPLICATION: | ZEPHYR_APPLICATION: | | | | | | | | | | * zephyr-rpmsg-demo | * zephyr-helloworld | * zephyr-helloworld | | | | | * zephyr-synchronization | * zephyr-synchronization | | | | | * zephyr-philosophers | * zephyr-philosophers | | +==============+=======================+==========================+==========================+=======================+ | zephyr_v8r64 | x | x | x | | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | linuxboot | x | x | | x | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | sysinfo | x | x | | x | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | basictests | x | x | | x | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | linuxlogin | x | x | | x | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | xen | x | x | x | x | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | ptest_docker | x | x | | x | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ | rpmsg | x | | | | +--------------+-----------------------+--------------------------+--------------------------+-----------------------+ For the build parameters ``XEN_DOM0LESS_DOMAINS`` and ``XEN_DOM0LESS_ZEPHYR_APPLICATION``, please refer to the section :ref:`Customizing Build Environment Parameters` in :ref:`User Guide`. .. _Basic Tests: *********** Basic Tests *********** The Basic Tests perform the basic checks on the devices and functionalities supported in the system. These tests are available for both the **Baremetal Linux** and **Virtualization** images and consist of a series of BATS tests that can be found in :repo:`meta-armv8r64-extras/recipes-refstack-tests/basic-tests/files/testcases`. Detailed below is information on which test is applicable to which stack and a brief description of what each test does: * Baremetal Linux: * rtc The BATS script implementing the test is ``test-rtc.bats``. Checks that the rtc (real-time clock) device and its correct driver are available and accessible via the filesystem and verifies that the ``hwclock`` command runs successfully. * watchdog The BATS script implementing the test is ``test-watchdog.bats``. Checks that the watchdog device and its correct driver are available and accessible via the filesystem. * networking The BATS script implementing the test is ``test-net.bats``. Checks that the network device and its correct driver are available and accessible via the filesystem and that outbound connections work (invoking ``wget``). * smp The BATS script implementing the test is ``test-smp.bats``. Checks for CPU availability and that basic functionality works, like enabling and stopping CPUs and preventing all of them from being disabled at the same time. * virtiorng The BATS script implementing the test is ``test-virtiorng.bats``. Check that the virtio-rng device is available through the filesystem and that it is able to generate random numbers when required. * Virtualization: * Linux Domain * Same tests as Baremetal Linux * shmem The BATS script implementing the test is ``test-shmem.bats``. Check that the ``xen_mem`` device is available. * evtchn The BATS script implementing the test is ``test-evtchn.bats``. Check that the ``xen/evtchn`` device is available. The Basic Tests are built and installed in the image according to the following BitBake recipe: :repo:`meta-armv8r64-extras/recipes-refstack-tests/basic-tests/basic-tests.bb`. ******************************** Integration Tests Implementation ******************************** This section gives a high-level description of how the integration testing logic is implemented. To enable the integration tests, the `testimage.bbclass`_ is used. This class supports running automated tests against images. The class handles loading the tests and starting the image. .. _testimage.bbclass: https://docs.yoctoproject.org/langdale/ref-manual/classes.html#testimage-bbclass The `Writing New Tests`_ section of the Yocto Manual explains how to write new tests when using the ``testimage.bbclass``. These are placed under :repo:`meta-armv8r64-extras/lib/oeqa/runtime/cases` and will be selected by the different machines/configurations by modifying the ``TEST_SUITES`` variable. For example, the file :repo:`meta-armv8r64-extras/classes/refstack-virtualization-tests.bbclass` adds different combinations of test cases to the ``TEST_SUITES`` variable according to the build parameters of the virtualizaiton stack. .. _Writing New Tests: https://docs.yoctoproject.org/langdale/dev-manual/common-tasks.html#writing-new-tests For the Basic Tests, the ``basic-tests.bb`` that inherits the Ptest framework/class and installs the BATS test files located under :repo:`meta-armv8r64-extras/recipes-refstack-tests/basic-tests`. Each image recipe (**Baremetal Linux** or **Virtualization**) includes ``basic-tests.bb``.