Running KVM. QEMU provides the system emulation for KVM, it can emulate a number of SoCs, but in this case I want a pure virtual machine. The OS was setup here.

Requirements:

  • Fast network IO with a guest IP Address unique from the host.
  • Boot from a raw disk image in the host filesystem.
  • Fast block io access to the whole disk image for an external USB raid.
  • Access to the serial console to check boot and login if network config fails.
  • Using a small fraction, ~25%, of the host resources (cpu, memory).

Don’t need:

  • Graphical interface, framebuffer, input devices, USB.
  • Hot plug CPU, memory.

References:

Host OS Preparation

The host OS was compiled with KVM support and VFIO support previously (cross-compiling the kernel). It also needs MACVTAP support for the efficient bridged networking. None of those are included in the default rk3399 config.

  • CONFIG_KVM=y

  • CONFIG_VFIO_PCI=y, CONFIG_VFIO_AMBA=y

  • CONFIG_MACVTAP=y 
    > Device drivers
> Network device support
[x] MAC-VLAN support 
  [x] MAC-VLAN based tap driver

Network Setup

A macvtap interface needs to be created. This emulates a separate MAC address on the existing ethernet interface - using promiscuous mode - inside the kernel. (something similar could be done in user space via a raw socket - but with worst performance)

Before running QEMU:

if ip link show macvtap0 > /dev/null ; then
 echo "SETUP macvlan0 - DONE"
else
  echo "SETUP macvlan0 - START"
  sudo ip link add link wlan0 name macvtap0 type macvtap
  sudo ip link set macvtap0 address 00:12:04:de:ad:01 up
fi

tap=`find /sys/class/net/macvtap0/ -type d -name "tap*" -printf "%f\n" | tr -d '/'`

Notes:

  • On the rk3399 eth0 seems to be unstable, and the mactap does not always pass packets, wlan0 was more consistent.
  • The /sbin/ip command is provided by the iproute2 package.
  • The $tap variable holds the name of the /dev/tap device to be used by QEMU.
  • A unique MAC address should be chosen.

After exiting QEMU:

sudo ip link delete macvtap0

QEMU Invocation

Some notes:

  • The machine type is “virt”, and cpu is “host” with “KVM” acceleration, everything should be running on the native ARM cores.
  • The Firefly-RK3399 has 4G, so I’ve allocated 25% to the VM. (-m 1024)
  • There are 6 cores on the RK3399, so 33% have been allocated to this VM. (-smp 2)
  • All devices have been allocated as virtio, except serial. (-net, -drive, -object rng)
  • QEMU by default emulates an ARM pl011 for the serial interface (ttyAMA0). Virtio is another option, but the early boot log is easier to see via the pl011.
  • The “-append” option is needed to set the kernel boot args to find the /dev/vda boot disk. The -drive option for the root filesystem must have “index=0”.
    • The boot log shows the device “[ 1.890600] fe00 8388608 vda [ 1.891363] driver: virtio_blk “
  • The serial interface is stdio, this is easy to debug but should be changed if setup as a daemon.
  • The ARM PSCI interface is used, so -no-acpi is set. (This matches SoCs using ARM Trusted Firmware)
sudo qemu-system-aarch64 \
    -m 1024 \
    -machine virt \
    -cpu host \
    -smp 2 \
    -accel kvm \
    -enable-kvm \
    -nographic \
    -no-acpi \
    -net nic,model=virtio,macaddr=00:12:04:de:ad:01 -net tap,fd=3 3<>/dev/${tap} \
    -kernel Image.4.9.164.gz \
    -drive file=rootfs.img,format=raw,if=virtio,id=drive-virtio-disk0,index=0 \
    -drive file=/dev/sda,format=raw,if=virtio,id=drive-virtio-disk1,index=1 \
    -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-device,rng=objrng0,id=rng0 \
    -append "root=fe00 rw" \
    -pidfile run.pid

Result


xxxx@Firefly-RK3399:/vm/armbian$ sudo ./run-4.9.164.sh
SETUP macvlan1 - START
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.164 (xxxx@Firefly-RK3399) (gcc version 7.3.0 (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04) ) #1 SMP PREEMPT Sun Mar 24 00:13:03 ACDT 2019
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 16 MiB at 0x000000007f000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] percpu: Embedded 21 pages/cpu @ffff80003efc0000 s47000 r8192 d30824 u86016
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 258048
[    0.000000] Kernel command line: root=fe00 rw
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Memory: 1002416K/1048576K available (7740K kernel code, 470K rwdata, 2476K rodata, 512K init, 253K bss, 29776K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff000008810000   (  7744 KB)
[    0.000000]     .rodata : 0xffff000008810000 - 0xffff000008a80000   (  2496 KB)
[    0.000000]       .init : 0xffff000008a80000 - 0xffff000008b00000   (   512 KB)
[    0.000000]       .data : 0xffff000008b00000 - 0xffff000008b75a00   (   471 KB)
[    0.000000]        .bss : 0xffff000008b75a00 - 0xffff000008bb4f34   (   254 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fb000 - 0xffff7dfffec00000   (  4116 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e0001000000   (    16 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff800040000000   (  1024 MB)

....

Debian GNU/Linux 9 armbian-old-raid ttyAMA0

armbian-old-raid login: xxxx
Password:
Last login: Tue Mar 19 11:44:10 UTC 2019 on ttyAMA0
u ____            _    ____               __   _  _
|  _ \ ___   ___| | _|  _ \ _ __ ___    / /_ | || |
| |_) / _ \ / __| |/ / |_) | '__/ _ \  | '_ \| || |_
|  _ < (_) | (__|   <|  __/| | | (_) | | (_) |__   _|
|_| \_\___/ \___|_|\_\_|   |_|  \___/   \___/   |_|


Welcome to ARMBIAN 5.73 stable Debian GNU/Linux 9 (stretch) 4.9.164
Linux armbian-old-raid 4.9.164 #1 SMP PREEMPT Sun Mar 24 00:13:03 ACDT 2019 aarch64
System load:   0.74 0.16 0.05   Up time:       0 min
Memory usage:  3 % of 995MB     IP:            192.168.1.110
Usage of /:    13% of 7.9G

xxxx@armbian-old-raid:~$ uname -a
Linux armbian-old-raid 4.9.164 #1 SMP PREEMPT Sun Mar 24 00:13:03 ACDT 2019 aarch64 GNU/Linux
xxxx@armbian-old-raid:~$ cd /proc/device-tree
xxxx@armbian-old-raid:/proc/device-tree$ ls
#address-cells    pl011@9000000        virtio_mmio@a000c00  virtio_mmio@a002600
apb-pclk          pl031@9010000        virtio_mmio@a000e00  virtio_mmio@a002800
chosen            pl061@9030000        virtio_mmio@a001000  virtio_mmio@a002a00
compatible        platform@c000000     virtio_mmio@a001200  virtio_mmio@a002c00
cpus              psci                 virtio_mmio@a001400  virtio_mmio@a002e00
flash@0           #size-cells          virtio_mmio@a001600  virtio_mmio@a003000
fw-cfg@9020000    timer                virtio_mmio@a001800  virtio_mmio@a003200
gpio-keys         virtio_mmio@a000000  virtio_mmio@a001a00  virtio_mmio@a003400
intc              virtio_mmio@a000200  virtio_mmio@a001c00  virtio_mmio@a003600
interrupt-parent  virtio_mmio@a000400  virtio_mmio@a001e00  virtio_mmio@a003800
memory            virtio_mmio@a000600  virtio_mmio@a002000  virtio_mmio@a003a00
name              virtio_mmio@a000800  virtio_mmio@a002200  virtio_mmio@a003c00
pcie@10000000     virtio_mmio@a000a00  virtio_mmio@a002400  virtio_mmio@a003e00
xxxx@armbian-old-raid:/proc/device-tree$
xxxx@armbian-old-raid:/proc/device-tree$ lspci
00:00.0 Host bridge: Red Hat, Inc. Device 0008
00:01.0 Ethernet controller: Red Hat, Inc Virtio network device
00:02.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:03.0 SCSI storage controller: Red Hat, Inc Virtio block device
xxxx@armbian-old-raid:/proc/device-tree$ lscpu
Architecture:          aarch64
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
Model:                 2
BogoMIPS:              48.00
Flags:                 fp asimd evtstrm aes pmull sha1 sha2 crc32

A few more packages were installed :

sudo apt update
sudo apt install armbian-config

Exit with shutdown command:

xxxx@armbian-old-raid:/proc/device-tree$ shutdown -t now
[  OK  ] Stopped target Graphical Interface.
...
[  517.969840] systemd-shutdown[1]: All loop devices detached.
[  517.977828] reboot: Power down
xxxx@Firefly-RK3399:/vm/armbian$

Remaining Questions

These are some questions remaining after setting this up:

  • The command fails 50% of the time with an error: “kvm_arm_vcpu_init failed: Invalid argument”
    • Is this related to the big-little architecture?
    • The VM could be allocated A72 cores or A52 cores, should a cpumask be used to limit that?
  • Macvlan does not provide reliable networking yet.
    • It seems to only work for one invocation.
    • For subsequent invocations ARP and other packets are not responded to (TX works, RX does not).
  • Virtio-pci is the default virtio mode for “-machine virt”.
    • What is the benefit of this? An ARM64 SoC would generally use MMIO for all peripherals.
    • (Although high end devices do have PCIe ports, those are usually on the same MMIO bus as other peripherals.)
  • An ARM64 SoC would generally have DMA for network and block devices. Can that be setup with QEMU?
  • Is ARM PSCI management interface all that is needed, do we need EFI, ACPI etc, or a EL3 passthrough?.