[arm64] The initrd stage VNC console does not have output when running on QEMU-KVM

[ader1990]

Description

On ARM64, when booting on QEMU-KVM with VNC enabled, the VNC console does not show any output in the Flatcar initrd stage early boot stage.

Impact

No way to debug from VNC if any ignition or initrd stage error appears.

Environment and steps to reproduce

TBD.

Additional information

Please add any information here that does not fit the above format.

[jepio]

Try adding console=tty0 to grub.cfg.

[ader1990]

To give more context what happens, as there are two scenarios that behave a little bit different.

Scenario 1: ARM64 QEM-KVM Flatcar instance using qemu-kvm commands

This scenario can be easily reproduced on an ARM64 box (DO NOT USE emulation), with the command:

bash flatcar_production_qemu_uefi.sh -nographic -vnc :1 -device virtio-gpu-pci -device virtio-keyboard -device virtio-mouse

In this scenario, after the VM is started and a VNC console is opened, one can see in the VNC console, in this order: the TianoCore bootloader, Grub 2 with the Flatcar boot options (default, usr-a, usr-b), Booting Flatcar default message, and if everything went well, after some time, the login console.

NOTE that between the Booting Flatcar default message and the login console, the VNC console will display a message Display output is not active or Guest has not initialized the display output yet and sometimes, for a short time, a blinking _ (underscore) on the top left part of the screen.

Scenario 2: ARM64 QEM-KVM Flatcar instance using libvirt

Example Libvirt XML:

<domain type='kvm' id='5'>
  <name>flatcararm64</name>
  <uuid>58d48567-0cf0-448c-a519-03ccb030efd0</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://libosinfo.org/linux/2022"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='aarch64' machine='virt-6.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/var/lib/libvirt/images/flatcar-bootloader.img</loader>
    <nvram>/var/lib/libvirt/images/flatcar-nvram.img</nvram>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough' check='none'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/flatcar.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <alias name='pci.9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x11'/>
      <alias name='pci.10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x12'/>
      <alias name='pci.11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x13'/>
      <alias name='pci.12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x14'/>
      <alias name='pci.13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x15'/>
      <alias name='pci.14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='15' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='15' port='0x16'/>
      <alias name='pci.15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='16' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.16'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target type='system-serial' port='0'>
        <model name='pl011'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-flatcararm64/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
      <address type='usb' bus='0' port='2'/>
    </input>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0'/>
      <alias name='tpm0'/>
    </tpm>
    <graphics type='vnc' port='5901' autoport='no' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x10' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-58d48567-0cf0-448c-a519-03ccb030efd0</label>
    <imagelabel>libvirt-58d48567-0cf0-448c-a519-03ccb030efd0</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+108</label>
    <imagelabel>+64055:+108</imagelabel>
  </seclabel>
</domain>

Curiously, the video device as set in the libvirt config xml translates to a qemu-kvm command -device virtio-vga,id=video0,max_outputs=1,bus=pci.16,addr=0x1.

NOTE that in scenario 2, the message is Guest has not initialized the display output yet and sometimes, for a short time, a blinking _ (underscore) on the top left part of the screen. Sometimes, there is no bootloader (TianoCore and grub display).

More debug is required.

[ader1990]

Try adding console=tty0 to grub.cfg.

The ARM64 ttyAMA0 is already set correctly in the bootengine. This most likely looks like an initrd issue with either a missing kernel module not built or kernel parameter or even systemd parameter.

[ader1990]

Try adding console=tty0 to grub.cfg.

The ARM64 ttyAMA0 is already set correctly in the bootengine. This most likely looks like an initrd issue with either a missing kernel module not built or kernel parameter or even systemd parameter.

My bad, it is set correctly inhere: https://github.com/flatcar/scripts/blob/main/build_library/grub.cfg#L82

[jepio]

ttyAM0 is a serial console, vnc connects to a graphical console. If you want to see boot output on both you need to set tty0 as well

[ader1990]

ttyAM0 is a serial console, vnc connects to a graphical console. If you want to see boot output on both you need to set tty0 as well

Unfortunately, it does not work, at least on my environment - I tried to add the console=tty0 in the /oem/grub.cfg, or with the ignition.ign at first boot or in the grub bootloader screen edit, the same result, during the initrd stage, there was no output to the VNC console. There is output during the bootloaders, and after the initrd pivot, but not on the initrd stage. Maybe there is another place to add the setting?