Make networking work in SO3 under QEMU virt

[daniel-rossier]

SO3 networking has never worked on QEMU: CONFIG_NET is off in every defconfig,
and for good reason — QEMU's virt machine offers only virtio-net, which SO3 has
no driver for. Its single Ethernet driver is smc911x, so the machine has to
provide an SMSC LAN9118.

QEMU

An SMSC LAN9118 is added to the machine model next to the PL111/PL050/absmouse
devices the so3 patch already contributes, at 0x08804000 (VIRT_ETH) with
SPI 15. Two things the obvious version gets wrong:

• 0x1a000000 does not work. That is the address the SO3 device trees
  inherited from vexpress-a15, and it falls inside VIRT_PCIE_MMIO, whose gpex
  alias covers the whole window and shadows anything mapped underneath it
  (info mtree -f shows 10000000-2fffffff as one gpex_mmio_window extent).
  The guest reads back nothing and never detects the chip. Hence an address in
  the block the so3 patch already owns.
• Ordering matters. The device must be created before create_pcie(),
  which hands every nd_table entry no on-board device has claimed to
  pci_nic_init_nofail() — that one exits with Unsupported NIC model: lan9118. Slots an on-board NIC has taken are now skipped in that loop.

The MAC is only instantiated when the command line fills the legacy on-board NIC
slot, so a run without -nic/-net leaves the machine exactly as it was.
st.sh passes -nic user,model=lan9118 on its own slirp stack, alongside the
virtio-net device the Linux agency uses (which keeps the 2222 → 22 host
forward). Each guest ignores the NIC it cannot drive.

Adding a third patch to the QEMU set also exposed a .gitignore trap: the
qemu entry meant for the source tree build.sh downloads is unanchored, so it
matches the directories of that name inside the layers
(meta-qemu/recipes-qemu/qemu/) as well. The two existing patches survive only
because they were already tracked; the new one was silently swallowed by git add. The entry is now anchored as /qemu.

SO3

• device tree nodes moved to the new address/IRQ (the stale xvisor/vexpress
  reference links go with them);
• CONFIG_SMC911X was unreachable on 64-bit (depends on VIRT32);
• CONFIG_NET + CONFIG_SMC911X on in the four QEMU virt defconfigs
  (virt32/virt64, plain and _fb) — rpi4 and verdin stay off, they have no
  driver for their own MAC;
• the driver no longer built: DBG() used to come from soo/debug.h, which is
  only on the include path under CONFIG_SOO. It gets a local definition (the
  call sites also passed a status argument the format string had no conversion
  for);
• recvfrom() dropped the peer address lwIP had filled in — every reply looked
  like it came from 0.0.0.0. accept() copied it back with a memcpy of the
  lwIP layout, whose leading sa_len byte shifts sin_family. Both now go
  through the reverse of user_to_lwip_sockadd().

Testing

virt64 (virt64_fb_defconfig) and virt32 (virt32_fb_defconfig), each with
a full build.sh bsp-so3 + deploy.sh bsp-so3 and booted with st.sh — 0%
loss on both, and four consecutive runs in one session on each:

Network Interface Controller (NIC) found LAN9118
smc911x: detected LAN9118 controller
IP Network up and running with address 10.0.2.15
/ % ping -c 10 10.0.2.2
64 bytes from 10.0.2.2: icmp_seq=1 ttl=255 time=6.497070 ms
64 bytes from 10.0.2.2: icmp_seq=2 ttl=255 time=0.601074 ms
...
10 packets transmitted, 10 received, 0.000000% packet loss
rtt min/avg/max = 0.557861/1.227100/6.415039 ms

The QEMU patch set was regenerated with updiff.sh qemu and validated by a
clean build.sh -c qemu (fetch → patch → build), after which the working tree
matches the hand-edited sources byte for byte. scripts/check-format.sh is
clean.

The lost second packet

The first version of this branch lost exactly one packet per ping run — the
reply to the second echo request — and reported implausible sub-0.2 ms
round-trip times for every reply after it. Root cause: the driver programmed an
RX Status Level of 1 into FIFO_INT. The controller raises RSFL when the RX
status FIFO holds more entries than that level, so the interrupt only came
once a second frame had arrived, and reception was permanently one frame
behind:

• the ARP reply sat in the FIFO until echo reply 1 arrived and released both
  (hence the 300 ms first round-trip);
• echo reply 2 then sat there alone with nothing to trigger the interrupt, and
  recvfrom() timed out;
• echo request 3's reply released it, and from then on every recvfrom()
  returned the previous reply, already queued — hence the near-zero timings.

Level 0 (the chip's reset value) makes a single waiting frame raise RSFL.

Socket lifetime, and what ping reports

Testing the error paths turned up two more things.

The third ping run of a session lost replies for good. The VFS calls
fops->read/write/close with the global descriptor, while the socket
syscalls get the process-local one — read_sock(), write_sock() and
close_sock() ran the local→global translation anyway. For close_sock() that
landed on -1 (sys_close() releases fd_array[fd] before calling the
operation), indexing lwip_fds[] out of bounds, so lwip_close() never reached
the socket. Every run leaked its raw pcb and the packets queued on it, and with
MEMP_NUM_NETBUF at 4 the third run started timing out. The fops now have their
own accessor, close_sock() clears the slot, and lwip_fds[] is initialised to
-1 (0 being a valid lwIP socket id, a zeroed table made every descriptor look
like socket 0). Six consecutive ping -c 3 runs: 0% loss each.

ping misreported failures. A failed recvfrom() was only reported from the
second request onwards, so the first timeout of a run fell through to the reply
parsing and printed Truncated reply of -1 bytes; and ICMP errors came out as
raw numbers. Now:

/ % ping -c 3 10.0.0.1
Packet receive failed!!            (x3, 100% loss)
/ % ping -c 3 192.168.1.1
From 192.168.1.1 icmp_seq=1 Destination Port Unreachable
/ % ping -c 4 10.0.2.2
4 packets transmitted, 4 received, 0.000000% packet loss

That port-unreachable is host-side, not SO3: net.ipv4.ping_group_range
defaults to the empty range 1 0 on several distributions, so slirp cannot open
an ICMP socket and emulates the echo over UDP. Documented in the lwIP chapter.