Conversation
|
39dfaf2 contains a brutal approach to make MTU 9000 work (without any offloading). Performs quite ok, but needs more testing |
39dfaf2 to
c702d70
Compare
c702d70 to
66f6684
Compare
|
I'm still waiting on documentation from Broadcom to read the official word on how jumbo frames with offload was meant to work (if it was). Seeing as it was the offloading headers that seemed to cause issues, I did wonder if disabling offloading would allow it to work with bigger buffers. I only had a very quick read through the patches, but wonder if we can "dynamically" disable offload when the mtu is increased above the magic threshold. Possibly not based on the comment of losing the queues as well. I had considered VLAN headers, but didn't know the answer off the top of my head, and wasn't in a position to set up VLANs to test. Thanks for taking care of it. I'm not fussed over |
|
@ffainelli and @Ryceancurry if you can spare some time, your thoughts on this would be really appreciated (as always). Thanks! |
|
https://lore.kernel.org/netdev/20260406-devel-autonomous-eee-v1-1-b335e7143711@tipi-net.de/t/
https://datasheets.raspberrypi.com/cm4/cm4-datasheet.pdf#page=7
https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54210
https://magazine.raspberrypi.com/articles/raspberry-pi-4-in-detail
https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54213
While we do understand the importance of taking one step at a time, it would be great to test whether |
Can you give me more color on the failure? Do we see fragmented packets? Is the packet corrupted? Or do we not receive a RX descriptor at all? Full disclosure, I threw AI at the RTL(I'm a SW guy), it suggests a RTL bug where the RSB is reserved at every packet ready threshold. So I wonder if we are seeing a 64B hole between each 3820B chunk within the jumbo packet. At least that is the running theory right now. I will continue to dig. |
|
Status blocks off, what's the true max frame length the MAC can handle ? |
|
Thanks for looking into this too!
With status block enabled and threshold at 0xf0 I get a descriptor (one per oversized frame): 3904 = 64 (RSB) + 2 (align) + 3838. Payload is fine and matches my test pattern. I also don't see any holes, just a hart cut off. I've also tested with 3840, 5000 and 9014 B frames and all of them produce the same descriptor. Some things I've noticed and might be worth mentioning:
For contrast, with |
I've tested further and came up with a solution which allows to switch to higher MTU on a live interface (tested 1514, 4096, 8192, 9014 B frames with threshold at 0xf0). Anyway, blocks needs to be disabled for anything higher. If I keep the TSB to preserve TX checksum offload, RX is still fine at 986 Mbit/s but TX drops to 0. A while ago I proposed to get rid of the TX queues in genet [1]. Florian and Justin reviewed and tested it, but the reasoning was not good enough. Even though the queues are not absolutely blocking it,the TSB has no queue selection role anymore and it would simplify the jumbo patch. Might be worth a v2. [1] https://lore.kernel.org/netdev/20260612205915.3156127-1-nb@tipi-net.de/ |
66f6684 to
36e861a
Compare
|
Patch tested on Audiolinux. |
As far as I can see the only limitation is the MAC's 14 bit frame len field. so 16383B I reproduced the 9000B frames with RSB enabled. I printed out the entire 9000B packet and see corruption at each PKT RDY THRESHOLD. 3838B and ~7700B. This confirms my suspicion. The HW puts a 64B header per PKT RDY THRESHOLD. I think the correct way to do this is to set rx_buf_size to PKT RDY THRESHOLD. Then use rx scatter gather with multiple descriptors. We need to strip 64B off of each fragment. Unfortunately this means a big rework on the RX side. |
|
@nbuchwitz we can test 16K with the current patch, it is just a question of ENET_MAX_JUMBO_MTU ? |
b060753 to
124162d
Compare
That helped a lot, thanks. I swapped the MTU 9000 patch for your approach and it works well. The TSB even can stay on with a little quirk. I also bumped max_mtu to what the 14 bit UMAC_MAX_FRAME_LEN allows, 16347. Do you now if the status block bug is in all GENET (non v1) versions? I only have v5 here to test. |
|
@nbuchwitz work great in 1G (I get 9184 the limitation is on other side) but configured in 100M there are packet loss with mtu > 9080 |
Could you show me the little quirk you used to keep the TSB enabled? On my Raspberry Pi 4 / GENET v5, MTU 13500 works, but around 13505 it becomes unstable and MTU 14000 fails with RX CRC errors. I noticed that a 14000 MTU results in a 14014-byte skb becoming a 14078-byte DMA buffer after the 64-byte TSB is added. |
The "quirk" is to dynamically switch of TX checksum based on the mtu (threshold is the previous 3820). See the last patch for details. Pi4 is afaik limited by the phy around 10k (see comment above). Cm4 should (theoretically) something around 18k |
Haven't tested it yet with fast ethernet. If the time permits I will do some measurements with different mtu and speed. I want to measure the cpu impact of sw checksum. For jumbo frames I assume not much of a penalty |
Thanks, that clarifies the TSB quirk. Interestingly, with two Pi4 Model B (Rev 1.1 and Rev 1.5) directly connected, I can get MTU 13500 working reliably in one direction (10/10 pings), while the opposite direction fails. Around 13503–13507 it becomes unstable/fails. So the Pi4 PHY seems capable of going significantly beyond 10k in at least some cases. Do you know what exactly imposes the ~10k PHY limit you mentioned (PHY register/buffer/specification), and whether it differs between Pi4 board revisions? |
|
Do you really see such big payload or is this already capped by phy and it just works "magically" with the 10k limit? Limit is stated in the datasheet. So I'd assume it's related to the buffer / state machine |
Yes, at least at the GENET MAC/driver level I really see the full size. For 10 successful MTU 13500 pings, txq3_packets increases by 10 and txq3_bytes by 135140, i.e. exactly 13514 bytes per packet. tx_oversize also increases by 10 on TX and rx_oversize by 10 on RX, with no additional CRC errors. I'm using However, I haven't verified on the wire between MAC and PHY, so you're right that this doesn't prove the PHY actually handles the full ~13.5K frame as such. Interestingly, Rev 1.1 -> Rev 1.5 works at MTU 13500, while Rev 1.5 -> Rev 1.1 fails, even though the receiving side counts the request and generates a 13514-byte reply. Which PHY datasheet/section states the ~10K limit? I'd like to check exactly what that limit refers to. |
|
https://www.broadcom.com/products/ethernet-connectivity/phy-and-poe/copper/gigabit/bcm54213pe
|
Thanks! yes, I confirmed that both of my Pi4s (Rev 1.1 and Rev 1.5) are using the BCM54213PE PHY ( So the 10 KB limit you mentioned is indeed the one stated in the BCM54213PE datasheet. However, MTU 13500 is really passing end-to-end in my tests: with Above that it becomes unreliable very quickly (around 13503–13507 in my tests), and at MTU 14000 it fails. I also see RX CRC errors when operating around this boundary, so this is clearly outside the PHY's guaranteed operating range. Interestingly, both Pi4 revisions use exactly the same BCM54213PE, so the different behaviour I saw between the two boards isn't explained by a different PHY model. I agree that this could be related to an internal PHY buffer/state-machine limit rather than a simple hard packet-size check. |
|
I will be able to test 16k CM4 tomorrow |
bcmgenet_hfb_init() runs INIT_LIST_HEAD() on priv->rxnfc_list, which drops every rule off the list, and bcmgenet_open() calls it on each ifup. Every rule the user configured is silently lost: # ethtool -N eth0 flow-type ether dst $MAC action 0 Added rule with ID 0 # ethtool -n eth0 | grep -c Filter: 1 # ip link set eth0 down && ip link set eth0 up # ethtool -n eth0 | grep -c Filter: 0 Initialise the lists once at probe and restore the rules on open, as bcmgenet_resume() already does. Fixes: 3e37095 ("net: bcmgenet: add support for ethtool rxnfc flows") Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
2121751 to
79b69ad
Compare
|
79b69ad should fix this hopefully. Please note if you're testing against another RPi you might want to patch both systems, as this affects the transceiver side. |
|
@herisson-88 It would be also great if you can test #7623 as this will be the version for upstream (with backports). It also contains the phy fix |
|
@nbuchwitz currently compile kernel with #7623, RTL8156BG arrived (hope there is not issue on this side to not pollute with false positive... It is supposed go to 16K - friends are using it for that.) I have also another old RPI4 that I could use with the CM4 in the other side to use your kernel patchs on both side. |
Maybe the right way to think about it is that the original one descriptor per jumbo frame wasn't meant to work. It just so happened to work with the RSB disabled. So it is safe to assume this is the correct way to do things in all revisions of genet. The RTL also corroborates. Thanks for taking this on. Good work so far! |
bcmgenet jumbo patch on CM4 — native DSD over jumbo frames, test report (2026-09-14)Three tests, all conclusive: a native DSD512 Diretta stream (2 × 22.58 MHz, ~46 Mb/s) carried in Setup
1. Native DSD512 at 1000 Mb/s, MTU 16000 on both ends — 30 s of playbackHost side, interface counters (
CM4 side,
Load average on the CM4 during playback: 0.32. Audio continuous, no dropout during the session; For scale: with the Diretta host SDK's default 2. Ping test: 12 combinations, zero lossMethod: 100 ICMPv6 echo requests per combination, DF bit set (no fragmentation allowed),
Identical result in the other direction (CM4 → host): 0 lost out of 100 in every combination. This closes the 100BASE-TX issue of the 2026-09-13 report (frames above ~9080 bytes were 3. Native DSD256 at 100 Mb/s, MTU 10000 on both ends — the case that failed on 6.18.50-1Link forced to 100 Mb/s full duplex,
Note on the MIB statistics (cosmetic, no effect on traffic)Measured on both the 16000-byte and the 10000-byte streams: every jumbo frame increments ReproductionThanks for the patch — it does exactly what was hoped for. Will test with Rpi 4 tomorow, but CM4 is 16k compliant now. |
bcmgenet jumbo patch on Raspberry Pi 4 Model B (BCM54213PE) — test report (2026-09-15)Follow-up to the CM4 report above, same kernel build, this time with the Pi 4 Model B as the Setup
1. Ping sweep at 1000 Mb/s, MTU 16000 on both ends — 1000 pings per size
Pi 4 Sizes 13 478 and 13 520 were chosen deliberately: 13 478 is where Snyder's Pi 4 started losing 2. Native DSD512 at 1000 Mb/s, MTU 16000 on both ends — real payload, both ends countedPings prove the PHY passes the frames; this proves the frames carry audio that plays. The Pi 4 Counters read on both ends around the same 31 s window, mid-album (08:42:42 → 08:43:13):
Every frame the Pi 4 put on the wire was received intact by the CM4: the 230-frame difference The ping sweep of section 1 ran while this stream was playing; neither disturbed the other. Note on the MIB statistics (same as on the CM4)Every jumbo frame increments Not covered here100 Mb/s and 10 Mb/s on the Pi 4 side (the CM4 report already covers those speeds with the PHY Thanks again — with the PHY fix, Pi 4 Model B is 16 K-clean at 1 Gb/s too. |
|
Thanks for testing. I will have a look regarding the MIB counters, but am not really confident that this can be fixed as Functionality is not affected and the interface statistics are correct (rx_packets matches and no error counter), so I'd assume this is ok. |
Thanks for the confirmation. I will mention this in my cover letter, so clashiko does not have to ask 😄 |
|
@herisson-88 may I add a |
|
If you want 🤗 but it is very small things... |
|
It is always good to see if someone else than the author has actually tested the patches. I would need the name and mail i should use for the tag |
|
Pierre-Marin Leclercq pierremarinleclercq88@gmail.com |
|
Wow, 16000MTU is fantastic achivement! |
|
First of all, thank you all so much for your continued efforts. Unfortunately, a quick smoke test (based #5561 (comment)): for ((MTU=1400; MTU<4000; ((MTU=MTU+4)) )) do sudo ifconfig eth0 mtu ${MTU} ; ifconfig eth0 ; dd if=/dev/zero bs=65536 count=10 |netcat -u 169.254.1.1 65000 -q 1; donestill causes eth0 on my RPi4 to freeze: Hopefully I'm wrong and I is just some mistake. |
|
@wtschueller I can reproduce your MTU sweep freeze here too, on the CM4 (Holo Red) from my reports above. Kernel: the same Test: your loop, MTU 1400 to 4000 in steps of 4, 10 × 65000-byte UDP datagrams per step. Two small differences: Result: 543 steps from 1400 to 3568 go through, ~0.16 s each, 10 datagrams sent every time, no error. The log stops after the step at 3568; the CM4 stopped answering (ping, ssh) during the step at 3572 and did not come back. The CM4 has a single interface, so no kernel log survived (volatile journal, log on tmpfs); I had to power-cycle it. A scripted The last successful MTU (3568) and the failing one (3572) match your Happy to run it again with a serial console attached to capture the watchdog messages, or any variant you want (fixed 3572 under load, sweep 3500 to 3600 only, TX checksum forced off). |
bcmgenet jumbo patch on CM4 — TX freeze at MTU 3572, and where it starts and stops (2026-09-17/18)Follow-up to @wtschueller's MTU sweep report. Same hardware and kernel as my two reports above. Setup
1. The MTU sweep (wtschueller's loop)Loop: MTU 1400 to 4000 in steps of 4; at each step
wtschueller's log ends with 2. Fixed-MTU probesFor each MTU: "OK" = 60 datagrams sent, Frame = MTU + 14 (Ethernet header, no VLAN, FCS not counted).
Not tested: MTU 3576 to 3596 (inside the freezing range), and MTU 4100 to 7144. 3. The next multiples of the thresholdReading the driver, the packet ready threshold on this build is bounded by the 4K page at
Same criteria as section 2: 60 datagrams each, Counts
Reproduction# on the CM4, PEER = link-local IPv6 of the peer on end0
for MTU in 3608 3604; do # 3608 works, 3604 freezes
ip link set end0 mtu $MTU
python3 - "$PEER" <<'PY'
import socket, sys
s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
for i in range(30):
s.sendto(b'\0' * 65000, (sys.argv[1], 65000, 0, socket.if_nametoindex('end0')))
PY
ping -6 -c 3 -W 1 "$PEER%end0"
ip link set end0 mtu 16000
done |
4. Sweep above the windowSame loop as section 1, MTU 3608 to 16344 in steps of 4: 3185 steps, 10 datagrams each, no freeze, no watchdog, MTU 1400 to 16344 is now covered in 4-byte steps. The only freezing range found is MTU 3572 to 3604 (frames 3586 to 3618 bytes). |
|
Thanks for reporting and already pinning it down. I will grab some coffee and have a look... |
|
Found it. The transmit packet ready threshold was derived from the MTU, and a frame ending just past it stops the transmitter for good. What actually matters is the frame length, not the MTU. A 3586 byte frame hangs at MTU 9000 too, while 3578 and 3622 are fine. MTU 3572 to 3604 is just where IP fragmentation produces those sizes. My fix pins the threshold at its maximum. Pushed to #7623 (which matches the upstream series). Looking forward to your test results. Thanks again for the reports, the fixed MTU table made this quick to find! |
|
Same sweep on a Pi 4 Model B Rev 1.4 (Pi OS Trixie,
MTU 7660 to 7700 and 11500 to 11540 (around 2 × 3840 and 3 × 3840) pass. One step at MTU 4500 took 14.6 s instead of 0.11 s, with no kernel message, then the sweep went on normally. After each freeze |
|
Not what I have hoped for and at the moment I'm not sure how to proceed: The transmitter stops for good when a frame slightly longer than TBUF_PKT_RDY_THLD is followed by a shorter one. Watchdog fires, TX DMA won't halt and every Raw frames at MTU 9000:
So roughly threshold + 2 to + 46, and it moves with the register. 300 identical 3842 B frames are fine, but 17 of them followed by one shorter frame kill it on the first burst. Unfortunately that is what IP fragmentation produces, N full frames and a short tail. Since the register tops out at 0xf0, every jumbo frame is past the threshold, so a few questions I hope @Ryceancurry can make sense of:
|
|
@herisson-88 I overlooked that part of your earlier report, sorry. First, I noticed the freezes when I played with the very old patch (global mtu change) and kernel 5.10. As far as I remember, triggering the freeze with packets of only same size took hours. import socket
import os
import subprocess
HOST = "169.254.1.1"
PORT = 5000
INTERFACE = "eth0"
mtu = 9000
subprocess.check_call([
"sudo", "ifconfig", INTERFACE,
"mtu", str(mtu)
])
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
sock.connect((HOST, PORT))
sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 2**20)
blob = os.urandom(10*1024)
for payload_size in range(1, 8192+36):
print("payload size %d" % payload_size, flush=True)
payload_1 = blob[:8192]
payload_2 = blob[:payload_size]
for _ in range(10000):
sock.send(payload_1)
sock.send(payload_2My guess was that there might be some extra settings in the MAC allowing a safe write-through of packets larger than TBUF_PKT_RDY_THLD. Something similar is already done in the PHY. |
|
Padding seems to mitigate the issue at least in my testing. I've also ran the stress test from @wtschueller and it seems to work: c0beb41 |
|
@nbuchwitz trigger the ci on the PR would facilitate testing on my side 🤗 |
|
The builds in 7623 are up to date (Phil already triggered the builds last night) |
c0beb41 on Pi 4: sweep and TX stress clean, one RX length-error patternPi 4 Model B Rev 1.4, 1. MTU sweep (10 × 65000-byte UDP datagrams per step, steps of 4)
Yesterday's freezing ranges (MTU 3832 to 3868 and 15352 to 15356) are gone. 2. Stress test with payload validationwtschueller's pattern at fixed MTU: one full-size datagram then one short datagram of variable size, N pairs per size, sender paced at 850 Mb/s. Each datagram carries a sequence number and a CRC32; the receiver checks the CRC and counts sequence gaps, reordering and duplicates. IPv6 link-local: frame = UDP payload + 62 bytes. Pi → x86 (GENET transmits)
No watchdog, no bcmgenet message, x86 → Pi (GENET receives)
The 2000 lost datagrams match 3. The lost sizesSizes around k × 3584 replayed with 1000 pairs each, then the suspects with their ±1 byte neighbours, at MTU 16000 and 9000. Every time exactly one size is lost, 1000 of 1000, neighbours arrive:
Summary: on c0beb41 a received frame of exactly k × 3584 − 1 bytes (FCS excluded, k = 1..4) is dropped with a length error; k × 3584 − 2 and k × 3584 are received. Logs available on request. |
|
Thanks for testing. The remaining RX issue is related to the already merged page pool migration (which upstream doesn't hit since max MTU is 1500). I have added another commit which hopefully should fix it. |
f90d595 on Pi 4: k × 3584 − 1 RX drop fixed, stress clean in both directionsSame setup as my previous report: Pi 4 Model B Rev 1.4, 1. The previously rejected sizes (x86 → Pi, MTU 16000, 1000 pairs per size)
24 004 datagrams sent, 24 004 received with a valid CRC, 2. Full stress test (wtschueller's pattern, MTU 9000, short payload 8 to 8228, 1000 pairs per size, sender paced at 850 Mb/s)
On c0beb41 the x86 → Pi pass lost exactly 2000 datagrams (the two k × 3584 − 1 sizes in range); it now loses none. After both passes on the Pi: 3. MTU sweep (10 × 65000-byte UDP datagrams per step, steps of 4)
Summary: f90d595 fixes the k × 3584 − 1 receive drop and I see no regression against c0beb41. Logs available on request. |
|
@nbuchwitz thanks for your hard work to fix all ! |
8 participants
Based on @6by9's #7614, with the goal of upstreaming the MTU support.
I've tested the original patch on CM4 and discovered some issues. So I've created a slightly different patch (series) which I intend to send to netdev. It also contains some fixes Sashiko would have flagged any way...
TBUF_PKT_RDY_THLD (TBUF + 0x10)is never programmed and it stays at0x80. At MTU3824TX iperf3 is stuck at 0.00 Mbit/s while ping works and the link is up.tx_pkts rises, buttx_good_pktsdoesn't. Kudos to @wtschueller who discovered this Jumbo frame support on Pi4 ethernet (Genet) #5561UMAC_MAX_FRAME_LENgets the MTU value, but it's a frame length and counts the FCS. Frames from3824up result inrx_length_errors(at least in my testing), so the real limit seems to be MTU3806.RX_BUF_LENGTH 10240costs no throughput (936/941 at MTU 1500, same as unpatched) but is above KMALLOC_MAX_CACHE_SIZE on arm64, thus it cant hurt to derive it from the MTU instead.0xf0seems to be the real limit:0xfbreceives fine but resulted in TX hard-hung on my setup.Happy to add @6by9 as Co-developed-by since it's based on your findings. But this requires a Signed-off, which I wouldn't add without consent.