System hang occurs after UDP Tx, when using Tx descriptor writeback

[ricera]

System console becomes unresponsive after running UDP Tx using netperf or tftp.

• ssh/mosh sessions continue to function, and programs can be run in them
• kldunload if_ixl or shutting down the system will kill ssh/mosh sessions, but the system will hang and not unload the driver/shutdown the system
• no error/watchdog messages appear in the system log
• when running netperf in demo mode, the interim throughput shown is higher than the physical capability of the port (e.g. 14Gb/s throughput on a 10G port)

Hardware:

• Occurs on 10/25 GbE cards, but not 40GbE cards

[ricera]

These sysctls look interesting:

dev.ixl.1.iflib.txq00.ring_state: pidx_head: 1668 pidx_tail: 1668 cidx: 1670 state: STALLED

• There are only 1024 tx/rx descriptors by default, so these indices look weird
• EDIT: The buf_ring is created with a size of 2048, so these indices would be valid

dev.ixl.1.iflib.txq00.r_drops: 32212354

• If I run netperf UDP_STREAM again after noticing the system hang, this is the only queue stat that increments -- this matches Drew's suggestion that this may be an issue with how drops are being handled

[W8BSD]

So r_drops indicates that there was no room in the mp ring to add the buffer. It sounds like the ring is simply not draining. If you enable INVARIANTS, you'll get some extra counters which will include the number of encap failures.

[W8BSD]

Or just manually define IFLIB_DEBUG_COUNTERS to 1 in iflib.c

[W8BSD]

FYI, the additional counters show up under net.iflib not dev.X.Y.iflib

[ricera]

Manually defining IFLIB_DEBUG_COUNTERS to 1 made the problem go away for me. :/

[ricera]

Belay that -- after a dozen tries of netperf, I see the hang again

[ricera]

net.iflib.tx_seen seems to increment pretty rapidly.

The igb interface I'm using for remote access is affecting all of the other debug counters, but that one seems to be going up by a couple hundred thousand per second, compared to tx_encap, which only increments by one every second (which I guess might be due to igb and me using mosh).

The encap_ and txq_drain_ sysctls are all at 0.

[ricera]

I was going to try using dtrace, but after the script runs for ~30s it stops with the message:

dtrace: processing aborted: Abort due to systemic unresponsiveness

That's still enough time to start checking what functions are getting called while the system is hanging, I think.

[ricera]

ixl_if_tx_queue_intr_enable:entry
ixl_isc_txd_credits_update:return (retval = 0)
_task_fn_tx:entry  (mtx name for txq = ixl1:tx(0):call)
ifmp_ring_check_drainage:entry
ifmp_ring_enqueue:return (retval = 55 / ENOBUFS)

These functions are all getting called rapidly during the hang, according to dtrace probes

[ricera]

I'm going to agree with Drew's guess: iflib is detecting that the queues are hung, and while trying to recover the queues, it gets into a deadlock over the txq lock and the ctx lock.

But why are the Tx queues hanging? Are one of the Tx/Rx functions in the driver not returning the right values for iflib to use? Are Tx interrupts not working correctly, maybe?

[ricera]

Updates:

• Drew found that reverting to head writeback for TX descriptor completion reporting stopped the hangs; I'll integrate that into another commit soon. We still need descriptor writeback to work for the future, though, so I'll continue to investigate, leading into...

• I found that adding "#define NO_64BIT_ATOMICS 1" to mp_ring.h caused the hangs to stop. There might be an issue there, but I don't know why that wouldn't have shown up in em or ixgbe.

[jepiepe]

Disregard, I think this was an environment issue on my part :)