Severe performance degradation when TCP is funneled over TCP (GSO/TSO)

[msune]

Summary

There is a severe performance degradation when TCP is funneled over TCP on flows within the same host.

The problem is very similar to #11, and has to do with GSO. On a stablished TCP Connection (over TCP via sfunnel), packets are marked as GSOed with SKB_GSO_TCPV4. On low volume traffic, GSO doesn't kick in.

The problem manifests when more traffic is sent and the Kernel attemps to GSO. Since we are pushing a TCP header with SYN=1 and invalid - precisely to avoid the kernel to do GSO/TSO, as we don't want sfunnel to hold connection state -, becuse the skb still has SKB_GSO_TCPV4 the kernel will attempt to GSO, and will fail miserably (as SYN==1, all seq numbers are broken etc..).

Please note that there seems to be a bug in the kernel; disabling all GSO/TSO offloads keeps marking egress SKBs as TCP_GSO

[msune]

Fixing the issue

bpf_skb_change_tail()

As described here, bpf_skb_change_tail() could be used precisely for this purpose, but then packets are dropped due to MTU issues.

✔️ Don't push a new header, set a flow_id in the pkt

For TCP over TCP, the only thing needed to keep flow affinity is to rewrite the dest_port, and revert this on the other end.

msune/ebpf_gso:tcp_in_tcp prototype encodes a "flow id" as part of the TCP reserved bits (which should be 0x0). Based on the flow_id, the other end can undo the translation. The other Other options could be using IP diffserv for the same purpose (all other options, e.g. abusing ip->id etc, don't work).

The sfunnel ruleset syntax should to accept a flow_id as part of funnel/unfunnel actions, and only be valid for TCP (and make sure it matches TCP), e.g. for traffic in one direction:

ip tcp dport 8080 actions funnel   tcp flow-id 1 dport 80
ip tcp dport 80   actions unfunnel tcp flow-id 1 dport 8080

Waiting for #11 to be fixed / workarounded in case more changes to syntax are needed.