Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Why is the XDP_TX packet dropped by the kernel? #5

Open
someonebw opened this issue Aug 24, 2023 · 8 comments
Open

Why is the XDP_TX packet dropped by the kernel? #5

someonebw opened this issue Aug 24, 2023 · 8 comments

Comments

@someonebw
Copy link

Jamesits 你好，我又来了：）

测试环境：
debian11

iproute2/oldstable,now 5.10.0-4 amd64 [installed,automatic]

root@debian11:~# cat /boot/config-5.10.0-25-cloud-amd64 |grep BTF
CONFIG_DEBUG_INFO_BTF=y

root@debian11:~# ip -V
ip utility, iproute2-5.9.0, libbpf 0.3.0

gre tunnel config：
local ip：172.19.92.248
remote ip：172.19.100.254

1.//bpftool prog traclog的结果（libbpf测试过0.0.8的版本，1.2.0的版本，情况一样）
处理in的数据包 ，然后截取head后，xdp_tx 。

      <idle>-0       [001] d.s. 69839.759732: bpf_trace_printk: New packet

      <idle>-0       [001] dNs. 69839.759767: bpf_trace_printk: Packet header dump:

      <idle>-0       [001] dNs. 69839.759770: bpf_trace_printk: #0: 45

      <idle>-0       [001] dNs. 69839.759771: bpf_trace_printk: #1: c0

      <idle>-0       [001] dNs. 69839.759772: bpf_trace_printk: #2: 0

      <idle>-0       [001] dNs. 69839.759773: bpf_trace_printk: #3: 30

      <idle>-0       [001] dNs. 69839.759774: bpf_trace_printk: #4: a

      <idle>-0       [001] dNs. 69839.759775: bpf_trace_printk: #5: 1f

      <idle>-0       [001] dNs. 69839.759775: bpf_trace_printk: #6: 0

      <idle>-0       [001] dNs. 69839.759776: bpf_trace_printk: #7: 0

      <idle>-0       [001] dNs. 69839.759777: bpf_trace_printk: #8: ff

      <idle>-0       [001] dNs. 69839.759778: bpf_trace_printk: #9: 2f

      <idle>-0       [001] dNs. 69839.759779: bpf_trace_printk: #10: 96

      <idle>-0       [001] dNs. 69839.759780: bpf_trace_printk: #11: a2

      <idle>-0       [001] dNs. 69839.759781: bpf_trace_printk: #12: ac

      <idle>-0       [001] dNs. 69839.759782: bpf_trace_printk: #13: 13

      <idle>-0       [001] dNs. 69839.759782: bpf_trace_printk: #14: 64

      <idle>-0       [001] dNs. 69839.759783: bpf_trace_printk: #15: fe

      <idle>-0       [001] dNs. 69839.759784: bpf_trace_printk: #16: ac

      <idle>-0       [001] dNs. 69839.759785: bpf_trace_printk: #17: 13

      <idle>-0       [001] dNs. 69839.759786: bpf_trace_printk: #18: 5c

      <idle>-0       [001] dNs. 69839.759786: bpf_trace_printk: #19: f8

      <idle>-0       [001] dNs. 69839.759787: bpf_trace_printk: #20: 0

      <idle>-0       [001] dNs. 69839.759788: bpf_trace_printk: #21: 0

      <idle>-0       [001] dNs. 69839.759789: bpf_trace_printk: #22: 8

      <idle>-0       [001] dNs. 69839.759789: bpf_trace_printk: #23: 0

      <idle>-0       [001] dNs. 69839.759790: bpf_trace_printk: #24: 45

      <idle>-0       [001] dNs. 69839.759791: bpf_trace_printk: #25: c0

      <idle>-0       [001] dNs. 69839.759792: bpf_trace_printk: #26: 0

      <idle>-0       [001] dNs. 69839.759793: bpf_trace_printk: #27: 18

      <idle>-0       [001] dNs. 69839.759793: bpf_trace_printk: #28: a

      <idle>-0       [001] dNs. 69839.759794: bpf_trace_printk: #29: 1e

      <idle>-0       [001] dNs. 69839.759795: bpf_trace_printk: #30: 0

      <idle>-0       [001] dNs. 69839.759796: bpf_trace_printk: #31: 0

      <idle>-0       [001] dNs. 69839.759797: bpf_trace_printk: Outer GRE flags=0x0 proto=8

      <idle>-0       [001] dNs. 69839.759799: bpf_trace_printk: IPv4 packet_size=0x14, proto=0x2f

      <idle>-0       [001] dNs. 69839.759800: bpf_trace_printk: Inner is GRE4, proto=0

      <idle>-0       [001] dNs. 69839.759801: bpf_trace_printk: GRE4 keepalive received!

但是。
#xdp_tx的包，有问题，导致gre2卡的dropped队列增加。

gre2: flags=209<UP,POINTOPOINT,RUNNING,NOARP> mtu 1476
inet 6.6.6.2 netmask 255.255.255.0 destination 6.6.6.2
inet6 fe80::5efe:ac13:5cf8 prefixlen 64 scopeid 0x20
unspec AC-13-5C-F8-00-00-00-55-00-00-00-00-00-00-00-00 txqueuelen 1000 (UNSPEC)
RX packets 13839 bytes 332136 (324.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 36 bytes 2240 (2.1 KiB)
TX errors 0 dropped 9457 overruns 0 carrier 0 collisions 0

3.通过nettrace 对核心丢包，进行分析，如下

//[69535.293726] [enqueue_to_backlog ] ether protocol: 44051

***************** ffff88fec0aeb900 ***************
[69535.293595] [__netif_receive_skb_core] GRE: 172.19.100.254 -> 172.19.92.248
[69535.293618] [ip_rcv_core ] GRE: 172.19.100.254 -> 172.19.92.248
[69535.293631] [ip_route_input_slow ] GRE: 172.19.100.254 -> 172.19.92.248
[69535.293661] [fib_validate_source ] GRE: 172.19.100.254 -> 172.19.92.248
[69535.293674] [ip_local_deliver ] GRE: 172.19.100.254 -> 172.19.92.248
[69535.293684] [ip_local_deliver_finish] GRE: 172.19.100.254 -> 172.19.92.248
[69535.293726] [enqueue_to_backlog ] ether protocol: 44051
[69535.293740] [__netif_receive_skb_core] ether protocol: 44051
[69535.293754] [netif_receive_generic_xdp] ether protocol: 44051
[69535.293904] [kfree_skb ] ether protocol: 44051

***************** ffff88fec0aeb900 ***************

xdpdump 抓包的数据如下：

XDP_TX的是
异常数据？

这里type是ac 13（十进制的44051）

0000 45 c0 00 30 aa f4 00 00 ff 2f f5 cc ac 13 64 fe
0010 ac 13 5c f8 00 00 08 00 45 c0 00 18 aa f3 00 00
0020 ff 2f f5 e5 ac 13 5c f8 ac 13 64 fe 00 00 00 00

5.问题：为什么
if (bpf_xdp_adjust_head(ctx, (int)(cutoff_pos - data_start))) return -1;
action = XDP_TX;
拆解head后的包，是异常包。
导致kernel处理的时候，作为异常包给直接drop了。
@Jamesits
Copy link
Owner

感谢耐心测试！这个问题很奇怪，让我研究一下~

@someonebw
Copy link
Author

感激你的答复，[Jamesits]
其实之前研究这个问题，也是进行到中途过程中，出现了一些波折，导致跟不下去了，而中断了研究。

之前那次，也是感觉xdp-tx发包了，但是因为当时，缺乏调试手段（dropwatch，xdpdump，nettrace这几个工具，都不会用）
这次，掌握这些调试技术后，才能够更好的，全面分析这个问题。

并且，还碰到一个比较奇怪的问题。（原因不明）
在某些版本的kernel的环境下。明明是一个正常的来自remote ip的keepalive包，但是
keepalive_gre.c程序解出来的结果，确是“Packet size too small, dump failed”。

@someonebw
Copy link
Author

不能正常解包的kernel如下的版本，测试过的如下：
"Packet size too small, dump failed"
6.4.11-1.el7.elrepo.x86_6
5.14.0-352.el9.x86_64

可以正常解包的kernel版本如下：
5.4.253-1.el7.elrepo.x86_6
5.10.0-25-cloud-amd64 #1 SMP Debian 5.10.191-1
Debian 4.19.249-2

@Jamesits
Copy link
Owner

感谢推荐调试工具，这几个工具我之前也不了解，得看一下文档~

Packet size too small 是因为包长度小于 DEBUG_PRINT_HEADER_SIZE，可以把 DEBUG_PRINT_HEADER_SIZE 改小一点看看能不能把内容打出来。或者在那加个动态判断也行（比较难写，因为 BPF 必须 unroll 循环）

@someonebw
Copy link
Author

（dropwatch，xdpdump，nettrace这几个工具都很好用）
以下建议，供参考。
xdpdump抓包方便，dropwatch里面的dwdump也可以抓包（kernel支持btf的情况下）
下载地址：
xdpdump（https://github.com/xdp-project/xdp-tools）
dropwatch（https://github.com/nhorman/dropwatch）
nettrace（https://github.com/OpenCloudOS/nettrace）（腾讯开源工具，做的蛮好的）

最好用的是nettrace，可以直接打印出drop包的堆栈。（也最好需要核心支持btf），有安装包，可以直接装了，开箱即用。

@someonebw
Copy link
Author

一个正常的。来自remote ip的keepalive包的长度如下：

#包体长度如下
ethhd 14byte
#如下包体长度（20+4+20+4=48byte）
outiphd 20 byte
gre 4byte
iniphd 20 byte
gre 4byte

debian11 5.10.0-25-cloud-amd64版本的核心下。
解包正常，

DEBUG_PRINT_HEADER_SIZE这里，我最大可以配置成48都是正常的。

但是，5.14.0-352.el9.x86_64版本下。
同样长度的包体。
解包异常

DEBUG_PRINT_HEADER_SIZE这里，我最大只能配置成24。

配置成24后，不会提示"Packet size too small, dump failed\n"了。
但是解包异常的。
bpf_trace_printk: Outer GRE flags=0x0 pro to=0
bpf_trace_printk: Unknown proto 0 inside

图片

DEBUG_PRINT_HEADER_SIZE这里的限制，是kernel的哪里控制的呢？疑问！！！

@someonebw
Copy link
Author

someonebw commented Aug 28, 2023 •
edited
Loading

新的实验结果
感觉测试的这个，linux的kernel，已经实现了gre keepalive的回包逻辑，不需要我们通过附加xdp程序到gre接口上了。
不知道是ip_gre实现的，还是在哪里实现的。

测试环境如下：
Debian11
debian11 5.10.0-25-cloud-amd64
ip utility, iproute2-5.9.0, libbpf 0.3.0
remote gre ip(HuaWei FW)

###验证的结果：（必要的条件，必须要配置；是否需要加载xdp相关的程序：不需要）
net.ipv4.conf.gre2.accept_local = 1
net.ipv4.conf.gre2.forwarding = 1
net.ipv4.conf.gre2.rp_filter = 0

#配置如下的时候

sysctl -w net.ipv4.conf.gre2.accept_local=1
sysctl -w net.ipv4.conf.gre2.forwarding=1
sysctl -w net.ipv4.conf.gre2.rp_filter = 0

#有正常的gre keepalive 回包。
#nettrace的trace结果如下：
[5490.409110] [napi_gro_receive_entry] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409165] [dev_gro_receive ] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409182] [__netif_receive_skb_core] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409194] [tpacket_rcv ] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409209] [ip_rcv_core ] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409224] [ip_route_input_slow ] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409242] [fib_validate_source ] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409256] [ip_local_deliver ] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409267] [ip_local_deliver_finish] GRE: 172.19.100.254 -> 172.19.92.248
[5490.409288] [napi_gro_receive_entry] ether protocol: 44051
[5490.409298] [dev_gro_receive ] ether protocol: 44051
[5490.409309] [__netif_receive_skb_core] ether protocol: 44051
[5490.409325] [ip_rcv_core ] ether protocol: 44051
[5490.409334] [ip_route_input_slow ] ether protocol: 44051
[5490.409343] [fib_validate_source ] ether protocol: 44051
[5490.409358] [ip_forward ] ether protocol: 44051
[5490.409376] [ip_output ] ether protocol: 44051
[5490.409386] [nf_hook_slow ] ether protocol: 44051 ipv4 in chain: POST_ROUTING
[5490.409403] [ip_finish_output ] ether protocol: 44051
[5490.409412] [ip_finish_output2 ] ether protocol: 44051
[5490.409422] [__dev_queue_xmit ] ether protocol: 23155
[5490.409436] [sch_direct_xmit ] GRE: 172.19.92.248 -> 172.19.100.254 queue state: 0, flags: 174, last update: 212ms, len: 0
[5490.409450] [dev_hard_start_xmit ] GRE: 172.19.92.248 -> 172.19.100.254 skb is successfully sent to the NIC driver
[5490.409461] [skb_clone ] GRE: 172.19.92.248 -> 172.19.100.254
[5490.409506] [tpacket_rcv ] GRE: 172.19.92.248 -> 172.19.100.254
[5490.409518] [consume_skb ] GRE: 172.19.92.248 -> 172.19.100.254
[5490.409676] [consume_skb ] GRE: 172.19.92.248 -> 172.19.100.254

#tcpdump (结果）
07:38:45.083884 IP (tos 0xc0, ttl 255, id 37900, offset 0, flags [none], proto GRE (47), length 48)
172.19.100.254 > 172.19.92.248: GREv0, Flags [none], length 28
IP (tos 0xc0, ttl 255, id 37899, offset 0, flags [none], proto GRE (47), length 24)
172.19.92.248 > 172.19.100.254: GREv0, Flags [none], length 4
gre-proto-0x0
0x0000: 45c0 0030 940c 0000 ff2f 0cb5 ac13 64fe
0x0010: ac13 5cf8 0000 0800 45c0 0018 940b 0000
0x0020: ff2f 0cce ac13 5cf8 ac13 64fe 0000 0000
07:38:45.084174 IP (tos 0xc0, ttl 254, id 37899, offset 0, flags [none], proto GRE (47), length 24)
172.19.92.248 > 172.19.100.254: GREv0, Flags [none], length 4
gre-proto-0x0
0x0000: 45c0 0018 940b 0000 fe2f 0dce ac13 5cf8
0x0010: ac13 64fe 0000 0000

#配置如下的时候
sysctl -w net.ipv4.conf.gre2.accept_local=0
sysctl -w net.ipv4.conf.gre2.forwarding=1
sysctl -w net.ipv4.conf.gre2.rp_filter=0

##无正常的gre keepalive 回包。

#nettrace的trace结果如下：
[492.939232] [napi_gro_receive_entry] GRE: 172.19.100.254 -> 172.19.92.248
[492.939272] [dev_gro_receive ] GRE: 172.19.100.254 -> 172.19.92.248
[492.939288] [__netif_receive_skb_core] GRE: 172.19.100.254 -> 172.19.92.248
[492.939299] [tpacket_rcv ] GRE: 172.19.100.254 -> 172.19.92.248
[492.939313] [ip_rcv_core ] GRE: 172.19.100.254 -> 172.19.92.248
[492.939325] [ip_route_input_slow ] GRE: 172.19.100.254 -> 172.19.92.248
[492.939338] [fib_validate_source ] GRE: 172.19.100.254 -> 172.19.92.248
[492.939350] [ip_local_deliver ] GRE: 172.19.100.254 -> 172.19.92.248
[492.939360] [ip_local_deliver_finish] GRE: 172.19.100.254 -> 172.19.92.248
[492.939456] [napi_gro_receive_entry] ether protocol: 44051
[492.939471] [dev_gro_receive ] ether protocol: 44051
[492.939482] [__netif_receive_skb_core] ether protocol: 44051
[492.939492] [ip_rcv_core ] ether protocol: 44051
[492.939502] [ip_route_input_slow ] ether protocol: 44051
[492.939513] [fib_validate_source ] ether protocol: 44051
[492.943448] [kfree_skb ] ether protocol: 44051

#配置如下的时候
sysctl -w net.ipv4.conf.gre2.accept_local=1
sysctl -w net.ipv4.conf.gre2.forwarding=0
sysctl -w net.ipv4.conf.gre2.rp_filter=0

##无正常的gre keepalive 回包。
#nettrace的trace结果如下：
[573.021961] [napi_gro_receive_entry] GRE: 172.19.100.254 -> 172.19.92.248
[573.022001] [dev_gro_receive ] GRE: 172.19.100.254 -> 172.19.92.248
[573.022018] [__netif_receive_skb_core] GRE: 172.19.100.254 -> 172.19.92.248
[573.022029] [tpacket_rcv ] GRE: 172.19.100.254 -> 172.19.92.248
[573.022044] [ip_rcv_core ] GRE: 172.19.100.254 -> 172.19.92.248
[573.022057] [ip_route_input_slow ] GRE: 172.19.100.254 -> 172.19.92.248
[573.022073] [fib_validate_source ] GRE: 172.19.100.254 -> 172.19.92.248
[573.022086] [ip_local_deliver ] GRE: 172.19.100.254 -> 172.19.92.248
[573.022096] [ip_local_deliver_finish] GRE: 172.19.100.254 -> 172.19.92.248
[573.022142] [napi_gro_receive_entry] ether protocol: 44051
[573.022153] [dev_gro_receive ] ether protocol: 44051
[573.022164] [__netif_receive_skb_core] ether protocol: 44051
[573.022173] [ip_rcv_core ] ether protocol: 44051
[573.022182] [ip_route_input_slow ] ether protocol: 44051
[573.022196] [kfree_skb ] ether protocol: 44051

#配置
net.ipv4.conf.gre2.accept_local = 1
net.ipv4.conf.gre2.forwarding = 1
net.ipv4.conf.gre2.rp_filter = 1
的时候，报如下错误。

##无正常的gre keepalive 回包。
#nettrace的trace结果如下
#[285.557748] GRE: 172.19.92.247 -> 172.19.100.254, reason: IP_RPFILTER, ip_rcv_finish_core.constprop.0+0x1d7

@someonebw someonebw mentioned this issue Aug 28, 2023
Open
@someonebw
Copy link
Author

###使用linux kernel 自带的 gre keepalive 回包（必要的条件，必须要配置；）
net.ipv4.conf.gre2.accept_local = 1
net.ipv4.conf.gre2.forwarding = 1
net.ipv4.conf.gre2.rp_filter = 0

测试通过的环境如下：
测试环境如下：
Centos 9
Linux localhost 6.1.47-1.el9.elrepo.x86_64
ip utility, iproute2-6.2.0, libbpf 1.2.0
remote gre ip(HuaWei FW)

Centos 9
5.14.0-160.el9.x86_64
ip utility, iproute2-6.2.0, libbpf 1.2.0
remote gre ip(HuaWei FW)

Debian11
debian11 5.10.0-25-cloud-amd64
ip utility, iproute2-5.9.0, libbpf 0.3.0
remote gre ip(HuaWei FW)

Debian10
debian10 4.19.0-21-amd64
ip utility, iproute2-6.2.0, libbpf 1.2.0
remote gre ip(HuaWei FW)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Jamesits @someonebw