udp: add local "peek offset enabled" flag
authorPaolo Abeni <pabeni@redhat.com>
Tue, 20 Feb 2024 11:00:01 +0000 (12:00 +0100)
committerJakub Kicinski <kuba@kernel.org>
Thu, 22 Feb 2024 01:05:01 +0000 (17:05 -0800)
commitf796feabb9f5b1e5c48780a7a0023ab4b82336dd
treef50132512a9ed9f9a500aa2b79140f5cde52e8be
parent26b2a265d888f7e345c6a461ab5f6d430791d09d
udp: add local "peek offset enabled" flag

We want to re-organize the struct sock layout. The sk_peek_off
field location is problematic, as most protocols want it in the
RX read area, while UDP wants it on a cacheline different from
sk_receive_queue.

Create a local (inside udp_sock) copy of the 'peek offset is enabled'
flag and place it inside the same cacheline of reader_queue.

Check such flag before reading sk_peek_off. This will save potential
false sharing and cache misses in the fast-path.

Tested under UDP flood with small packets. The struct sock layout
update causes a 4% performance drop, and this patch restores completely
the original tput.

Signed-off-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Link: https://lore.kernel.org/r/67ab679c15fbf49fa05b3ffe05d91c47ab84f147.1708426665.git.pabeni@redhat.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
include/linux/udp.h
net/ipv4/af_inet.c
net/ipv4/udp.c
net/ipv6/af_inet6.c