Jool 4.1.7/Ubuntu 22.10 (kernel 5.19): dst_output() returned errcode 1 #400
Description
I am running a Jool.mx Netfilter instance in a separate namespace. This configuration was working for several years, but suddenly it is unable to translate packets. I have tested self-built DKMS module as well as the prebuilt 4.1.7 Debian packages, still no luck.
The error I am seeing in dmesg when enabling jool global update logging-debug true is:
[Wed Apr 19 19:21:16 2023] Jool NAT64/95be8000/default: dst_output() returned errcode 1.
The full packet dump from dmesg is:
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: ===============================================
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Packet: <lan64-prefix>:29cc:cf1c:3dd:dfeb->fd00:64::1139:9237
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: TCP 57494->443
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Step 1: Determining the Incoming Tuple
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Tuple: <lan64-prefix>:29cc:cf1c:3dd:dfeb#57494 -> fd00:64::1139:9237#443 (TCP)
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Done step 1.
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Step 2: Filtering and Updating
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: BIB entry: <lan64-prefix>:29cc:cf1c:3dd:dfeb#57494 - 192.0.2.1#63425 (TCP)
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Session entry: <lan64-prefix>:29cc:cf1c:3dd:dfeb#57494 - fd00:64::1139:9237#443 | 192.0.2.1#63425 - 17.57.146.55#443 (TCP)
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Done: Step 2.
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Step 3: Computing the Outgoing Tuple
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Tuple: 192.0.2.1#63425 -> 17.57.146.55#443 (TCP)
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Done step 3.
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Step 4: Translating the Packet
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Translating packet addresses <lan64-prefix>:29cc:cf1c:3dd:dfeb->fd00:64::1139:9237...
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Result: 192.0.2.1->17.57.146.55
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Packet routed via device 'from_jool'.
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Done step 4.
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: Sending packet.
[Wed Apr 19 19:28:53 2023] Jool NAT64/95be8000/default: dst_output() returned errcode 1.
[Wed Apr 19 19:28:53 2023] Jool: Dropping packet.
The setup:
[IPv6, internal NAT64-enabled LAN] -> [IPv6, router main namespace, LAN port] -> [IPv6, namespace joolns] -> [jool] -> [IPv4, namespace joolns] -> [IPv4, router main namespace, NAT to the egress interface] -> [the Internet]
I see the IPv6 packets incoming in the joolns namespace, but the IPv4 packets never appear leaving the namespace (the in/out interface is just one).
The init script:
modprobe jool
# add devices
ip netns add joolns
ip link add name to_jool type veth peer name from_jool
ip link set dev from_jool netns joolns
ip link set to_jool up
ip -6 addr flush dev to_jool scope link
ip -6 addr add fe80::1/64 dev to_jool scope link
ip addr add 192.0.2.8/24 dev to_jool
ip link set to_jool mtu 1500
ip netns exec joolns bash -c "
ip link set from_jool mtu 1500
ip link set from_jool up
tc qdisc add dev from_jool root fq
ip -6 addr flush dev from_jool scope link
ip addr add fe80::2/64 dev from_jool scope link
ip addr add 192.0.2.1/24 dev from_jool
ip route add default via 192.0.2.8
ip -6 route add default via fe80::1 dev from_jool
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
jool instance add --netfilter --pool6 fd00:64::/96
jool pool4 add --tcp 192.0.2.1 61001-65535
jool pool4 add --udp 192.0.2.1 61001-65535
jool pool4 add --icmp 192.0.2.1 61001-65535
"
ip route add fd00:64::/96 via fe80::2 dev to_jool
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
exit 0
(Outgoing NAT on the router is handled outside of this script.)
Interface config within the namespace:
root@router:~/jool-4.1.7# ip a s from_jool
27: from_jool@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
link/ether 9a:69:04:28:0f:6a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.0.2.1/24 scope global from_jool
valid_lft forever preferred_lft forever
inet6 fe80::2/64 scope link
valid_lft forever preferred_lft forever
Jool status:
root@router:~# jool instance status
Running
root@router:~# jool instance display
+--------------------+-----------------+-----------+
| Namespace | Name | Framework |
+--------------------+-----------------+-----------+
| 95be8000 | default | netfilter |
+--------------------+-----------------+-----------+
I have tried setting rp_filter to 0, but I still have no luck and packets get dropped.
echo 0 > /proc/sys/net/ipv4/conf/from_jool/rp_filter
uname -a:
root@router:~/jool-4.1.7# uname -a
Linux router.local 5.19.0-38-generic #39-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 17 17:33:16 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Is there any known issue with these newer kernels? I have checked and there is no newer version of Jool available on Jool.mx.
Thank you.