Repository navigation
Description
OVS br-ex drop rule silently discards OVN reply traffic for LoadBalancer connections
Description
On a single-node MicroShift 4.21.0 (OKD SCOS) cluster using OVN-Kubernetes, reply traffic from OVN for externally-originated LoadBalancer connections is dropped by a low-priority OVS flow rule on br-ex. This is a second, independent bug that prevents external LoadBalancer access even after Bug 1 (ETP DNAT to unreachable 169.254.169.3) is bypassed.
Traffic that follows the OVN-KUBE-EXTERNALIP DNAT path to the ClusterIP enters OVN successfully and reaches the ingress router pod, but reply packets are dropped by priority=9, in_port=2, actions=drop on br-ex table 0.
Environment
| Component | Version / Detail |
|---|---|
| MicroShift | 4.21.0_g29f429c21_4.21.0_okd_scos.ec.15 |
| OS | RHEL 9.7 (kernel 5.14) |
| OVS | 3.5.2-51.el9fdp |
| CNI | OVN-Kubernetes (cniPlugin: "ovnk") |
| Node | Single-node, 4 CPU, 1 NIC (enp0s25) |
| Node IP | 192.168.137.31 |
Affected Service
Service: router-default (openshift-ingress)
Type: LoadBalancer
ClusterIP: 10.43.52.69
ExternalTrafficPolicy: Local
NodePorts: 30703 (HTTP), 32298 (HTTPS)
Steps to Reproduce
- Install MicroShift 4.21.0 (OKD SCOS) on a single-node RHEL 9 system with OVN-K CNI
- Bypass Bug 1 by inserting RETURN rules in the OVN-KUBE-ETP chain:
iptables -t nat -I OVN-KUBE-ETP 1 -d <node-ip> -p tcp --dport 443 -j RETURN iptables -t nat -I OVN-KUBE-ETP 1 -d <node-ip> -p tcp --dport 80 -j RETURN
- From an external machine, attempt to access a route:
curl -kv https://<route-hostname> - Observe: connection still times out
Expected Behavior
Reply traffic from OVN (entering br-ex via the OVN patch port, in_port=2) for externally-originated connections should be properly conntracked and forwarded back to the external client.
Actual Behavior
Reply packets are caught by the catch-all drop rule and never reach the external client.
Traffic Path
External client → 192.168.137.31:443
→ iptables PREROUTING: OVN-KUBE-ETP (bypassed via RETURN)
→ iptables PREROUTING: OVN-KUBE-EXTERNALIP DNAT to 10.43.52.69:443
→ iptables FORWARD: accepted
→ br-ex → OVN patch port (port 2) → OVN cluster network → ingress pod
→ Reply from pod enters br-ex via port 2
→ OVS table 0: priority=9, in_port=2, actions=drop ← DROPPED HERE
Evidence
Kernel logs confirm br-ex bridge loop pattern
IN=br-ex OUT=br-ex SRC=<client> DST=10.43.52.69 ... SYN
(retransmitted 3x at 1-second intervals, no response)
Conntrack shows SYN_SENT with no reply
tcp SYN_SENT src=169.254.169.2 dst=10.43.52.69 sport=42286 dport=443 [UNREPLIED]
zone=9 mark=10
tcp SYN_SENT src=169.254.169.2 dst=10.43.52.69 sport=42286 dport=443 [UNREPLIED]
zone=64001 mark=0
OVS drop rule on br-ex table 0
$ ovs-ofctl dump-flows br-ex table=0
...
priority=9, in_port=2, actions=drop ← catches all OVN reply traffic
...
OVS NAT tables show zero packet matches
table=4: ct(commit,table=3,zone=64002,nat(src=169.254.169.1)) n_packets=0
table=5: ct(commit,table=2,zone=64001,nat) n_packets=0
The NAT tables (4-5) that should handle return traffic conntrack and NAT processing have zero packet hits, confirming traffic is dropped before reaching them.
Proof that fixing the reply path changes behavior
Adding a high-priority OVS flow for the reply path changed behavior from timeout (exit 28) to connection refused (exit 7):
$ ovs-ofctl add-flow br-ex "table=0,priority=200,in_port=2,ip,nw_dst=169.254.169.2,actions=ct(table=3,zone=64001,nat)"
$ curl -k https://fuseki-packagegraph.apps.kafka.tel/
# exit code 7 (connection refused) instead of 28 (timeout)
However, OVN reconciliation removes custom OVS flows within seconds, making this impractical.
Analysis
The priority=9, in_port=2, actions=drop rule on br-ex table 0 acts as a catch-all for traffic arriving from the OVN patch port. Higher-priority rules in table 0 are expected to match legitimate return traffic and route it through the conntrack/NAT pipeline (tables 3-5). For externally-originated LoadBalancer traffic that was DNAT'd via iptables (the OVN-KUBE-EXTERNALIP path), no matching higher-priority rule exists — the return traffic falls through to the drop rule.
Additionally, net.ipv4.conf.br-ex.rp_filter = 1 (strict reverse path filtering) may contribute to packet drops for reply traffic where the source IP is a ClusterIP (10.43.52.69) that is not routable via br-ex. However, disabling rp_filter alone does not fix the issue because the OVS drop rule catches traffic first.
Additional Context
OVN load balancer configuration is correct
VIP 10.43.52.69:80 → 10.42.0.73:80 (ingress pod)
VIP 10.43.52.69:443 → 10.42.0.73:443 (ingress pod)
VIP 192.168.137.31:80 → 10.42.0.73:80
VIP 192.168.137.31:443 → 10.42.0.73:443
Load balancer rules exist on both the cluster router and the external gateway router. The control plane is correct — the bug is in the dataplane flow programming.
Direct pod access works
$ curl -k https://10.42.0.73 -H 'Host: fuseki-packagegraph.apps.kafka.tel'
# Returns expected response
Firewall is not the issue
Ports 80, 443, and NodePort range 30000-32767 are open. FORWARD policy is ACCEPT. Problem persists with all firewall rules bypassed.
Diagnostic Commands
# Verify OVS drop rule
ovs-ofctl dump-flows br-ex table=0 | grep "in_port=2.*drop"
# Verify conntrack shows unreplied connections
conntrack -L -p tcp --dport 443 2>/dev/null | grep UNREPLIED
# Check NAT table packet counts
ovs-ofctl dump-flows br-ex table=4
ovs-ofctl dump-flows br-ex table=5Workaround
A host-level TCP proxy bypasses the entire OVN LoadBalancer dataplane:
# /usr/local/bin/ingress-proxy.py — forwards external ports to ClusterIP
# Systemd services: ingress-proxy-http.service, ingress-proxy-https.service
ExecStart=/usr/bin/python3 /usr/local/bin/ingress-proxy.py 443 10.43.52.69 443
ExecStart=/usr/bin/python3 /usr/local/bin/ingress-proxy.py 80 10.43.52.69 80Combined with iptables ACCEPT rules in PREROUTING to prevent DNAT to the broken paths. Note: iptables rules may be reconciled by OVN-K on restart.
Prior Art
- Red Hat Solution 6999225 — "Connections to NodePort and LoadBalancer services with externalTrafficPolicy: Local fail and timeout on OVNKubernetes" (may describe a related root cause; resolution behind paywall)
Related
- Upstream: OVS br-ex drop rule silently discards OVN reply traffic (OVS br-ex table 0 drop rule discards reply traffic for iptables-DNAT'd LoadBalancer connections ovn-kubernetes/ovn-kubernetes#6250)
- (Related - OVN-K) OVN-K issue filed at ovn-org/ovn-kubernetes (OVN-KUBE-ETP iptables chain DNATs LoadBalancer traffic to unreachable 169.254.169.3 on single-node ovn-kubernetes/ovn-kubernetes#6249)
- (Related - Microshift) OVN-KUBE-ETP iptables DNAT redirects LoadBalancer traffic to unreachable 169.254.169.3 #213
Activity
eslutsky commentedon Oct 6, 2026
Investigation Results — Fixed in 4.22.0
I reproduced this issue alongside #213. The bug is fixed in 4.22.0 (released 2026-10-05).
What we confirmed
The
priority=9, in_port=1, actions=droprule still exists inbr-extable 0:(Port numbering differs between bare-metal and containers —
in_port=2on bare metal,in_port=1in containers — but it's the same catch-all drop for unmatched OVN patch port traffic.)Why it no longer kills traffic
In 4.22.0, OVN-K programs higher-priority flows that correctly handle return traffic for the ETP:Local path before it reaches the drop rule. Throughout our end-to-end test, the drop rule's packet counter stayed at
n_packets=0.This is the direct consequence of fixing the issue described in #213: once traffic is correctly routed via the management port → ovnkube nodeport listener → ingress router, the return path goes through OVN's proper conntrack flows rather than falling through to the drop rule.
End-to-end test on 4.22.0
The fix is in the upstream OVN-K component. Please test with
4.22.0_202610051505_gfa0f8cee8_4.22.0_okd_scos.ec.16and confirm if the issue is resolved.