Linux Traffic Control:

U32 hashing filters

(document version 0.5 -preview)

Tomas Dulik, FAI TBU Zlin

Why hashing?

Imagine network with thousands of users

"Classic" approach with TC U32 filters:

1 computer IP = 1 "tc filter" rule OK for few tens of rules, slow for hundreds or thousands of rules! We can create new tc filter hashtables to optimize the rules matching. This way we can match (e.g.) 65535 IP addresses in only 2 or 3 steps. Using this technique, we can move the bottleneck from U32 filter to other parts of Linux kernel.

"Hashing" approach with TC U32 filters

"Classic" approach with U32

#!/bin/bash DEV=eth0 CEIL=240 #CEIL is maximum rate in mbits COUNT=20 #we want to share the $CEIL connectivity between COUNT users RATE=$(($CEIL*1000/$COUNT)) #RATE is rate in kbits guaranteed for each user tc qdisc del dev $DEV root tc qdisc add dev $DEV root handle 1: htb default 1 tc class add dev $DEV parent 1: classid 1:1 htb rate ${CEIL}mbit ceil ${CEIL}mbit for i in `seq 2 $COUNT`; do flowid=`printf '%x' $i` tc class add dev $DEV parent 1:1 classid 1:$flowid htb ${RATE}kbit tc filter add dev $DEV parent 1:0 protocol ip prio 1 u32 match ip src 192.168.0.$i flowid 1:$flowid done

1 2

3 4

Explanation of the example step 1

tc qdisc add dev $DEV root handle 1: htb default 0
Add new queing discipline for device $DEV Parent specification: here we use to the root = the network device $DEV We can use also class here: parent {classid} Optional handle specification: any value betwen 0 and FFFF Specified in hexadecimal! Default value is: 8001 The qdisc type=HTB Default class ID = 0 determines which class enqueues packets not matching any filters

The result of this step:

Enqueue

Dequeue

Qdisc HTB, handle 1: Network device $DEV

Explanation of the example step 2

tc class add dev $DEV parent 1: classid 1:1 htb rate ${CEIL}mbit ceil ${CEIL}mbit ` `
Add new class to the qdisc "1:" attached to device $DEV classid of the new class: {qdisc handle}:{class minor id} also called as {major}:{minor} Where {class id} is any value betwen 0 and FFFF Specified in hexadecimal! The class type = htb, followed by its parameters

pfifo is default qdisc for HTB classes. It is invisible for tc commands. Class (its qdisc) does not enqueue packets until some filter or qdisc parameter default instructs to do so

The result of this step:

Enqueue Qdisc pfifo Class classid 1:1

Dequeue

Qdisc HTB, handle 1: Network device $DEV

Explanation of the example step 3

for i in `seq 2 $COUNT`; do flowid=`printf '%x' $i` tc class add dev $DEV parent 1:1 classid 1:$flowid htb ${RATE}kbit
Add other new class(es) as childs of the class "1:1" created on previous slide Class (its qdisc) does not enqueue packets until some filter or qdisc parameter default instructs to do so

The result of this step:

Dataflow view:
Enqueue Qdiscpfifo pfifo Qdiscpfifo Qdiscpfifo Classclassid 1:$COUNT classid ClassQdisc 1:$COUNT Class classid 1:$COUNT Class classid 1:1 Dequeue

Hierarchical view of the classes:

Qdisc HTB, handle 1: Network device $DEV

class 1:2

class 1:1

class class 1:3 1:4

...

class 1:$COUNT

Explanation of the example step 4

for i in `seq 2 $COUNT`; do flowid=`printf '%x' $i` tc class add dev $DEV parent 1:1 classid 1:$flowid htb ${RATE}kbit tc filter add dev $DEV parent 1:0 protocol ip prio 1 u32 match ip src 192.168.0.$i flowid 1:$flowid
Add classification filter(s) to the qdisc 1:(0) In HTB, filters must be attached to the root qdisc! The filter will be added into the priority 1 filter chain Use the U32 (32 bits) selector with single IP match (mask=0xFFFFFFFF) In case packet matches, send it to class 1:$flowid

The result of this step:

Enqueue Filter 2 Filter 3

...

Dequeue Qdisc pfifo Qdisc pfifo Qdisc 1:1 Class classidpfifo Qdisc 1:1 Class classidpfifo Class classid 1:1 Class classid 1:$COUNT

Filter $COUNT

Qdisc HTB, handle 1: Network device $DEV

What is the class hierarchy for ?

Enqueue operation:

In general, within classful qdisc, the packet enqueue point is searched by traversing the class hierarchy from the qdisc root and consulting filters attached to each class. The CBQ qdisc is an example of this approach. However: within HTB qdisc, the class hierarchy is not used during the enqueue operation !!! The reason is that HTB does not allow filters to be attached to the classes the filters can be attached only to the qdisc root (=the HTB qdisc) In shallow class trees (e.g., depth=2), attaching filters to individual classes brings almost no performance improvement the only approach for good performance is using the U32 filter hash tables attached to the root qdisc. Therefore, we do not have to think about the class hierarchy function during the enqueue operation

Dequeue operation

The class hierarchy plays its main role in the dequeue() operation:

The dequeue() operation is called by kernel on the root qdisc, which calls dequeue() on all its children classes, each child then calls dequeue on all its children until all leafs are reached. the classes share the bandwidth (in HTB, bandwidth is represented by tokens) in a way that e.g. they can borrow unused bandwidth (tokens) from their siblings (=classes with the same parent in the class hierarchy)

This is not used here in our example, we have default=0 (1:0 is non-existent class)

The hierarchical view of step 4 result

Example of Enqueue for HTB default=(1:)1

class 1:1
De qu eu e

Dequeue

Enqueue

Filter 2 Filter 3 Filter 4

Enqueue

class 1:2

class 1:3

class 1:4

...

class
1:$COUNT Dequeue

...

Filter $COUNT

Qdisc HTB, handle 1: Network device $DEV

Note: the filters can instruct packets to enqueue into any class, not just into leaf classes as shown in this simple example!

Intel Celeron, 2.5 GHz, i865, 2x INTEL PRO/1000 GT (PCI 32bit, 33/66MHz) Linux 2.6.17
Througput vs. number of rules: throughput drops down from 400 rules
Zvislost routovacho vkonu na potu QoS pravidel (OS:Debian, HW: Router A)
110

Classic approach: performace benchmark 1

CPU load during max. throughput vs. number of rules: 600 rules=100% CPU load
Zvislost vyten CPU na potu QoS pravidel (OS:Debian, HW: Router A)

400 390 Routovac vkon (Mbps) 380 370 360 350 340 330 320 310 300 0 200 400 600 800 1000 Poet QoS pravidel (Testovan parametr: Destination IP) Vyten CPU (%)

100 90 80 70 60 50 40 30 0 200 400 600 800 Poet QoS pravidel (Testovan parametr: Destination IP)

Area of good performance thanks to CPU load<60%

(Graphs taken from [3])

Classic approach: performace benchmark 2

Intel Celeron, 2.5 GHz, i865, 2x INTEL PRO/1000 GT (PCI 32bit, 33/66MHz) Mikrotik 2.9.27 (Linux 2.4??) Througput vs. number of rules: CPU load vs. number of rules:
Zvislost routovacho vkonu na potu QoS pravidel (OS:Mikrotik, HW: Router A)
110

Zvislost vyten CPU na potu QoS pravidel (OS:Mikrotik, HW: Router A)

400 380 Routovac vkon (Mbps) 360 340 320 300 280 260 240 220 200 0 100 200 300 400 500 600 700 800 Poet QoS pravidel (Testovan parametr: Destination IP) 50 0 200 400 600 800 Poet QoS pravidel (Testovan parametr: Destination IP) Vyten CPU (%) 90 80 70 60 100

(Graphs taken from [3])

Classic approach: performace benchmark 3

Routerboard RB532 Mikrotik 2.9.27 Router througput vs. number of rules: (notice the strange peak...)
Zvislost routovacho vkonu na potu QoS pravidel (OS:Mikrotik, HW: Router C)
30

Routovac vkon (Mbps)

25

20

15

10

5 0 50 100 150 200 Poet QoS pravidel (Testovan parametr: Destination IP)

(Graph taken from [3])

Classic approach internals

Each filter priority creates an individual chain of rules. The chains are matched sequentially (highest priority chain first, lowest priority chain as the last one)

U32 implementation details

U32 is one of the few filters available in tc (the others are fw, rsvp, route and tcindex). U32 is the most advanced one and has very few documentation available, so it is worthy to dig deep inside Key handle range is 0 - 0xF FFFF #define TC_U32_KEY(h) ((h)&0xFFFFF) Hash table ID range is 0x000 00000 0xFFF 00000, from user point of view 0-0xFFF (=we can create 4095 hash tables) #define TC_U32_HTID(h) ((h)&0xFFF00000) #define TC_U32_USERHTID(h) (TC_U32_HTID(h)>>20)

Structures in net/sched/cls_u32.c
struct tcf_proto
struct tc_u_hnode *root

tc_u_knode - the key node:

struct tc_u_knode *next u32 handle; struct tc_u_hnode *ht_up; struct tcf_exts exts; u8 fshift; struct tcf_result res; struct tc_u_hnode *ht_down; struct tc_u32_sel sel;

struct tc_u_hnode
struct tc_u_hnode *next; u32 handle, prio; struct tc_u_common *tp_c; int refcnt; unsigned divisor; struct tc_u_knode *ht[1];

u32 implementation details

References:

1.http://lion.cs.uiuc.edu/courses/cs498hou_fall05/lectures/le 3.ek P., Diplomov prce 4.http://luxik.cdi.cz/~devik/qos/u32.htm 5.Linux kernel source

1. include/net/sch_generic.h 2. /include/linux/pkt_cls.h 3. /net/sched/cls_u32.c

2.http://dcn.ssu.ac.kr/~softgear/prog_sw_report_summer_9

6.http://feela.network.cz/ltc.html

tc filter classic approach

The tc filter uses default hash table with ID=800, which has 256 buckets A hash function is used to map a packet on to an index (0-255) into the hash table:
Packet hashing_function() Default tc filter hash table Bucket 0 Bucket 1 Bucket 2 Bucket 253 Bucket 254 Bucket 255

include/linux/pkt_cls.h:#define TC_U32_HASH(h) (((h)>>12)&0xFF)

What to do if we need hundreds or thousands of tc filter rules?

Unfortunatelly, we can't make the hash table bigger. But we can create new hashtables !!!

References

http://lartc.org/lartc.html#LARTC.ADV-FILTER.HASHING http://brownian.org.ua/?page_id=6&langswitch_lang=en http://vcalinus.gemenii.ro/?p=9 http://www.czfree.net/forum/showthread.php?postid=210938