NetFlow to guard the infrastructure

Yann Berthier

NANOG39 - Feb 4-7, 07

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 1 / 126
Outline I
1 An introductory view of NetFlow
2 The flow cache
Overview
Populating the cache
Keeping the cache under control
Querying the cache in real time
Exporting the content of the cache
3 Collect & analysis
Overview
Storage
Tools
nfdump / NfSen
direction & biflows
From PCAP to NetFlow
NetFlow and BGP
Other sources
Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 2 / 126
Outline II
Reductio ab aggregatio

4 What’s next
Perhaps not really next
Resources

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 3 / 126
 An introductory view of NetFlow

Outline

1 An introductory view of NetFlow

2 The flow cache

3 Collect & analysis

4 What’s next

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 4 / 126
 An introductory view of NetFlow

What is NetFlow ?

A technology to gather informations on forwarded packets

In one or several caches
Whose content can be queried real time for troubleshouting
And exported to collectors
A protocol to export those records to collectors
To use sophisticated tools for analysis
Near real-time analysis
For long-term analysis, trending, whatever
To store and retrieve if needed (forensics) / when asked
Comes from one vendor, but industry + IETF standard (not quite,
but ...)
Other concurrent technologies: CRANE, Diameter, LFAP, IPDR,
sFlow, ...

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 5 / 126
 An introductory view of NetFlow

What is a flow ?

A loose definition
A set of packets having common characteristics

Definition
A flow is a unidirectional set of packets that arrive at the router on the
same subinterface, have the same source and destination IP addresses,
Layer 4 protocol, TCP/UDP source and destination ports, and the same
ToS (type of service) byte in the IP headers

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 6 / 126
 An introductory view of NetFlow

Anatomy of a flow (v5)

7-tupple key fields

saddr, daddr, sport, dport, L3 proto, ToS, input ifIndex
Additional fields
Byte count, packet count, start time, end time, output ifIndex, TCP
flags, next hop, src AS, dst AS

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 7 / 126
 An introductory view of NetFlow

Some characteristics

L3-L4
Flows are unidirectionals (ingress / egress)
Flow-cache comes before ACL lookup
Comes at a cost (memory, CPU) for the router
Not the perfect solution - but not a lot of other candidates either

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 8 / 126
 An introductory view of NetFlow

Rationale

Old techno (96)

A lot happening in the protocol and implementation front
A lot of interest and projects evolving around NetFlow
Questions keep being raised (How do i monitor ... What tool do you
use for capture ... I want to have an overview of the traffic ...) where
NetFlow is the answer - or at least part of it
Infrastructure is a potential target
What’s needed is actual insight

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 9 / 126
 An introductory view of NetFlow

NetFlow flavors

Several versions of the export protocol

Metering and export keep being worked on
Depends on vendor, hardware, software, and combinations of those
Even when and where supported, particularities to keep in mind

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 10 / 126
 An introductory view of NetFlow

Versions

v5 - De facto standard (supported by non-C vendors implementing

NetFlow)
v9 - template-based
Flexibility (for now means that the user has to be very flexible) -
complexity
May have some compelling features - SCTP, IPv6, some L2 fields,
additionnal L3 (ttl, ipid), BGP next-hop, ...
But probably not processed by your collector of choice anyway
(Fast) moving target
Some features are backported to v5 (e.g. transport independency)
v10 aka IPFIX - IETF-blessed standard (in its way to, at least)
Based on NetFlow v9 (”forked” from)
Other - v1 (obsolete), v7 (switches), v8 (router-based aggregation)

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 11 / 126
 An introductory view of NetFlow

Hardware peculiarities

Support is not equal on all devices

No support on the switches below the cat 45xx
Peculiarities where supported
On L3 switches, no TCP Flags, many specificities depending upon type
of sup engine, PFC, config, software, ...

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 12 / 126
 An introductory view of NetFlow

Hardware peculiarities - on routers too

Big CAUTION on every Cisco doco concerning NetFlow on the 12k

Entering this command on a Cisco 12000 Series Internet Router causes
packet forwarding to stop for a few seconds while NetFlow reloads the
route processor and line card CEF tables. To avoid interruption of service
to a live network, apply this command during a change window, or include
it in the startup-config file to be executed during a router reboot

Less advertised caution

Despite ASIC support in Engine 2, 3 and 4+ Linecards ’Full NetFlow’ still
inflicts a heavy burden on memory and therefore sampled netFlow is
preferred

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 13 / 126
 The flow cache

Outline

1 An introductory view of NetFlow

2 The flow cache

Overview
Populating the cache
Keeping the cache under control
Querying the cache in real time
Exporting the content of the cache

3 Collect & analysis

4 What’s next

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 14 / 126
 The flow cache Overview

Principle

A global memory cache, enabled for each interface

number of entries - where it is kept
For each packet entering an interface
Either an entry matching the key fields (saddr, daddr, sport, dport, L3
proto, ToS, input ifIndex) exists
Byte and packet counts are summed, end time is updated, TCP flags
are ORed
Or a new entry is created
Expiration mechanism
Export mechanism if so configured

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 15 / 126
 The flow cache Overview

Expiration mechanisms

When one of the following conditions is met

Upon reception of a TCP RST or FIN packet
After 15 seconds (default) for inactive flows
After 30 minutes (default) for an active flow
When the cache is full

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 16 / 126
 The flow cache Populating the cache

Cisco, v5

Example
router(config)#ip flow-export version 5 [origin-as|peer-as]
router(config-if)#ip flow {ingress | egress}

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 18 / 126
 The flow cache Populating the cache

v9 specifics

Example
#ip flow-export version 9 [origin-as|peer-as] bgp-nexthop
#ip flow-capture ip-id
#ip flow-capture mac-addresses
#ip flow-capture packet-length
#ip flow-capture ttl
#ip flow-capture vlan-id

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 20 / 126
 The flow cache Populating the cache

NDE - NetFlow Data Export

NDE on the MSFC exports statistics for flows routed in software

NDE on the PFC exports statistics for flows routed in hardware
Flow masks on the PFC - from source-only to full-interface

Example
Switch(config)#mls netflow
Switch(config)#mls flow ip full
! To report L2 bridged traffic - not all PFCs
Switch(config)#ip flow export layer2-switched
Switch(config)#mls nde sender version 5
Switch(config-if)#ip flow {ingress | egress}
Switch(config)#ip flow-export destination 192.0.2.1 2055

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 22 / 126
 The flow cache Populating the cache

cflowd on Juniper

Relies on Service Cards / Adaptive Service Interfaces

Configuration of a sampling rule in the firewall statement
Rate of said sampling is configured in the forwarding-options
Can be 1
Input filter calling the firewall rule as defined above for each interface

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 24 / 126
 The flow cache Populating the cache

cflowd on Juniper
Example
interfaces {
sp-0/0/0 unit 0 family inet;
fe-0/0/0 unit 0 family inet {
filter input catch_all;
address 10.88.17.126/28; }}
firewall {
family inet filter catch_all term default then {
sample; accept; }}
forwarding-options {
sampling {
input family inet {
rate 1; max-packets-per-second 5000; }
output { }}

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 26 / 126
 The flow cache Populating the cache

FreeBSD

In-kernel ng netflow(4) Netgraph node coupled with the ng ipfw(4) ipfw

Netgraph hook
Example
# ipfw add ngtee 10 ip from any to any
# ngctl mkpeer ipfw: netflow 10 iface0
# ngctl name ipfw:10 catchall
# ngctl msg catchall: setdlt { iface=0 dlt=12 }
# ngctl msg countall: settimeouts { inactive=3 active=300 }

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 28 / 126
 The flow cache Populating the cache

FreeBSD cont’d.

Makes it easy to define flow export policies

Example
# ipfw add ngtee 5 tcp from any to me 22 in
# ngctl mkpeer ipfw: netflow 5 iface0
# ngctl name ipfw:5 ssh_in
# ngctl msg ssh_in: setdlt { iface=0 dlt=12 }
# ngctl msg ssh_in: settimeouts { inactive=3 active=300 }

OOooohhHH
Bad news is - AS fields are not populated - AAaahh...

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 30 / 126
 The flow cache Keeping the cache under control

Tuning its size (32 to 256k depending on hw)

Aging of entries
Aggregation router side
Sampling
Input filtering
Flow masks on Cat6.5k

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 31 / 126
 The flow cache Keeping the cache under control

Some performance considerations

64 bytes / entry in a cache

Aggregation is more costly CPU-wise
Done in hw for some platforms (Cat6.5k) - only export consumes
CPU cycles
Not fitted for more-than-reasonably busy routers
Plan / test / monitor / prepare to rollback

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 32 / 126
 The flow cache Keeping the cache under control

Sampling or not

Depending on use / sf / hw
Flow vs packet vs time - deterministic vs random
Choice is not yours - check with your rep
Effective (random sampled netflow) and scalable
Granularity loss may not be an acceptable tradeoff for security
Performance loss may not be an acceptable tradeoff for operations :D

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 33 / 126
 The flow cache Querying the cache in real time

Cisco

#show ip cache flow

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Vl90 10.11.10.50 Local 192.168.11.2 06 090D 0016 6
Vl97 192.168.19.214 Null 192.168.19.255 11 008A 008A 1
Vl16 192.168.17.165 Null 255.255.255.255 11 4001 006F 1
Vl10 192.168.15.65 Null 192.168.14.1 11 007B 007B 1
Vl10 192.168.15.166 Null 192.168.15.255 11 008A 008A 1
Vl10 192.168.15.84 Null 192.168.15.255 11 008A 008A 1
Vl10 192.168.15.75 Null 192.168.15.255 11 008A 008A 1

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 35 / 126
 The flow cache Querying the cache in real time

The top-talkers cache

Possibility to create a dedicated cache for top-talkers

Matching criteria, sorting order and number of entries are fixed at
configuration
Not the same as the Dynamic Top Talkers feature on the main cache

Example
router(config)#ip flow-top-talkers
router(config-flow-top-talkers)#top 30
router(config-flow-top-talkers)#sort-by bytes

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 37 / 126
 The flow cache Querying the cache in real time

The top-talkers cache, cont’d.

#show ip flow top-talkers

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Bytes
Se1/6:0 10.18.16.77 Fa0/1 10.11.19.88 06 05FD 0455 34K
Se1/6:0 10.18.16.77 Fa0/1 10.11.19.65 06 05FD 045D 10K
Se1/6:0 10.18.16.77 Fa0/1 10.11.14.51 06 05FD 04BA 9508
Se1/6:0 10.18.16.77 Fa0/1 192.168.9.7 06 05FD 059E 7064
Fa0/1 10.11.19.88 Se1/6:0 10.18.16.77 06 0455 05FD 5380

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 39 / 126
 The flow cache Querying the cache in real time

The JunOS way

#show services accounting flow-detail | match bgp | trim 26

AAA.88.137.113 2146 Unknown BBB.88.137.126 bgp(179) 628 135136
00:30:00 585 95087

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 41 / 126
 The flow cache Querying the cache in real time

FreeBSD

#flowctl catchall show

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
em0 10.19.11.2 lo0 10.18.5.89 1 0000 0000 51
(null) 10.18.5.89 em0 10.19.11.2 1 0000 0000 51
em0 10.21.5.19 lo0 10.18.5.89 1 0000 0000 8
(null) 10.18.5.89 em0 10.64.60.7 6 1466 9f8a 1

And the usual Unix tools | sort +n | head -n 10

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 43 / 126
 The flow cache Exporting the content of the cache

Export mechanism

When expired, flows are packed together and sent to one or more
collectors
UDP based
A header (24 bytes)
Version, number of PDUs, sequence number, ...
1-30 flow records (48 bytes each)
1464-bytes packets
Loss detection
SEQ number
No provision for retransmission

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 44 / 126
 The flow cache Exporting the content of the cache

On Cisco

Example
router(config)#ip flow-export destination 192.0.2.1 2055
router(config)#ip flow-export source loopback0

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 46 / 126
 The flow cache Exporting the content of the cache

Exporting the content of the cache, Juniper

Example
forwarding-options {
sampling {
output {
cflowd 192.168.25.1 {
port 2055; version 5; autonomous-system-type origin; }
interface sp-0/0/0 { source-address 192.168.25.2; };

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 48 / 126
 The flow cache Exporting the content of the cache

On FreeBSD

Example
# ngctl mkpeer catchall: ksocket export inet/dgram/udp
# ngctl msg catchall:export connect inet/192.0.2.1:2055

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 50 / 126
 The flow cache Exporting the content of the cache

Others

Soft meters / exporters

PCAP to NetFlow: fprobe, softflowd, nProbe, ...
Other sources: fw logs (PF, netfilter)

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 51 / 126
 The flow cache Exporting the content of the cache

Improving the reliability of export

Multiple (2) destinations where supported

Multicast-aware collectors
Export destination can be a multicast address
Collectors join the multicast group
Out-of-the-box redundancy, scalability, easy testing of other collectors,
collector diversity, ...
Consider using other transport protocols - SCTP with backup ?
The way of the future as they say, IPFIX implementations MUST
support SCTP
Though for now the list of collectors supporting SCTP is ... small
(read: I don’t know one, except NTOP being not a collector either)

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 52 / 126
 The flow cache Exporting the content of the cache

Security considerations

Unidirectional from router to collector

UDP
No crypto checksum
No authentication of the exporter
48 bytes-long records for min 28-bytes UDP packets
You may want to protect the link between exporter and collector from a
non invited participant
ACLs, uRPF, TCP-Wrappers or alike

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 53 / 126
 Collect & analysis

Outline
1 An introductory view of NetFlow

2 The flow cache

3 Collect & analysis

Overview
Storage
Tools
nfdump / NfSen
direction & biflows
From PCAP to NetFlow
NetFlow and BGP
Other sources
Reductio ab aggregatio

4 What’s next()
Yann Berthier NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 54 / 126
 Collect & analysis Overview

Glories and duties of a collector

Takes flows as input

May report loss
Pre-processes
Deduplicates, long flows, filters, various aggregation schemes
Stores
Post-processes
Deduplicates, long flows, filters, various aggregation schemes
Retrieval
Sorts, filters, aggregates,
Graphs

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 55 / 126
 Collect & analysis Overview

Glories and duties of a collector

Per record granularity

Aggregation facilities and higher level views (while retaining
granularity)
Versatility
Powerful CLI
Graphs
Can be splitted over different tools

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 56 / 126
 Collect & analysis Overview

Glories and duties of the analyst

Look at graphs
Look at TopNs
Build filters to look at subsets of traffic
Control & management plane
Traffic targetting adresses in infrastructure subnets
Outgoing traffic from said addresses

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 57 / 126
 Collect & analysis Overview

More considerations

IP-centric
Changing with v9: some L2, MPLS, ...
Not any-over-any friendly
TCP flags are blended together
’Isolated’ flags are still worth looking at:
nfdump -R . -o long ’flags S and not flags ARPFU’
nfdump -R . -o long ’flags R and not flags SPFU’
L7 streams are splitted upon 2+ records
Source and destination does not match sockets’ notion of client/server

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 58 / 126
 Collect & analysis Storage

Lost in collection

Some figures: 3 STM1s, 2 borders, 1 (/1) sampling: 7 GB/day

Mid term strategy: compression
Long term: prune data and keep heavily aggregated summaries
(topNs, ...)

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 59 / 126
 Collect & analysis Storage

Flat file vs database

Db solution sounds sexy but is costly - per-record overhead

B plan - keep granularity in ”efficient” storages, and send aggregated
and associated data in a db
Probably no integration with used tools
No normalized on-disk format

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 60 / 126
 Collect & analysis Tools

An overview of popular collectors

A lot in both the opensource and commercial world - varying a lot in scope
and qualities (scoop)
Arbor Peakflow SP, Lancope, Cisco MARS, Q1labs’ QRadar, many
more
The venerable flow-tools, the yet more venerable CAIDA’s cflowd,
argus, SiLK, Stager, nfdump/NfSen, many more
Others
aguri, glflow, panoptis, NTOP, stager, ...

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 61 / 126
 Collect & analysis Tools

Flow-tools / Flowscan

Second historical collection-and-analysis package

Many patches / associated scripts / ... / floating around
flow-tool to db, Flow Extract, ...
Big user community
Somewhat baroque (my opinion) CLI
GUI - flowscan / cuflow / jkflow
Not actively maintained (latest rel 0505) - no v9 support

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 62 / 126
 Collect & analysis Tools

nfdump/NfSen
Clean modular design
Powerful and efficient (easy) yet simple (way harder) CLI with
PCAP-like syntax
tightly coupled GUI with nice graphs
Takes input from NetFlow v5, v7, v9 (not all fields though) and sFlow
Actively maintained
Some plugins / patches floating around (porttracker, nfsplit,
Holt-Winter)
Community is growing steadily
Versatile - troubleshooting, perf, security, ... - real time, trending,
long-term analyses & forensics
Comes with a flow-tools 2 nfdump converter
All your flows are belong to us
And the winner is ...
Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 63 / 126
 Collect & analysis nfdump / NfSen

Architecture

nfcapd / sfcapd
NetFlow / sFlow capture daemon
No pre-post processing of the records collected
Reads data from the network and writes to disk
Fixed-time binary files (default 5 mn)
May demux to other collector
nfdump
Reads files and displays records after run-time processing

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 65 / 126
 Collect & analysis nfdump / NfSen

Architecture

Others - self-explicit
nfreplay, nfexpire, nfprofile, ft2nfdump
NfSen
Graphs based on rrdtools
Front-end to nfdump queries
Plugins

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 67 / 126
 Collect & analysis nfdump / NfSen

Installation

Straightforward
http://sourceforge.net/projects/{nfdump,nfsen}
Stable branch vs snapshots
En-route to NfSen 1.3 - major usability improvements
Whatever the version, keep nfdump and nfsen synchronized
Upgrade path provided between releases/snapshots
Nfdump dependencies: c compiler
Nfsen dependencies: perl, php, rrdtools, php-extensions (SESSION,
SOCKETS)
Bumping of max. number of SVIPC semaphores may be needed (one
per collector)

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 68 / 126
 Collect & analysis nfdump / NfSen

nfdump

Reads nfcapd files

Processes
Parses pcap-like filters (usuals + if, packets, bytes, pps, bps, bpp, as,
duration)
Matches long flows
Aggregates (proto, prefix/length, ASN, port)
TopNs (record, ip, port, tos, as, if, proto)
Orders by flow, packet, bytes, pps, bps, bpp
Anonymizes
Displays
Pre-defined formats / user-defined formats
Statistics (number of flows, per proto, packets, bytes, ...)

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 69 / 126
 Collect & analysis nfdump / NfSen

nfdump output
Pre-defined formats: -o "line|long|extended|pipe|raw|fmt"
-o "fmt:%ts %td %sap %dap"

-r nfcapd.200701202315 -c 3 -o ”fmt:%td %sap %dap”

Duration Src IP Addr:Port Dst IP Addr:Port
0.000 BBB.158.102.14:53 AAA.189.5.89:53
11.000 CCC.249.66.142:52812 AAA.189.5.89:80
1.000 DDD.218.231.90:4509 AAA.189.5.90:445

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 71 / 126
 Collect & analysis nfdump / NfSen

AS Matrix

nfdump -M <source_list> \
-R nfcapd.$start_tslot:nfcapd.$end_tslot \
-s record/bytes -A srcas,dstas -n 0 \
-o "fmt:%sas %das %byt"

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 73 / 126
 Collect & analysis nfdump / NfSen

If user-defined is not enough - nfdump.c

Define your output format string.
Test the format using standard syntax -o "fmt:<your format>"
Create a #define statement for your output format,
similar than the standard output formats above.
Add another line into the printmap[] struct below
BEFORE the last NULL line for you format:
{ "formatname", format special, FORMAT definition, NULL },
The first parameter is the name of your format as
recognized on the command line as -o <formatname>
The second parameter is always ’format special’ -
the printing function.
The third parameter is your format definition as defined
in #define.
The forth parameter is always NULL for user defined formats.
Recompile nfdump

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 75 / 126
 Collect & analysis nfdump / NfSen

$CONFDIR/nfsen.conf

Various paths ($BASEDIR, $BINDIR, $LIBEXECDIR, $CONFDIR,

$HTMLDIR, $VARDIR, $PROFILESTATDIR, $PROFILEDATADIR,
$BACKEND PLUGINDIR, $FRONTEND PLUGINDIR, $PREFIX
User of the {s,n}fcapd processes
Data layout (see below)
Disk filling % (warning message)
Disk filling water marks for profiles
Sources
$BINDIR/nfsen reconfig to add more

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 76 / 126
 Collect & analysis nfdump / NfSen

Main tab

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 77 / 126
 Collect & analysis nfdump / NfSen

Navigation tab

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 78 / 126
 Collect & analysis nfdump / NfSen

Data organization
Sources - aka exporters
nfsplit
External contrib
Stands between the actual flows and nfcapd
Splits per interface
Data lives in $BASEDIR/profiles/<profile>/<source>
# 0 default no hierachy levels - flat layout - compatible
# 1 %Y/%m/%d year/month/day
# 2 %Y/%m/%d/%H year/month/day/hour
# 3 %Y/%W/%u year/week_of_year/day_of_week
# 4 %Y/%W/%u/%H year/week_of_year/day_of_week/hour
# 5 %Y/%j year/day-of-year
# 6 %Y/%j/%H year/day-of-year/hour
# 7 %Y-%m-%d year-month-day
# 8 %Y-%m-%d/%H year-month-day/hour

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 80 / 126
 Collect & analysis nfdump / NfSen

Data organization, cont’d

Profiles - ’live’ post-install

Managed through web and CLI
Data is duplicated between profiles (should change)
Can be retroactive
Time-delimited or continuous
Expiration of old data - based on duration and/on size
Beware - thresholds are per profile, no global check
Beware - it is recommended to put the stats ($PROFILESTATDIR)
and the data ($PROFILEDATADIR) on different volumes to prevent
corruption in case of a disk full
Channels - defined by filters

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 81 / 126
 Collect & analysis nfdump / NfSen

Managing profiles

List all profiles

$BINDIR/nfsen -A

Add a profile
$BINDIR/nfsen -a <profile> -c desc -B <starts> -E <ends>

Modify a profile
$BINDIR/nfsen -m <profile>

Delete a profile
$BINDIR/nfsen -d <profile>

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 83 / 126
 Collect & analysis nfdump / NfSen

Managing profiles
/data/nfsen/bin/nfsen -a slammer -B 2006-10-12-23-45 -S
other in:other out ’proto udp and port 1434’
#
name slammer
tstart Thu Oct 12 23:45:00 2006
tend Wed Dec 6 02:55:00 2006
updated Thu Oct 12 23:40:00 2006
filter filter.txt
expire 0 hours
size 0
maxsize 0
sources other_in:other_out
type continuous
locked 1
status new

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 85 / 126
 Collect & analysis nfdump / NfSen

Managing profiles
/data/nfsen/bin/nfsen -l slammer
#
name slammer
tstart Thu Oct 12 23:45:00 2006
tend Wed Dec 6 02:55:00 2006
updated Thu Oct 12 23:40:00 2006
filter filter.txt
expire 0 hours
size 0
maxsize 0
sources other_in:other_out
type continuous
locked 1
status built 53.9%

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 87 / 126
 Collect & analysis nfdump / NfSen

Navigation tab

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 88 / 126
 Collect & analysis nfdump / NfSen

Channels

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 89 / 126
 Collect & analysis nfdump / NfSen

Detail tab

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 90 / 126
 Collect & analysis nfdump / NfSen

AS resolution

$BASEDIR/libexec/Lookup.pm
my $whois_socket = IO::Socket::INET->new(
PeerAddr => ’whois.cyberabuse.org’,
PeerPort => 43,
Proto => ’tcp’,
timeout => 10 );

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 92 / 126
 Collect & analysis nfdump / NfSen

Plugins

NfSen is ”plugin” friendly

Backend in perl and frontend in PHP
Skeleton available in $BASEDIR/plugins/demoplugin.pm and
$HTMLDIR/plugins/demoplugin.php
No repository of plugins
Only publicly available plugin is PortTracker
Tracks ... TopN ports

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 93 / 126
 Collect & analysis nfdump / NfSen

Holt-Winter patches

hw-patched rrdtools for NfSen

External contrib
Discussions pending regarding integration or not
Can be installed in parallel to an existing NfSen installation
Useful to spot traffic irregularities, especially among more verbose
sources

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 94 / 126
 Collect & analysis nfdump / NfSen

Holt-Winter - main view

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 95 / 126
 Collect & analysis nfdump / NfSen

Anomaly view

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 96 / 126
 Collect & analysis nfdump / NfSen

The example below is purely fictional

Example
+---------------------------------------+
| 1. The packet contains a specific |
| crafted IP option. |
|---------------------------------------|
| AND |
|---------------------------------------|
| 2. The packet is one of the following |
| protocols: |
|---------------------------------------|
| * ICMP - Echo (Type 8) - ’ping’ |
|---------------------------------------|
| * ICMP - Timestamp (Type 13) |
|---------------------------------------|
| * ICMP - Information Request (Type |
| 15) |
+---------------------------------------+
Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 98 / 126
 Collect & analysis nfdump / NfSen

Example
+---------------------------------------+
| * ICMP - Address Mask Request (Type |
| 17) |
|---------------------------------------|
| * PIMv2 - IP protocol 103 |
|---------------------------------------|
| * PGM - IP protocol 113 |
|---------------------------------------|
| * URD - TCP Port 465 |
|---------------------------------------|
| AND |
|---------------------------------------|
| 3. The packet is sent to a physical |
| or virtual IPv4 address configured on |
| the affected device. |
+---------------------------------------+

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 100 / 126
 Collect & analysis nfdump / NfSen

The ICMP case

Source port == 0
Code+type shoehorned into dst port - code is lower 8 bits, type
higher 8 bits

tstamp request & tstamp reply

ICMP 192.168.2.11:0 -> AAA.189.5.89:13.0
ICMP AAA.189.5.89:0 -> 192.168.2.11:14.0

No specific filters for ICMP though

Type 8, 13, 15, 17

proto icmp and (port 2048 or port 3328 or port 3840 or port 4352)

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 102 / 126
 Collect & analysis nfdump / NfSen

Adding a new profile and channel

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 103 / 126
 Collect & analysis direction & biflows

Analyst’s temptation

Looking at ’streams’ rather than flows

Example
stime, client:sport, sflags, sbytes, spkts -> server:dport,
dflags, dbytes, dpkts, ...

Makes it easier to track some irregularities / asymmetries

SYN / RST couples
Unusual proportions of s{bytes, pkts, ...} vs d{bytes, pkts, ...}
Easier classification of L7 when knowing connect() and bind() sides
Orthogonal problems

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 105 / 126
 Collect & analysis direction & biflows

Direction

Records convey no direction information

No 3-way handshake granularity with TCP flags
No sub-ms granularity for start times
Besides, many reasons to have timestamps in the ’wrong’ order
No good heuristics beyond
Parse time-sorted
Swap saddr and daddr if
sport < 1025 && dport > 1024
Still too many incorrectly headed flows

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 107 / 126
 Collect & analysis direction & biflows

Biflows

Basically 3 places to aggregate 2 flows into 1 biflow

Router-side
Collect - one shot - increases complexity
post-process - run-time
Some IPFIX drafts

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 108 / 126
 Collect & analysis From PCAP to NetFlow

Who’d want that ?

Everybody trying to make sense of a (big, for some meaning of)

packet capture
In need of efficient tools
Re-use of existing tools / methods

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 109 / 126
 Collect & analysis From PCAP to NetFlow

Argus

http://qosient.com/argus/
Collect and analysis of network data
PCAP (live or off-line)
NetFlow
Stores flow-like data
Bidirectionnal records

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 110 / 126
 Collect & analysis From PCAP to NetFlow

The other way around ? (from NetFlow to PCAP)

Nobody would want that

Lot of tools working on PCAP data where NetFlow records would be
fine
Specialized reports, DoS, ...
No normalized on-disk format - no incentive for tools’ authors
Disclaimer: very clumsy and inelegant
nfdump -> ASCII -> ipsumpdump -> PCAP
- OUCH
http://www.cs.ucla.edu/∼kohler/ipsumdump/

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 112 / 126
 Collect & analysis NetFlow and BGP

BGP

Heartbeat-like traffic
Whose level of steroids in blood increases over time, but this is another
story
Pinpointing of BGP events (which scale ?)
At least timeframe is provided - up to the ops to dig into BGP logs

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 113 / 126
 Collect & analysis NetFlow and BGP

BGP heartbeat

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 114 / 126
 Collect & analysis NetFlow and BGP

eBGP multihop instability

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 115 / 126
 Collect & analysis NetFlow and BGP

eBGP multihop instability

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 116 / 126
 Collect & analysis Other sources

Looking at the noise

Sink Holes
Non used address space
Diverted traffic

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 117 / 126
 Collect & analysis Other sources

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 118 / 126
 Collect & analysis Reductio ab aggregatio

Aguri

http://www.csl.sony.co.jp/ kjc/software.html#aguri
Automatic traffic aggregation - prefixes & ports
PCAP-based

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 119 / 126
 Collect & analysis Reductio ab aggregatio

Aguri

aguri
[src address] 1049390337 (100.00%)
AAA.64.60.71 1008893224 (96.14%)
BBB.189.5.89 36286986 (3.46%)
[dst address] 1049390337 (100.00%)
AAA.64.60.71 35493322 (3.38%)
BBB.189.5.89 1012495103 (96.48%)
[ip:proto:srcport] 1049390337 (100.00%)
4:6:22 1008772431 (96.13%)
4:6:62008 35303106 (3.36%)
[ip:proto:dstport] 1049390337 (100.00%)
4:6:22 35439904 (3.38%)
4:6:62008 1008498420 (96.10%)
Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 121 / 126
 What’s next

Outline

1 An introductory view of NetFlow

2 The flow cache

3 Collect & analysis

4 What’s next
Perhaps not really next
Resources

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 122 / 126
 What’s next

NetFlow v9 and v10

Disclaimer: I’m not affiliated with ...

Disclaimer: check with your reps
Templates ...
Other transports
User-defined caches and export policies
Other vendors should follow

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 123 / 126
 What’s next Perhaps not really next

Dissemination of (net)flow specification rules

Disclaimer
draft-marques-idr-flow-spec-03
”Successor” of uRPF + BGP null routes
Former is a clever engineering trick to null route prefixes
Latter formalizes the propagation of flow-like informations through
BGP for further action
Dst prefix, src prefix, proto, src & dst port, icmp type & code, TCP
flags, packet length, ToS, fragment
Basically a flow record

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 124 / 126
 What’s next Perhaps not really next

State of the flow-spec draft

Implemented by one vendor

Other is waiting for customer traction ?
A NetFlow vendor from MI
No (public) implementation on any Unix BGP daemon - too bad
Spot traffic to be acted upon from a BGP-speaking NetFlow analysis
workstation
Use flow-spec support to inject and propagate said records

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 125 / 126
 What’s next Resources

Some links

http://www.cisco.com/go/netflow/
http://www.cisco.com/en/US/products/ps6601/
prod presentation list.html
http://www.switch.ch/tf-tant/floma/software.html

Yann Berthier () NetFlow to guard the infrastructure NANOG39 - Feb 4-7, 07 126 / 126