Netflow Analysis Cisco
Netflow Analysis Cisco
INTRODUCTION
The Cisco IOS NetFlow feature set allows for the tracking of individual IP flows as they are received at a Cisco router or switching device. Network
administrators can use the NetFlow flow records for a variety of purposes, including accounting, billing, network planning, traffic engineering, and
user or application monitoring.
NetFlow services are available on Cisco IOS Software-based routers including Cisco 800 thru 7500 Series Routers, as well as the Cisco Catalyst
6500 Series Switch, Cisco 7600, 10000, 12000 Series Routers and CRS-1 devices.
For more details on NetFlow, refer to the NetFlow Services Solutions Guide white paper at
http://www.cisco.com/univercd/cc/td/doc/cisintwk/intsolns/netflsol/nfwhite.htm.
Although many Cisco customers want to deploy NetFlow services, they are naturally cautious about introducing new technology into their network
without completely understanding the potential performance impact. This paper examines the CPU impact of enabling NetFlow services in several
different scenarios and on several different Cisco hardware platforms.
This information is valuable for planning potential NetFlow implementations, but these results are not a replacement for proper customer lab testing,
pilot deployments, and other types of solution validation.
TESTING METHODOLOGY
All tests were performed using Cisco IOS Software Release 12.0S. The platforms and configurations tested include:
Ten test cases were defined and not all are run on all the platforms listed. The test cases are documented below. A mnemonic has been assigned to
each test case to make it easier to understand the charts and graphs in the Test Results and Test Analysis sections of this document. Table 1
describes the test cases.
All contents are Copyright 19922005 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement.
Page 1 of 19
Table 1. NetFlow Test Cases
Baseline Baseline test without NetFlow enabled; provides a context for the tests that follow
Nf-load Statistics generated immediately after NetFlow services are enabled on the router; tests any unusual initialization
requirements of NetFlow
Nf-enable NetFlow Version 5 enabled but no export destination defined; documents the effects of NetFlow on the router itself
Nf-NDE NetFlow Version 5 enabled and NetFlow data export (NDE) destination also defined; tests the effects of NDE on
the router
Nf-NDE-2 NetFlow Version 5 enabled and two different NDE destinations also defined; tests the effects of NDE with multiple
destinations on the router
v9-NDE1 NetFlow Version 9 enabled and NetFlow data export (NDE) destination also defined; tests the effects of NDE on
the router
v9-NDE2 NetFlow Version 9 enabled and two different NDE destinations also defined; tests the effects of NDE with multiple
destinations on the router
Nf-NDE-AS NetFlow Version 5 enabled and recording autonomous-system origin of packets; NDE destination also defined;
used to test the effects of maintaining information about autonomous systems with NetFlow
NF-Prefix-V8 NetFlow configured with a Version 8 prefix aggregation scheme (but no NDE); compare results with NetFlow
Version 5
NF-AS-V8-NDE NetFlow configured with a Version 8 autonomous system aggregation scheme and NDE; compare results with
NetFlow v5
NF-AS-TOS NetFlow configured with a Version 8 autonomous system- type of service (ToS) aggregation scheme and NDE;
compare results with NetFlow Version 5
Each test case was performed with three different IP flow rates: 10,000, 45,000, and 65,000 flows (this represents the number of unique IP flows that
were seen by the router for each test). For some platforms 70K flows were tested instead of 65k flows. The flows were sent in a loop, so that the
NetFlow cache was populated by the first iteration of the traffic stream and used for switching the packets on subsequent iterations.
To ensure accuracy of the results and to eliminate any anomalies, output of the tests was not collected until NetFlow had been running for ten
minutes (the Nf-load test case was the exception to this rule). Six samples of output were taken at one-minute intervals following that. The numbers
presented in the Test Results portion of this document represent the average of those results.
Note that the testing provided a worst-case scenario in terms of the traffic flows seen by the routers, and the results must be viewed in that context.
All packet sizes were 64 bytes, a scenario that tends to be more stressful on a router than a mixed traffic stream of various sizes.
At each flow rate, there was no duplication of flows until the test began its second iteration through the testing loop.
TEST RESULTS
This section presents the results of the test cases described in the previous section. Interpretations and conclusions to be drawn from the data are
discussed in the Test Analysis and Conclusions section of this document.
The results are presented in raw, tabular format, so that readers will have all available information, and can utilize the numbers to extrapolate the
results into their own environment.
Baseline 16 16 16
Nf-load 25 42 49
Nf-enable 35 62 68
Nf-NDE 39 64 69
Nf-NDE-2 39 64 70
Nf-NDE-AS 39 64 70
NF-Prefix-V8 40 63 70
NF-AS-V8-NDE 40 63 70
NF-AS-TOS 40 65 70
Baseline 10 22 25
v5, NDE 1 14 36 48
v5, NDE 2 13 36 49
v9, NDE1 15 33 48
v9, NDE2 14 37 60
Baseline 6 3 7
Nf-load 9 13 15
Nf-enable 16 33 39
Nf-NDE 17 33 40
Nf-NDE-2 17 35 39
Nf-NDE-AS 17 33 39
NF-Prefix-V8 20 35 41
NF-AS-V8-NDE 19 33 41
NF-AS-TOS 17 33 42
Baseline 10 23 28
v5, NDE 1 14 35 47
v5, NDE 2 14 35 51
v9, NDE1 14 35 49
v9, NDE2 15 37 60
Baseline 7 11 13
v5, NDE 1 10 19 24
v5, NDE 2 10 19 25
v9, NDE1 10 19 25
v9, NDE2 10 20 33
Baseline 6 8 12
Nf-load 9 19 26
Nf-enable 9 19 27
Nf-NDE 9 19 26
Nf-NDE-2 9 19 27
Nf-NDE-AS 9 19 27
NF-Prefix-V8 9 19 27
NF-AS-V8-NDE 9 19 26
NF-AS-TOS 9 19 27
Baseline 11 27 35
v5, NDE 1 18 35 38
v5, NDE 2 16 35 43
v9, NDE1 17 37 40
v9, NDE2 16 27 35
Baseline 8 18 22
v5, NDE 1 8 20 24
v5, NDE 2 7 18 23
v9, NDE1 10 20 24
v9, NDE2 8 18 22
Baseline 2 3 5
Nf-load 5 15 21
Nf-enable 5 15 22
Nf-NDE 5 16 22
Nf-NDE-2 5 16 22
Nf-NDE-AS 5 16 22
NF-Prefix-V8 6 15 23
NF-AS-V8-NDE 6 15 22
NF-AS-TOS 6 16 22
Baseline 6 9 15
Nf-load 9 20 28
Nf-enable 9 20 28
Nf-NDE 9 20 28
Nf-NDE-2 9 20 28
Nf-NDE-AS 9 19 28
NF-Prefix-V8 9 20 29
NF-AS-V8-NDE 9 20 28
NF-AS-TOS 9 20 28
Because Distributed Cisco Express Forwarding was employed in this test, the CPU utilization numbers were collected on the
Versatile Interface Processors (VIPs), not the main CPU.
Baseline 7 8 11
Nf-load 12 23 30
Nf-enable 12 23 31
Nf-NDE 12 24 31
Nf-NDE-2 12 23 31
Nf-NDE-AS 12 24 31
NF-Prefix-V8 12 25 32
NF-AS-V8-NDE 12 23 31
NF-AS-TOS 12 23 31
Because Distributed Cisco Express Forwarding was employed in this test, the CPU utilization numbers were collected on the line cards,
not the main CPU.
Baseline 7 8 11
Nf-load 9 12 14
Nf-enable 9 12 14
Nf-NDE 9 12 14
Nf-NDE-2 9 12 14
Nf-NDE-AS 9 12 14
NF-Prefix-V8 9 12 14
NF-AS-V8-NDE 9 12 15
NF-AS-TOS 9 12 14
As above, because Distributed Cisco Express Forwarding was employed in this test, the CPU utilization numbers were collected on
the line cards, not the main CPU.
In terms of additional CPU utilization (over and above the baseline), a few trends can easily be discerned. As the number of flows increases, the delta
between the baseline and NetFlow-enabled CPU utilization widens. In other words, the more IP flows present, the more system resources NetFlow
requires. Although these results were expected, they confirm the accuracy of the expectation. The more active flows NetFlow is maintaining in its
cache, the larger the cache becomes and the more CPU it requires to sort through the cache.
Note that, in the Figures below CPU utilization does not seem to vary greatly, depending on the particular NetFlow features that are enabled. Neither
the recording of autonomous-system numbers nor the addition of NetFlow data export (even to multiple destinations) makes a large impact on overall
CPU utilization. Not surprisingly, the Cisco 2600 Router seems to vary the most in this regard.
Figure 13. Cisco 12000---Distributed Cisco Express Forwarding with 1:100 Sampling---CPU Utilization
There are two sampling techniques packet sampling and flow sampling. In packet sampling only a subset of the total packets switched in the box are
used to create flows in the NetFlow cache. Many customers utilize one in one hundred or one in one thousand packet with packet sampling and this
is optimum for capacity planning. When 1:100 packets are sampled then approximately 80% of the flows will be created in the NetFlow cache. This
obviously reduces the number of flows created and exported decreasing CPU utilization and export traffic volumes.
The Cisco Catalyst 6500 Series Switch uses flow sampling where a subset of the total flows in the NetFlow cache are exported to the collector and
again this saves on CPU utilization and export traffic volume. The following table outlines CPU decrease possible with packet sampling on a Cisco
7500 Router. This table is provided as a guideline to how sampling can reduce CPU utilization. The reduction of CPU utilization is dramatic when
packet sampling is implemented.
1:100 75
1:1000 82
The Cisco 12000 Series Router can make most forwarding decisions directly in the hardware on the line cards and, can achieve very high levels of
packet throughput.
When enabling NetFlow on the Cisco 12000 Series Router, the hardware forwarding mechanism is bypassed, and packet-forwarding decisions are
made in software.. This caveat is relevant only on line cards on the Cisco 12000 Series Router that utilize a software-based version of NetFlow
(Engine 0, 1). Cisco 12000 Engine 2, 3, 4+ and 5 line cards have NetFlow implemented on a hardware application-specific integrated circuit (ASIC),
so they can switch NetFlow packets at or near line rate. In addition, because all this occurs in hardware, there is no performance penalty on the CPU
of the line card.
When using a software-based NetFlow implementation on the Cisco 12000 Series Router, sampled NetFlow is recommended. The sampled NetFlow
feature allows the router to sample one out of x IP packets being forwarded. Sampled packets are accounted for in the router NetFlow flow cache.
Sampling packets substantially decrease the CPU utilization for NetFlow packets, allowing most packets to be switched faster without additional
NetFlow processing.
Although the use of sampled NetFlow on the Cisco 12000 Series Router has always been strongly encouraged, this testing makes the benefits quite
clear. Figure 14 illustrates that with a sampling rate of 1:100 (that is, one out of every 100 packets is sampled to give a representative picture of the
data), the increase in CPU utilization is negligible. This result can be contrasted with cases in which nonsampled NetFlow was enabled, and the
impact on CPU is significant.
Note that sampled NetFlow does not provide the level of granularity that full-flow NetFlow does. Thus, the user must always attempt to strike a
balance between granularity and performance. The configurable sampling rate can be used to help achieve that balance.
Software based platforms benefit from sampling with large reductions in CPU as shown in Table 14. The Cisco 12000 Series Router with 1:100
sampling is the clear winner here, averaging only 3 percent excess CPU utilization when enabling sampled NetFlow. However, as mentioned in the
previous section, there is a resulting loss of granularity when sampling.
Among the non-sampled platforms, the Cisco 7500 Series Router with Distributed Cisco Express Forwarding does the best, using only 13 percent
additional CPU when tracking 65,000 IP flows. The following results do not include Cisco 7200 NPE300, 2800, 3700 or 3800 Series Routers.
Trying to provide some reasonable expectations for CPU utilization when enabling NetFlow is a complex task. With the Cisco 12000 Series Router,
clearly the numbers will vary according to the sampling rate selected.
Averaging the results across the different (nonsampled) platforms yields some reasonable expectations for the impact of enabling NetFlow:
With ~10,000 active flows: 7.14 percentage points of additional CPU utilization
With ~45,000 active flows: 19.16 percentage points of additional CPU utilization
With ~65,000 active flows: 22.98 percentage points of additional CPU utilization
First, the Cisco 12000 Series Router numbers (both sampled and nonsampled) are an anomaly, because they depend so much on the sampling rate.
Therefore, these numbers were not used in calculating the average.
There is not a significant CPU difference caused by enabling any particular set of NetFlow features. This observation results in the average CPU
per platform in Figure 15.
This average obtained tends to obscure differences between platforms. However, the testing tends to confirm that the most relevant factor in
determining additional CPU utilization is the flow rate, where the difference is much more significant than the platform differences.
On the other hand, the numbers obtained are an average of an average (of numbers which, themselves, are averages). These numbers are provided
as an approximation of additional CPU utilization. Individual platform results may vary.
The testing does not bear out this assumption. As illustrated in Figure 16, there is minimal to no impact in turning on Version 8 aggregation schemes,
as compared to the same router running NetFlow Version 5. Although the Cisco 7500 Series Router was used to demonstrate this point, the results
are typical of the results across all tested platforms.
Figure 16. Cisco 7500 RSP 8 Cisco Express Forwarding---Version 5 vs. Version 8
RECOMMENDATIONS
This testing illustrated a significant benefit to using sampled NetFlow on the Cisco 12000 Series Router. Whereas a Cisco 12000 Series Router
running in nonsampled full-flow mode required 23.5-percent more CPU to handle 65,000 IP flows, the same router using 1:100 sampling required
only 3-percent additional CPU. CPU utilization will vary, depending on the sampling rate.
Customers are encouraged to use sampled NetFlow on the Cisco 12000 Series Router, and to use a high sampling rate (for example, 1 to 100) for
initial deployment. Then, if more granularity in the NetFlow output is required, the sampling rate can be adjusted downward.
These results are based on an average of several different tests on several different hardware platforms. Individual platform results may vary.
Estimate the average number of IP flows passing through their target NetFlow deployment platforms (an IP flow is defined as a unique
combination of the source and destination IP addresses, source and destination TCP/User Datagram Protocol [UDP] port numbers, IP protocol
type, IP type of service, and input interface).
Monitor the average CPU utilization on these platforms.
Customers who have considered enabling these features may now feel more confident about doing so.
NetFlow Version 8 decreases the bandwidth and storage requirements on the NetFlow collector device, because flow records are aggregated at the
router prior to export. However, the records it produces are also less granular, so customers must be sure that the Version 8 records will provide all
the information necessary for fulfilling their business needs.
NetFlow version 9 is used for the latest technology and features for NetFlow and should be considered for new deployments because the impact is no
greater than version 5.
These test results do not change that recommendation. They do, however, illustrate that concern over additional CPU requirements of NetFlow
version 8 or version 9 should not be one of the assessment criteria for its deployment.
CONCLUSION
The numbers included in this paper can be used as an initial reference for customers who wish to deploy the NetFlow feature set. Note that---as
extensive as the testing was---it was done on only a specific set of platforms, and with a specific code release. Cisco is constantly working to improve
the NetFlow caching scheme and hashing algorithms, so future releases of code may produce even better results.
As with all lab testing, an attempt was made to model real-world scenarios. However, no such modeling can ever be perfect. Remember that these
benchmarks are to guide initial deployment decisions, but they do not replace the due diligence of customer lab testing, pilot deployments, and other
types of solution validation.
Cisco Systems has more than 200 offices in the following countries and regions. Addresses, phone numbers, and fax numbers are listed on
the Cisco Website at www.cisco.com/go/offices.
Argentina Australia Austria Belgium Brazil Bulgaria Canada Chile China PRC Colombia Costa Rica Croatia Cyprus
Czech Republic Denmark Dubai, UAE Finland France Germany Greece Hong Kong SAR Hungary India Indonesia Ireland Israel
Italy Japan Korea Luxembourg Malaysia Mexico The Netherlands New Zealand Norway Peru Philippines Poland Portugal
Puerto Rico Romania Russia Saudi Arabia Scotland Singapore Slovakia Slovenia South Africa Spain Sweden Switzerland Taiwan
Thailand Turkey Ukraine United Kingdom United States Venezuela Vietnam Zimbabwe
Copyright 2005 Cisco Systems, Inc. All rights reserved. CCSP, CCVP, the Cisco Square Bridge logo, Follow Me Browsing, and StackWise are trademarks of Cisco Systems, Inc.;
Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Access Registrar, Aironet, ASIST, BPX, Catalyst, CCDA, CCDP,
CCIE, CCIP, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity,
Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, FormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ
Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, Linksys, MeetingPlace, MGX, the Networkers logo, Networking Academy, Network Registrar, Packet, PIX, Post-
Routing, Pre-Routing, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, StrataView Plus, TeleRouter, The Fastest Way to Increase Your Internet Quotient, and TransPath are
registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
the
All other trademarks mentioned in this document or Website are 2005 Cisco
property of Systems, Inc.owners.
their respective All rights reserved.
The use of the word partner does not imply a partnership relationship between
Cisco and any other company. (0502R) notices, privacy statements, and trademarks of Cisco Systems, Inc. can be found on cisco.com.
Important 205283.a_ETMG_SH_4.05
Page 19 of 20
Printed in the USA
2005 Cisco Systems, Inc. All rights reserved.
Important notices, privacy statements, and trademarks of Cisco Systems, Inc. can be found on cisco.com.
Page 20 of 20