Link aggregation and networking enhancements.

DataPower_70_Networking.ppt Page 1 of 40
Today we'll cover three major topics:
We'll start the new features relating to DataPower® Networking, primarily Link Aggregation
but also enhancements in other areas.
Then we'll move on to more general changes in networking configuration. The most
significant of these is the way the network configuration objects reflect errors back to the
administrator.
Lastly we will discuss problem determination – what to look at, what to collect when
contacting IBM support.

DataPower_70_Networking.ppt Page 2 of 40
The largest new networking feature in version 7 is the addition of link aggregation. This is
the often requested capability to use many physical Ethernet interfaces as a single
interface for better availability and increased bandwidth.
We also enhanced standby control. We now allow a standby control group on every
interface that has an IP configuration. We also added a new operational control that is
helpful when bringing one member out of a standby control group.
Lastly, we improved failure handling. In version 7, it is now considered a system wide
error when the underlying network stack is not operating as configured.

DataPower_70_Networking.ppt Page 3 of 40
Link Aggregation is the marquee networking feature of Version 7. It allows multiple
Ethernet interfaces to work together and act as a single network interface. This is a
technology that is widely used the data center as a component of the availability and
bandwidth management story.
With Link Aggregation, as long as one of the member links is available, the overall
aggregation interface is up. In this way, link aggregation is an enabling technology for
architectures that have no single point of failure. Additionally, with link aggregation it is
possible to 'stripe' network data across the aggregated interfaces. In this way, a link
aggregation interface can have more bandwidth than any one Ethernet interface.
Lastly, not only is link aggregation a standard component of enterprise network
deployments, it is a frequently requested DataPower feature.

DataPower_70_Networking.ppt Page 4 of 40
In order to use Link Aggregation, the first step is to know that it is compatible with the
network environment. This is an explicit step. Before Link Aggregation, as part of the
deployment process the site network team would provision switch ports in addition to IP
addresses. The team in the DataCenter would then cable the appliance to the ports
specified by the network team, and the DataPower team would configure the interfaces
with the addresses and routes specified by that provisioning process. None of this
process changes. What does change is that there are now a different set of switch ports
involved and a different configuration on DataPower.
DataPower supports three modes of Link Aggregation, each are useful in different
scenarios that vary based on the network topology, so the site network team should be
involved in choosing the appropriate technique. The three major modes, roughly in order
of preference, are LACP, or Link Aggregation Control Protocol. This is the IEEE standard
for link aggregation. The second technique is active-backup. It is most useful in cases
where the switch topology means that LACP is not an option. An example of this is the
blade form factor where the switch modules affect the switch topology and therefore
preclude LACP. Lastly is Transmit-based, which is useful in cases where active-backup
would be indicated but where increased transmit bandwidth is desired, such as with
DataPower's self balancing feature.
The next slides walk through screen shots of the configuration process. Link aggregation
is configured in several steps. First, Ethernet interfaces are placed into link aggregation
mode. Then one creates a link-aggregation interface. In this interface, add Ethernet
interfaces, configure the aggregation, and add IP layer configuration as needed.

DataPower_70_Networking.ppt Page 5 of 40
The first step to using Link Aggregation is to configure a number of Ethernet interfaces to
use Link Aggregation.
To do this, enter the Ethernet interface (as seen by (1)), and toggle link aggregation (by
(2)). That's it.
When an Ethernet interface is in link aggregation mode, there are no longer any IP layer
configuration properties available because these will be configured on the link-aggregation
interface.
After making these changes, they must be applied to take affect.

DataPower_70_Networking.ppt Page 6 of 40
Once there are Ethernet interfaces that to use in an aggregation, create the link-
aggregation interface.
Navigate to “Link Aggregation Interface” by (1), then click add by (2).

DataPower_70_Networking.ppt Page 7 of 40
After clicking 'add', the webgui will be in the main configuration panel for the link
aggregation object, as seen by (1).
Give the aggregation a name (by (2)). In this case, the name is 'dataagg1g', reflecting the
purpose of this aggregation, which in this example is to test an aggregation consisting of
eth10 through eth17.
Next select the aggregation mode (by (3)). I know that the switches to which this
appliance is cabled is configured for LACP, and I know that eth10 through eth17 are
cabled to the appropriate ports, so I make those selections here. I add eth10 through
eth17 by (4).
This is all that must be done in order to configure a link aggregation interface. If we apply
now, there will be no IP address on this interface, but it still could be used for vlans. In this
example, an IP layer configuration on 'datagg1g' is required, so we continue on the same
panel:

DataPower_70_Networking.ppt Page 8 of 40
Still on the main panel of the link aggregation interface, below the link-aggregation specific
settings is the IP layer configuration.
Here we add an IP address (1) and appropriate routing (2). Note that there are no default
gateways or other routes required for this interface, so they are blank.
The IP layer configuration is identical for every interface type.
Lastly, click 'Apply' at the top of the page to complete the link aggregation interface.

DataPower_70_Networking.ppt Page 9 of 40
VLANs and Link Aggregation are sometimes confused with each other. They are different
sides of the same coin, they share some terminology, and both are a lower level network
function than is typically handled by the administrator of an application layer proxy like
DataPower.
Vlans take a single network interface and subdivide it into many logical network interfaces,
each of which are separate from each other. VLANs are in use everywhere in every
datacenter, but they are often hidden from application layer devices like DataPower. The
terms “access mode”, “untagged”, and “native vlan” all refer to traffic that is on a vlan in
the network but that does not appear to be in a vlan from the point of view of the
appliance.
Link aggregation combines several network interfaces into one 'super' interface. Terms
that refer to link aggregation include “LACP”, “etherchannel”, “nic teaming”, “nic bonding”,
“nic bundling”, and “nic or port trunking”. But be careful – the Cisco switch command
“switchport mode trunk” refers to VLANs, not link aggregation.
Lastly, VLANs can and often are used in conjunction with link aggregation. This is a very
well developed standard pattern regularly seen in the networking world. It allows solving
two hard questions completely independently of each other – “How do I satisfy my
availability and bandwidth requirements?” and “How do I satisfy my network separation
requirements?”.

DataPower_70_Networking.ppt Page 10 of 40
To put a VLAN on a Link Aggregation interface, first navigate to the VLAN interface (1).
Select the appropriate VLAN, and either add it or note that the config panel is displaying
the correct vlan (2).
From there, the “VLAN hostd on” field (3) must be changed to “Link Aggregation” from
“Ethernet”.
Lastly, select which link aggregation interface the VLAN should be associated with (4).
The other properties of the VLAN need not change. The IP address or addresses, the
VLAN ID, etc all remain the same.

DataPower_70_Networking.ppt Page 11 of 40
There are a large number of ways that link aggregation can be used to help achieve the
business requirements for availability and throughput. Keep in mind that the principle at
play should be for DataPower to follow the existing network topology – DataPower is not
unique when it comes to how it should be configured on the network. The topologies that
are already present in the DataCenter exist to serve the same business requirements that
DataPower exists to serve.
The upcoming slides show a number of ways that it is possible to use link aggregation with
a focus on DataPower scenarios. Note that which is appropriate depends on the business
requirements, but some patterns are more general than others. Each of these sample
topologies demonstrate a point, they are not a recommendation or best practice even
though some have clear benefits over others.
To reiterate, it is usually the role of the site network team to specify how link aggregation is
used in that environment. And DataPower simply integrates into that environment. Even
different environments in the same organization may have different goals thus different
topologies. There is no one size fits all solution.

DataPower_70_Networking.ppt Page 12 of 40
In the first, and simplest sample topology, we have an appliance that exists in two security
zones and has a requirement for link layer availability. Because of that requirement, the
overall network topology is robust and can tolerate any single point of failure, so we can
assume that as long as one interface/switch per zone is available, the appliance has full
connectivity.
This sort of a topology is useful with the xi50b when the chassis is configured with switch
modules, where LACP (which is generally preferred) is not available.
Additionally, note that there are two different aggregations. There is absolutely no
DataPower requirement that different types of traffic be separated into different
aggregations, so the connection of the single appliance to multiple zones should be driven
by customer business requirements. Perhaps there is a physically separate management
network. Perhaps there is a requirement for physical separation in the DMZ. DataPower
is configured to match the buisness requirements.
If further network layer separation is required, it is achieved with VLANs. For instance, if
the 1G nics on the xi50b are truly dedicated for management, then the 10G nics might be
both internet facing and intranet facing, but on different vlans.
Lastly, note that this link aggregation configuration – with no switch support – will not give
additional bandwidth and that its failover properties will not be as graceful as with LACP.

DataPower_70_Networking.ppt Page 13 of 40
This second example modifies the first by using LACP instead of active-backup. This
delivers increased bandwidth, but removes the single point of failure that was present in
the first example.
Note that if either of the switches fail the appliance looses connectivity to an entire zone.
There is no switch redundancy!
Also, note that each of the switch ports now requires specific LACP configuration in order
to be used in this topology. Which in turn means that it is important exactly which switch
ports the appliance connects to.
Like the previous example, there are two different zones the appliance connects to. If this
is intentional and required, then there is no problem. However, there is an increased,
avoidable cost to this architecture if different physical interfaces are truly not required.
Consider using VLANs to separate zone 1 from zone 2.
Also, this example has a major potential issue with availability. There is still a single point
of failure. If the business requirement is to eliminate single points of failure, then this
topology cannot deliver.

DataPower_70_Networking.ppt Page 14 of 40
Improving on example 2 we have sample network topology 3. Here, we are still using
LACP, only we eliminated the second zone. All traffic isolation will be done with VLANs.
Additionally, note that the label on the switch changed to “Logical Switch”. By a “logical
switch”, I mean multiple physical switches that act, for the purposes of LACP like a single
switch. Multiple switch vendors have technology such as VLAG (Virtual Link Aggregation)
that virtualizes link aggregation, so the group of switches presents itself as a single
endpoint as the LACP protocol requires. If one of the switches fails, then all traffic can
continue through the other.
VLAG has faster and better recovery properties than the similar approach discussed next
and can “stripe” across all interfaces simultaneously when there is no failure. For physical
appliances where the network architecture and switch configuration support this
architecture, it is the most robust. Even so, keep in mind that it is the role of the
DataPower appliance to integrate into the existing network architecture. And the existing
network architecture must also be sufficient to meet the requirements of the business.

DataPower_70_Networking.ppt Page 15 of 40
Link Aggregation is about taking multiple physical interfaces and using them as one logical
interface. With Virtual DataPower, there are no physical interfaces. What then is the role
of Link Aggregation?
The short answer is that there is really no or very limited production role for link
aggregation in virtual edition. The only possible exception is the rare case where a
hypervisor dedicates physical hardware is dedicated to a guest. The hypervisor or vswitch
is responsible for managing the relationship between the physical switch and the
hypervisor nics.
Nevertheless, Link Aggregation is present on DataPower virtual edition appliances. Even
though Link Aggregation is not suitable for production use because it is simply not the role
of a guest, it is still useful to gain experience with configuration, monitoring, automation,
and training of the DataPower link aggregation interfaces in preproduction or development
environments. It is useful for working with SOMA. It is useful for understanding some of
the status providers. It is not useful for improving the availability of the virtual appliance –
that is the job of the hypervisor or the vswitch.

DataPower_70_Networking.ppt Page 17 of 40
While there are no known limitations to DataPower's link aggregation feature, there is still
much more in the world of link aggregation than DataPower supports.
Link Aggregation in DataPower must be complementary to the network into which it is
deployed. This requires coordination with the network operators. Additionally, each
network may have different operators. Be aware of any differences between
environments.
Lastly, on models 7198 and 7199, DataPower offers IPMI (Integrated Platform
Management Interface) on the mgt0 interface. DataPower's IPMI Lan console feature
offers out-of-band management including remote serial console and power control, but it is
only available on mgt0. For the fewest interruptions, consider dedicating mgt0 to IPMI
exclusively. IPMI lan consoe on mgt0 is incompatible with LACP link aggregation on mgt0.
If link aggregation on mgt0 is required in conjunction with IPMI lan console, then LACP
must not be the link aggregation mode. Out of band management capability, such as
IPMI, is a critical component of an appliance management strategy.

DataPower_70_Networking.ppt Page 18 of 40
Standby control is DataPower's front-side IP-based multiple appliance availability feature.
It allows two or more appliances to be jointly responsible for an IP address. All appliances
in a standby control group can handle any request that comes to the group.
New in version 7 is the ability to use standby control on every IP interface at the same
time. In previous releases, if there was a standby control group on an Ethernet interface,
then no vlan on that Ethernet interface could use standby control. This new feature will
especially help those who deploy multiple standby control groups as they adopt link
aggregation and those who use different standby control groups for different lines of
business and were constrained by the physical device limitation.
Also new in version 7 is an operational control called 'yield standby'. This action causes
standby control to 'start over', which includes giving up the role of active if the device was
active.
Another change is that the standby control virtual IP addresses can now be on a different
subnet than the VIP. It still must be on the same segment, but it is now possible to use for
example unroutable 10-dot addresses for the group members with a publicly routable VIP.
In version 7 we removed the ability to use MAC takeover for standby control.

DataPower_70_Networking.ppt Page 19 of 40
To configure standby control, navigate to the appropriate interface. The procedure applies
to each interface type – Ethernet, vlan, and link-aggregation. Standby Control is now on its
own tab, select the standby control tab.
When standby control is enabled (see (2)), the group, vip, and other settings are all
available for configuration.
Every interface also has the ability to yield standby (see (3)). When this used, this
member of the standby control group 'starts over'. It acts like it is a brand new member. If
it is active, it gracefully relinquishes the role so the standby device does not need to wait to
become active.

DataPower_70_Networking.ppt Page 20 of 40
In DataPower version 7, there are also general changes to the way networking is
configured.
First, all the network interface objects behave in a similar and consistent manner. The
web gui layout is similar. The command line interface matches. Changes are applied in a
manner consistent with the interface objects and the other DataPower configuration
objects.
When a network interface object is configured, that configuration takes place on “Apply” in
the webgui and on “exit” in the CLI. The network interface is always completely
reconfigured on Apply – it will “blip” out and return freshly reconfigured. If reconfiguration
is not the goal, then use “Cancel” in the WebGui and cli and the configuration of the
interface will not be changed.
The CLI commands are also different, although all the old commands are still accepted.
They are now consistent with each other and with the patterns predominant throughout the
CLI.

DataPower_70_Networking.ppt Page 21 of 40
There is a new configuration concept to clearly delineate when an interface cannot be
modified because it is controlled by another interface.
A “Locked” Ethernet interface becomes locked when it is a both configured for link
aggregation and member of an admin enabled link-aggregation.
A locked Ethernet interface cannot be modified. Any attempts to change it are blocked.
Neither 'Apply' in the webgui nor 'exit' in the cli will work, must “Cancel” instead.
Locked Ethernet interfaces can be unlocked by removing them from all aggregations or by
administratively disaabling the aggregation.
When an Ethernet interface is in an aggregation, the aggregation governs everything
about the Ethernet. Yet there still Ethernet settings (such as MAC address) that are
inherited by link aggregation from the Ethernet, changing these could only result in an
invalid persisted configuration.

DataPower_70_Networking.ppt Page 22 of 40
When we try to make any change to an Ethernet interface that is part of an aggregation,
no matter how small, and “Apply” that change (1), we see that the interface is locked, and
the change cannot be applied, and the only way off this screen is 'Cancel'.

DataPower_70_Networking.ppt Page 23 of 40
On the CLI, there is a warning as soon as the Ethernet object is entered, see (1).
Any attempt to exit (see (2)) is met with an error message.
Lastly, the only way to leave this locked Ethernet object is with 'cancel' (see (3)).

DataPower_70_Networking.ppt Page 24 of 40
Handling configuration failures of network interfaces is a tricky problem. On one hand,
administrators require immediate feedback of any problems. No one wants an appliance
in production with an invalid or broken network configuration.
On the other hand, DataPower administration is most often performed over the network.
While there are out-of-band management solutions, including terminal servers and IPMI
lan console for rack-mounted appliances, virtualization console for virtual edition
appliances, and the BladeCenter® management module for the xi50b, these are not
always deployed or accessible. As an aside, IBM strongly recommends including out-of-
band management in your appliance management strategy.
With this background, we now consider network configuration errors to be critical,
unrecoverable errors which should prevent the device from going into production – but we
allow access via the WebGui, CLI, telnet, and XML management interfaces so the
administrator can correct the problem. However, all other application traffic is blocked.
There are banners in the webgui and cli. Log messages. Application traffic is blocked to
provide backpressure to a load balancer and actually prevent bringing an appliance with a
broken network configuration into production.
There are some scenarios where this may not be a desired behavior, and for those cases
the enforcement can be disabled. However, the display banners cannot be disabled.

DataPower_70_Networking.ppt Page 25 of 40
As an example, I added an invalid default gateway to a vlan interface. The next-hop
address for the route was not on a local subnet.
I then (1) “Apply”'d the change, which failed. The failure notice (2) states that the failure is
because it “Cannot set route”. A default gateway is a route, so this matches.
At this point, based on standard DataPower semantics, we would expect that the vlan
would be down and that the vlan would not work. That is not what we want to happen
though – it could be that this vlan is used for management traffic. Instead, we bring the
interface up on a best-effort basis, display the message (3), block all network
communication for non-management services, and stop standby control.
When the configuration error is corrected, then the warning banners are no longer
displayed and traffic is handled as usual.

DataPower_70_Networking.ppt Page 26 of 40
The warning banners are displayed globally throughout the webgui. They are not limited
to the network interface configuration objects or even to the default domain. They are
seen globally.

DataPower_70_Networking.ppt Page 27 of 40
It is a similar story on the CLI.
The warning banner is displayed upon login (1).
The warning banner is displayed whenever leaving any configuration sub-mode (2).
And the warning banner is displayed whenever leaving configuration mode (3).

DataPower_70_Networking.ppt Page 28 of 40
Under some circumstances, it may be preferable to prevent blocking of nonmanagement
traffic in the event of a network configuration error.
The best example of such a time is in lower preproduction environments, where multiple
people or groups are sharing the same appliance and one is focusing on networking and
making changes. But the other groups may not even use the same interfaces.
To disable the blocking of nonmanagement traffic (but not the warnings), go to network
settings (1) and disable the option to block nonmanagement traffic (2). Then Apply.
One last note about this additional level of protection. It cannot guarantee that the
network configuration is correct. It can only guarantee that it is doing what was
configured. There are still ways that both production traffic and management traffic can be
broken by a network configuration change. For this reason, always have an out of band
management strategy.

DataPower_70_Networking.ppt Page 29 of 40
When migrating from earlier releases to DataPower version 7, there are some changes to
plan for.
Saved changes from earlier firmware versions will continue to work as expected, with one
exception. In v7, an interface may be either dhcp or IPv6 autoconfigure / slaac or staticly
configured. One cannot use dhcp for addresses and add other addresses and routes too.
Import in general works as expected, but exports from pre-v7 that contain standby control
groups will not import into v7.
Lastly, there are a large number of changes to the way status is displayed. There is more
information available. The IP address syntax changed so the same status is available in
SNMP. Configuration can be viewed on the CLI,, including Ethernet, vlan, link-
aggregation. There are new status providers 'link', 'ipaddress', 'link-aggregation-status',
and 'link-aggregation-member-status'.

DataPower_70_Networking.ppt Page 30 of 40
The new “Link” status provider has an entry for every logical network interface that is
present on the appliance, even those that are indirectly used by various DataPower
features. The ifindex is the key to this table – every enumerated low-level interface is
present, along with link level information such as mode, mtu, mac address, and an what (if
any) aggregation is is a member of.
The Aggregation interface is used as an index into the link-aggregation-status display.

DataPower_70_Networking.ppt Page 31 of 40
The IP Address status provider displays every IP address that is present on the appliance,
including the interface it is on and its subnet or prefix.

DataPower_70_Networking.ppt Page 32 of 40
The link aggregation status provider shows the current status (not configuration!) of what
is happening with each link aggregation interface. For active-backup aggregations, this
status shows which interface is the current active interface. For LACP aggregations, this
status provider shows LACP protocol state information that can be useful in conjunction
with the network administrator to understand how each device sees the other.
Note 'dataagg1g' has aggregator ID '1'. We can use this information combined with the
link aggregation member status to see which interfaces are actually being used in a LACP
aggregation.

DataPower_70_Networking.ppt Page 33 of 40
Each Ethernet interface that is currently operating in an aggregation appears in the Link
Aggregation Member status display.
Every member that is in a LACP aggregation has an “Aggregator ID”. If two aggregator
IDs are identical on the same link-aggregation interface, that indicates that LACP has
successfully aggregated those links together.
Note the aggregator IDs for dataagg10g. Eth12 and eth13 have aggregator ID 1, eth10
and eth11 have aggregator id 2. The rest of the interfaces have unique aggregator ids.
What does this show?
It shows that eth12 and eth13 can be used together, eth14 and 15 can similarly be used
together, and that none of the remaining interfaces can be used in conjunction with each
other.
From the previous slide, we noted that dataagg1g is using aggregator id '1'. This
corresponds to eth12 and eth13 – so those are the interfaces that are presently being
used.
Looking back at the link status, note that each the unmatched interfaces have no link.
Ethernet interfaces that have no link cannot be an participate in an aggregation, but if one
came up, and it could have agg id 1 or 2 – meaning it could be used once the link situation
is resolved.

DataPower_70_Networking.ppt Page 34 of 40
The vlan status provider shows the configuration name, kernel name, parent interface, and
VLAN identifier for every vlan that is presently in use on the appliance.

DataPower_70_Networking.ppt Page 35 of 40
There are a number of command line syntax changes. These changes bring the network
interface object syntax in line with each other, and they bring the form of the commands in
line with what is most common throughout the rest of the DataPower CLI – this mainly
means the new commands have hyphens instead of spaces, so “ip address” is “ip-
address”. Remember that all old forms are still accepted, but they are deprecated.
What used to be called 'Interface' is now 'Ethernet'. There are three different kinds of
directly configurable interfaces. Additionally, before version 7 when a command was
issued on the interface object, it took affect immediately. This is no longer the case in
version 7, when the complete configuration is applied whenever there is an 'exit' from
'Ethernet'.
The old 'vlan-sub-interface' command is now called simply 'vlan'.

DataPower_70_Networking.ppt Page 36 of 40
As we discussed, command line changes to Ethernet interfaces no longer take place
immediately, they happen on 'exit' or 'Apply'. If an operations team depended on the
immediate nature of this command, they would be impacted.
As the objects change, so does the SOMA schema. Differences are minor, but please
verify any SOMA integration with the network objects.
Network interfaces are now 'up' if they are successfully configured. Link state no longer
influences the object status.
Network interfaces are operationally 'down' if they are any configuration errors. They are
still 'down' even if they have a limited, best-effort interface configuration with errors, even
though that interface may be able to be used for some traffic.
It is much less likely than previous releases that invalid configuration will go unnoticed. It
is possible that a persisted configuration may lead to the limited capability mode if the
configuration contains errors or inconsistencies.
Lastly, in releases prior to v7 when a link carrier is lost the routes on that interface were
removed. This is no longer done. Now the routes on an interface are configured and stay
until the configuration is changed.

DataPower_70_Networking.ppt Page 37 of 40
There are no new steps required for network problem determination in DataPower version
7. The same principles that applied before v7 still apply. If contacting IBM support, collect
an error report and internal state. Have a high level description of the problem, packet
captures as appropriate, and iterate in small steps to focus the problem statement.
There are enhancements to the internal state that will help IBM support focus more quickly
on any problems that are identified. As a customer, the best first step if you have a
network error is to collect an internal state and error report.
Additionally, because any network configuration problems are identified much sooner, we
believe that it is more likely that problems can be identified earlier in the development to
deployment cycle. There is a cost to this, though. Errors that were ignored previously are
now not nearly as ignorable.

DataPower_70_Networking.ppt Page 38 of 40
In closing, deploying DataPower into a datacenter is an integration problem. Just as
DataPower integrates with LDAP or MQ, so must DataPower integrate into the target
network environment. The best way to do that is to involve the network team early in the
process. Thank you.

DataPower_70_Networking.ppt Page 39 of 40
DataPower_70_Networking.ppt Page 40 of 40