SONA: ONOS SDN Controller based

OpenStack/Kubernetes Network Management Solution

Trellis: Multi-Purpose Leaf-Spine Fabric Solution

Feb 22, 2019

Sangho Shin

0
Introduction to SDN & ONOS

1
Software-Defined Network (SDN) (1/2)

Control 1 1 m
App App App

Operating
System
App App App
Specialized Packet For
warding Hardware Operating
System
App App App Specialized Packet For

Operating
System
App App
Data App
1 n n
warding Hardware

Specialized Packet For

warding Hardware Operating
System

Other aspects of SDN follow

Specialized Packet For
App App App warding Hardware

Operating
System
Specialized Packet For
warding Hardware

2
Software-Defined Network (SDN) (2/2)
App App App
Network Operating System (Controller)

App App App

App App App

App App App

Operating
System Operating
Operating System
Specialized Packet
System Forwarding Hardware Specialized Packet
Specialized Packet Forwarding Hardware
Forwarding Hardware
App App App

Operating
App App App System
Specialized Packet
Operating
Forwarding Hardware
System
Specialized Packet
Forwarding Hardware
3
SDN Evolution and ONF

 Non-profit, carrier and

vendor neutral And Beyond
 Provide technical shepherding,
core team
 Build community
 Many organizations supports 2017 – ON.Lab and
ONF merger
Demonstrations

Platform Deployments 2012 –Define SDN

Development research agenda
Invention for the coming
2008-2011 – SIGCOMM years
2009 – Stanford 2011 – Open Networking
2007 – Ethane 2010 – GENI started Summit, Interop
2007 – Creation 2008 – OpenFlow and grew to 20
of SDN Concept 2009 – FlowVisor, universities
Mininet, NOX 2013 – 20 more cam
2010 – Beacon puses to be added

4
ONOS Architecture (1/2)
Contains user applications
E.g., reactive forwarding, ProxyARP, Applications
segment routing, SDN-IP, etc.
Northbound
Transfer network info to app layer (policy enforcement, conflict resolution)
Provide management interface for
controlling lower layer component
Distributed Core
Contains many core features (scalability, availability, performance, persistence)
Provide distributed clustering func.
for supporting HA and scalability
Southbound
(discover, observe, program, configure)
Provide an abstracted interface for Provider Provider ...
controlling the network infrastructure
OpenFlow NetConf ...
Network protocol implementation
for managing network elements
E.g., OpenFlow, NetConf

5
ONOS Architecture (2/2)
Applications
Apps

NB Core API

Distributed Core
(state management, notifications, high-availability & scale-out)

SB Core API

Providers Providers Providers Providers

Protocols Protocols Protocols Protocols

6
ONOS Subsystems (Services)
Off-platform Apps

REST API GUI CLI

SONA Proxy ARP L2 Forwarding SDN IP / BGP DHCP ...

Application UI Extension Security Device Cfg. Discovery Network Virt. Tenant ...

Config Storage Region Driver Path Tunnel Intent Statistics

Core Cluster Leadership Mastership Topology Network Cfg. Flow Objective Group

Event Messaging Graph Device Link Host Flow Rule Packet

OSGi / Apache Karaf OpenFlow NetConf OVSDB ...

OSGi Framework On-platform Applications South Bound

Interface module
Non-networking core subsystem On-platform Application Interfaces

Networking core subsystem Off-platform Applications

7
ONOS Project at ONF

Mobile Enterprise Residential

Services Services Services
Mobile Enterprise Residential

XOS

Trellis

ONOS

VOLTHA Stratum ODTN

PON PON ROADM

OLTs OLTs Shared Cloud Infrastructure (Core)

8
ONOS Community
Partners Collaborators

9
ONOS Release History
Q1/16 Falcon Q2/17 Kingfisher
YANG Tools 2.0
Q4/14 Avocet ONS Use Cases
{A, E, M} CORD OpenFlow 1.4 support
Base Architecture Intent F/W improment
Disaggregated ROADM
Global R&E Deployment vRouter, OpenROADM support

Q2/16 Goldeneye Q3/17 Loon

Q1/15 Blackbird CPMan Apps
OpenFlow 1.5 SBI
Performance Intents using Flow Objectives
P4 DEMO support gRPC NBI support
YANG tool chain P4 runtime initial support

Q2/15 Cardinal Q3/16 Hummingbird Q4/17 Magpie

ONS Use Cases
SDN-IP RabbitMQ, Kafka Message Topo2 initial support
Packet Optical YANG NBI, SBI CODECs More switch driver
R-CORD ACTN Traffic Engineering support

Q3/15 Drake Q4/16 Ibis Q1/18 Nightingale

ONF ATRIUM BUCK Build Tool
Secure Mode ONOS Trellis Fabric enhancement ISSU initial support
VxLAN LISP SBI support, REST Client, Trellis enhancement (T3)
Device Configuration FatTree simulator P4 support enhancment

Q4/15 Emu Q1/17 Junco Q2/18 Owl

OPNFV TL1 SBI support
SONA Virtualization support Coming soon…
AARNET Regionalization support
KREONET-S Dynamic conf. enhancement
10
Introduction to Trellis

11
 Trellis Overview
Multi-purpose leaf-spine Bare-metal hardware
fabric designed for NFV Open-source software
SDN-based (built on ONOS)

12
 Trellis Features
● Bridging with Access & Trunk VLANs (within a rack)
● Routing (inter-rack)
○ IPv4 & IPv6 Unicast routing with MPLS Segment-Routing
○ IPv4 & IPv6 Multicast routing
● Dual-homing for compute-nodes and external routers
● Multi-stage fabrics (2 layers of spines)
● vRouter - entire fabric behaves as a single router
○ BGP (v4/v6) support for external (upstream) connectivity
○ Static routes, route blackholing
○ DHCP L3 relay (IPv4/v6)
● MPLS Pseudowires
● QinQ termination
● T3 - Trellis Troubleshooting Tool
● ASIC Support
○ Broadcom Qumran, Tomahawk, Trident2 switches from EdgeCore & QCT
○ Preliminary support for Cavium Xpliant switches and P4-based Tofino switches 13
 White-Box = Bare-metal hw + Open-Source sw
Spine Switch Leaf/Spine Switch Software Stack
White Box Switch GE mgmt. to controller
EdgeCore 6712,7712 OpenFlow 1.3

Indigo OF Agent
32 x 40G/100G ports downlink to leaf switches OCP
Software OF-DPA
Trident2, Tomahawk, Qumran
(ONL,ONIE)
Leaf Switch BRCM ASIC
OCP Bare Metal Hardware
White Box Switch
OCP: Open Compute Project
EdgeCore 5712,5912
GE mgmt. ONL: Open Network Linux
QCT LY8
ONIE: Open Network Install Environment
BRCM: Broadcom Merchant Silicon ASICs
48 x 10G, 6 x 40G/100G OF-DPA: OpenFlow Datapath Abstraction
14
Fabric ASIC Pipeline* (BRCM’s OF-DPA)
* Simplified view
Why OF-DPA? Multi-
cast Ro
Abstracts underlying ASIC uting Ta
ble MPLS Phy
Enables programming of all L3 ECM
Label
Group
Port
P
flow-tables & port-groups Group MPLS
Label
Phy
Unicast Port
Group
Routing
Termin- Table L2 Interfa
Phy Ingress ACL
ce Phy
Vlan T ation M L3
Por Port Ta
able AC
Policy Group Port
Mcast
t ble
Table
Table
Group L2 Interfa
ce Phy
Group Port
MPLS
MPLS
Table
Vlan 1 L2 Por L2 Interfa Phy
Table t L2 Floo ce
Group Port
Table d
Group L2 Interfa Phy
ce
Group Port
Bridging
Table

1515
 Trellis & P4
Enhanced with P4 progra
Same set of Trellis applicat m deployment and pipeli
ions on ONOS ne configuration
Segment Routing DHCP L3 Relay vRouter Multicast SPGW-app

ONOS Cluster P4
OF-DPA driver fabric.p4 driver

Allowing new fun

ctionality on hard
OpenFlow NetConf P4Runtime gNMI ware (demo at M
WC ‘18)

Brcm Qumran Barefoot Tofino

P4
Brcm Tomahawk Cavium Xpliant
capable hardwa
Brcm Trident2 Mellanox re

16

16
Trellis @ Comcast

17 17
Introduction to SONA

18
Why SONA?
• Limitation of Neutron network
– Limited visibility of VM traffic
– Limited scalability of network node
Management Network

neutron-metadata-agent nova-compute horizon

nova-compute
neutron-DHCP-agent neutron-plugin-agent neutron-server
nova-compute
neutron-plugin-agent
neutron-L3-agent nova-scheduler
Compute Node nova-compute
neutron-plugin-agent
neutron-*plugin-agent
Compute Node keystone
neutron-plugin-agent
Compute Node
Network Node AMQP
Compute Node
Data Network nova-api

Control Node
External Network

19
SONA (Simplified Overlay Networking Architecture)
• SONA: Overlay Network Management Solution for SDDC
– ONOS based Virtual Network Management solution (support VxLAN, VLAN, FLAT)
– Empowered by SDN controller, a better replacement of neutron, scalable gateway
– Fully compatible with OpenStack (mitaka, newton, ocata, pike, queens)

Better VM - VM Traffic
visibility at Control Plane

Highly
scalable

20
SONA (Simplified Overlay Networking Architecture)
①
• Integration with OpenStack
– OpenStack neutron Neutron
ML2 Plugin ONOS
• Plugin: modular layer 2 plugin L3/LBaaS/
ML2 ONOS
– networking-onos DB Mech Driver
Plugins/
Drivers
• ONOS L3 plugin
• Drivers for LBaaS, FWaaS, etc.
② ③
– SONA
SONA Northbound
• Northbound interacts with networking-onos
– https://github.com/openstack/networking-onos
ONOS SONA Network/Rule Services
• Southbound protocol
– OpenFlow: install/uninstall flow rules OpenFlow OVSDB
– OVSDB: configure OpenvSwitch
» Add/delete virtual port
» Create/delete bridges (e.g., br-int, etc.)
VM VM

21
SONA Features

Direct communication

22
SONA Features

Scalable Gateway

23
SONA Features

UI based Flow Tracer

24
SONA Features
• SONA Fabric
– Pure OpenFlow based Leaf-Spine Fabric Solution
– Supports ECMP, Failure detection & auto recovery
– Physical + Virtual Network Integration

25
SONA Features
• vFlow Statistics
– Collect VM to VM real-time flow statistic REST Kafka gRPC influxDB …
– Stats collection is realized using OpenFlow
standards protocol (no extra overhead!) OpenstackTelemetry …

– Seamless integration with monitoring

systems through various NBIs SDN Controller
• REST, Kafka, gRPC, influxDB, etc.
– Realized through OpenstackTelemetry app
OpenFlow
– No additional software installations are
required at OpenStack side OVS
OVS
OVS

– No additional hardware installations are

required at compute/control node
– Open source!
VM VM
OVS

26
SONA Features
• vTap
OpenstackvTap …
– Mirror VM to VM real-time traffic
– Leverage OVS’s traffic mirroring feature
SDN Controller
– Two traffic mirroring schemes
• Port-based: specific to OVS
• Flow-based: uses OpenFlow group table
– Realized through OpenstackTelemetry app VM VM vDPI
– No additional software installations are
required at OpenStack side
OVS OVS
– Further improve the mirroring performance by
leveraging data plane acceleration technology
– Open source! eth0 eth0

27
SONA Failover
Crash

SONA SONA
ARP Request ARP Request

ARP Response ARP Response

Proxy Mode
VM VM VM VM

SONA SONA
ARP Request ARP Request

ARP Response
Broadcast Mode
ARP Response

VM VM VM VM

28
SONA Features
• Kubernetes Support
Kuryr-
Neutron
kubernetes

Kuryr CNI

OVS Pods

VM Container
Container
VM
VM Container
SONA Fabric
Spine

Leaf
...

Tenant A Tenant B Tenant C Tenant D

VM VM VM VM VM VM VM VM VM Baremetal
VM VM VM VM VM VM VM VM VM Baremetal
29
 How does SONA Process Packets?
• SONA Pipeline

OpenvSwitch
FLAT
(table = 2)
Ingress
Port InboundStat DHCP & ARP (vNet)
(table = 0) (table = 1)

vTAG ACL
(table = 10) (table = 20)

ConnTrack
(table = 21)

Egress
Port Switching OutboundStat Routing Jump
(table = 50) (table = 49) (table = 40) (table = 30)

30
SONA CI/CD
• Continuous Integration (CI)
– Fetch latest SONA source Jenkins
Machine
– Build against stable ONOS
– Run unit test
– Package & deploy SONA ONOS Build & OpenStack Tempest
Run Machine Control Node Machine
– Run integration test
– Notify the CI result via slack
– Deliver SONA container Gateway OpenStack … OpenStack
Node Compute Node Compute Node

https://hub.docker.com/r/opensona/onos-sona-nightly-docker/
31
SONA CI/CD
• Integration Test
– Initialize environment to spawn ONOS cluster
– Tempest basic test
• OpenStack API test
• OpenStack scenario test
– ONOS failure test
• Terminate ONOS nodes, run tempest
– SONA app failure test

32
Open Source Contribution
• Open Source Strategy
– 100% open source
– 136 commits were upstreamed in 2018 (2018.01 ~ now)
• https://gerrit.onosproject.org/#/q/project:onos+branch:master+topic:sona
• Helps from Community
– More tests and feedback from community
– Code contributions are always welcomed :)
• Wiki
– https://wiki.onosproject.org/display/ONOS/SONA%3A+DC+Network+Virtualization
• Slack Channel
– #sonaproject @ onosproject.slack.com

33