I will be presenting at the Cisco Connect Canada tour in Edmonton and Calgary on November 3rd and 5th, respectively. My presentation is about that three letter acronym that everyone loves to hate: SDN :-)

I will talk about SDN in general terms and describe what it really means; what we're really doing in the network when we say that it's "software defined". No unicorns or fairy tales here, just engineering.

Next I'll talk about three areas where Cisco is introducing programmability into its data center solutions:

  • Application Centric Infrastructure
  • Virtual Topology System
  • Open NX-OS

Below are the notes I made for myself while researching these topics and preparing for the presentation. At the bottom of this post is a Q&A section with some frequently asked questions.

Cisco Virtual Topology System (VTS)⌗

VTS is primarily an overlay provisioning and management system

  • VXLAN-based fabrics
  • Support integration with virtual and physical workloads, physical and virtual switches, and DCI (ASR 9k; MPLS L3 VPN)
  • Leverages NX-API to program the Nexus switching elements
  • RESTful northbound API for 3rd-party/software integration
    • Including virtual machine manager
  • Web UI for non-programmatic interfacing
  • Designed to work across the Nexus portfolio (3k, 5k, 7k, 9k)
  • 3-layer model:
    • Orchestration plane (virtual machine managers, cloud orchestration)
    • Control plane (VTS)
    • Data plane(switching elements, virtual switches, VTF)
    • In between each layer are APIs to allow inter-layer communication

VTS deployment

  • Available as an OVA
    • Policy plane is one OVA (topology database, infrastructure policy, resource management).
    • Control plane is another OVA
      • MP-BGP federation
  • Supports Nexus 3100, 5600, 7000, 9000 families and ASR 9000 series for DCI
    • Only 9000 and 7000 supported as border leaf or border spine
    • Only Nexus 7000 and ASR 9000 supported for DCI
  • Supports ESXi and KVM hypervisors

VTS Functionality

  • Topology discovery via LLDP
    • Manual topology design is also possible
  • VTS receives notification when a VM moves to a new host
    • VTS re-provisions the overlay to account for the VM being on a new host

Virtual Topology Forwarder (VTF)

  • A lightweight software agent that runs on a hypervisor host to run the data plane function within the virtual network
  • VTF is really a software VTEP
  • Runs as a VM or a host process
  • Uses Cisco Vector Packet Processing for line-rate throughput
    • VTF requires an Intel Data Path Development Kit (DPDK) compatible NIC
  • Multi-tenant aware
  • Rich L3 capabilities

Licensing

  • Perpetual license for VTS controller software
  • Right To Use (which is perpetual) license per top-of-rack switch or per top-of-rack port

Open NX-OS⌗

  • It's not just about speeds and feeds any more; it's about integration with customer's tool sets and with 3rd-party software
    • Making NXOS more open, extensible and easier to integrate (into your environment; with other software/other tools)
  • Opex makes up up to 75% of the costs in the data center
    • How do we address opex? Make the network simpler; allow customers to manage and provision their networks in the way(s) that makes the most sense to them.
  • Looking at "openness" across the whole Nexus OS stack; from bootup to provisioning to management

Packages as RPMs

  • Custom packages
    • Package build toolchain available from Cisco (DevNet)
    • Install on any Linux machine to do building & packaging of your RPMs
  • Packages from Cisco (eg: Chef, Puppet agents)
  • Also, software patches from Cisco for Nexus OS will leverage RPM packages
    • Eg: "yum install bgp_patch.rpm"
  • Install in:
    • Natively in NX-OS underlying Linux
    • In isolation, within a Secure Guestshell Environment
      • CentOS 7-based environment
      • A Linux container running on the switch

Bash shell

  • Now with ability to use familiar Linux tools to manage front panel ports (ifconfig, tcpdump, ip route, so on)
  • Execute NX-OS commands from bash and use scripting tools to parse (grep, awk, sed, etc)
    • vsh -c "show ..." | grep
  • Programmatic shell access to configure front panel ports
  • VRFs in NX-OS exposed as Linux "name spaces"

Access to Broadcom Trident 2 shell on the ASIC

  • For the really hardcore: issue commands directly to the ASIC

Nexus API and data model for programmatic access to NX-OS

Two API models:

  • Sending CLI commands in JSON format
    • Side note: This is what the Puppet/Chef agents use today to program the box
    • Results returned as either plain ASCII (like what you would see on the actual CLI) or as JSON or XML
  • Full object model (like Cisco UCS and ACI)
    • RESTful API based on objects
    • NX-OS 7.0(3)I2(1) (Camden) and later
    • RESTful API will evolve in the coming releases; not all of NX-OS features and constructs are exposed yet
  • Developer Sandbox on the switch
    • Web UI for learning proper JSON syntax
    • Enter a CLI command into the web UI and post it to the switch; UI returns the corresponding JSON syntax and will also generate the proper Python code to execute the request

iPXE

  • iPXE is an enhanced version of PXE that supports more protocols for image download than just tftp
  • iPXE allows use existing server mgmt infrastructure to do Power on Auto Provisioning (POAP) of a Nexus switch
    • POAP allows bring-up of a network element using post-boot Python scripts and auto config download (eg, to install a Puppet/Chef agent)
  • DHCP -> iPXE -> TFTP/HTTP -> pull down image -> validate image chksum -> boot the image
  • CLI option to control boot order: flash or pxe

Q&A⌗

What NX-OS version(s) have these "Open NX-OS" features?

NX-OS 7.0(3)I2(1), code-named Camden, released in Sept 2015

NOTE: 7.0(3)I2(1) has been deferred due to CSCuw65317. Please look for 7.0(3)I2(1a) instead.

Is "Open" NX-OS available on N2k/5k/7k/9k? Some? All?

The features described above are available on the Nexus 3000 and 9000 series at the time of this writing.

Is NX-API available on N5k/N7k?

  • Nexus 7000 starting in 7.2(0)D1(1)
  • Nexus 5000 starting in 7.2(0)N1(1)

The version of NX-API in these releases allow CLI commands to be sent to the switch (JSON formatted) and for the output to be received back in either plain ASCII or JSON or XML.

Is the NX-API RESTful on N5k/N7k?

The NX-API is only RESTful (as of this writing) on the n9k in NX-OS 7.0(3)I2(1)

The RESTful API exposes NX-OS configuration elements as objects against which create/update/delete operations can be performed. Eg: you can instruct the switch to create a new BGP neighbor by instantiating a "BGP neighbor" object and assigning it properties of "remote IP", "remote ASN" and so on. NX-OS will then take this object and translate it into the running-config.

How do I access the bash shell?

From NX-OS CLI: run bash

If I run tcpdump in the bash shell, can that capture data plane traffic?

tcpdump will only see packets that are punted to the supervisor (control plane packets or packets that CEF is punting)

If I install a third-party RPM on the switch, how do I sandbox that software so it doesn't interfere with or compromise NX-OS?

One of the features of the Bash shell is the ability to create Linux-based containers (referred to as the Secure Guest Shell). A secure guest shell is logically partitioned from the "main" shell and from NX-OS. It has its own CPU/memory resources and its own root file system.

Where can I find more details about these open interfaces to NX-OS?

The Nexus 9000 Programmability Guide

What platforms and NX-OS versions support VXLAN with BGP EVPN as the control plane?

  • Nexus 9000 [7.0(3)I1(1)]
  • Nexus 7000 [7.2(0)D1(1)]
  • Nexus 5600 [It's a roadmap item as of this writing]

Disclaimer: The opinions and information expressed in this blog article are my own and not necessarily those of Cisco Systems.