My First OpenDaylight

Over the last few days I’ve started to the play with the OpenDaylight Test VM Image. This image is was easy to get up and running and have a playground with mininet and a pre-baked OpenDaylight (ODL) controller to play with. After deploying the OVA file in Virtualbox poking around the file system I got down to “business” with getting a test topology in place. I made some changes to initial mininet configuration startup file to make the topology more complex and changing the startup command to look like the following,

sudo mn --controller 'remote,ip=,port=6633' --topo tree,3

This yielded a 8 hosts and 7 switches topology. At one point I have 63 hosts and some number of switches things broke pretty hard so I dialed it back a little bit. I want over to the webui for the controller and after some fiddling Names and Tiers on the switches. My test topology in the ODL console is show in the following screenshot.

ODL Home

I also had full reachability from all of the mininet hosts.

mininet> pingall
*** Ping: testing ping reachability
h1 > h2 h3 h4 h5 h6 h7 h8
h2 > h1 h3 h4 h5 h6 h7 h8
h3 > h1 h2 h4 h5 h6 h7 h8
h4 > h1 h2 h3 h5 h6 h7 h8
h5 > h1 h2 h3 h4 h6 h7 h8
h6 > h1 h2 h3 h4 h5 h7 h8
h7 > h1 h2 h3 h4 h5 h6 h8
h8 > h1 h2 h3 h4 h5 h6 h7
*** Results: 0% dropped (56/56 received)

Now that I had things working it was time to find ways to break it. Diving into the flow rules I threw together a basic Drop rule on one of the transit links.

Flow Rule Split Network

As expected the network was split into two.

mininet> pingall
*** Ping: testing ping reachability
h1 > h2 h3 h4 X X X X
h2 > h1 h3 h4 X X X X
h3 > h1 h2 h4 X X X X
h4 > h1 h2 h3 X X X X
h5 > X X X X h6 h7 h8
h6 > X X X X h5 h7 h8
h7 > X X X X h5 h6 h8
h8 > X X X X h5 h6 h7
*** Results: 57% dropped (24/56 received)

Lets see about black holing a single host now.

Drop H1 This drops all traffic from the host connected to port 1 on the switch which happens to be h1

mininet> pingall
*** Ping: testing ping reachability
h1 > X X X X X X X
h2 > X h3 h4 h5 h6 h7 h8
h3 > X h2 h4 h5 h6 h7 h8
h4 > X h2 h3 h5 h6 h7 h8
h5 > X h2 h3 h4 h6 h7 h8
h6 > X h2 h3 h4 h5 h7 h8
h7 > X h2 h3 h4 h5 h6 h8
h8 > X h2 h3 h4 h5 h6 h7
*** Results: 25% dropped (42/56 received)

OpenDaylight has always peaked my interested, I’ve been trying to follow the mailing lists and some of the discussions out there and the Test VM is a nice way to start to get under the hood. I have a lot more to learn and there are a ton of other plugins to start to explore. Not to mention to start to think about the API and writing some code against it.


  1. If you do not set switch roles properly end hosts my not show up on the topology.

  2. Flow rule names can not have spaces in them.

  3. The controller had the Access switches properly classified in the Tier however the transit switches were not set to either Distribution or Core.

IBM PURE systems networking

To start off I’ll cut past some of the marketing and state that PURE Systems are IBM BladeCenters with some predefined hardware configurations that support both x86 and POWER work loads.

With that being said the advantage to the PURE architecture is the software that IBM has assembled to orchestrate deployments of workloads across all of the integrated platforms. The orchestrator is named Flex System Manager (FSM). The FSM plugs into VMWare for x86, HMC for Power systems and other management system for virtualization platforms. The FSM will use these connections to automate deployment of systems and monitoring of the hardware, physical and virtual systems within the PURE System.

There are many details about the hardware I will not cover but one of the details IBM discusses is the increased speeds and feeds. This is accomplished by interconnections between the Nodes and the I/O Bays, each Node has multiple connection to the I/O Bays. The number of paths grow or shrink by the number of licenses, or as IBM says Pay as you Grow.

IBM Blade to IOM connectivity


Image copied from (

The portfolio of IO Modules is similar to any BladeCenter you may have seen in the past, with options for in Network Switches (BNT Switches, some supporting OpenFlow 1.0), Fiber Channel switches and passthrough modules (All the options can be found here:

Where I see the need for great improvement is the POWER Series networking. POWER utilizes a Virtual IO Server (VIOS) to connect the LPARs to each other and the outside world. Essentially the VIOS is a AIX server that acts as a layer 2 bridge. The VIOS lacks the ability most network switches have had to do private VLAN configurations and layer 3 inspection. There also currently is no support at this time for next generation such as OpenFlow, IBM DOVE, or VXLAN.

IBM PURE Block Diagram


This brings many complications in a multi-hypervisor environment. For example locating an IBM LPAR next to a VMWare workload you will need glue it together with VLANs and legacy networking. This will require networking teams maintain network controls separately from how you may treat the rest of your virtualized work loads on the VMWare platform.

Even though I have a bit of a beef with the VIOS, the PURE system is a good approach for IBM shops to consolidate their workloads into a single Private Cloud style configuration.

How does Riverbed Steelhead Auto Discovery work?

Riverbed Steelhead devices have a method they use to find each other on the network. Riverbed has named this Enhanced Auto Discovery. This is intended to reduce time to deployment and simplify the configuration on the devices. The core of this method uses setting Options in the TCP headers within the initial 3 way handshake. There are a few concepts to go over to fully understand the process of Steelhead Auto Discovery.

The Steelhead is a Layer 2 Bridge

Every Riverbed is a layer 2 bridge, for traffic to enter the optimization engine it must be bridged through the Steelhead. In the appliance there are 2 interfaces that make of the bridge, they are named the LAN and WAN interfaces.  The LAN interface connects the network where the client machines, or server machines are located. The WAN interfaces connects to the external network where the router for the VPN, MPLS, P2P, 3G Radio, or whatever medium may be in use resides.

These interfaces together are called the IN PATH interfaces. In a Cisco device that is doing IRB style bridging the IN PATH interface would be similar to a BVI interface. The IN PATH interface is required to have an IP address assigned to it, this is the IP Steelheads use to communicate to each other on. For example Inner Channel is negotiated between IP addresses of IN PATH interfaces, this is one reason that IP reachability between the IN PATH IP addresses is important.

Clients and Servers

There are a couple of designations that help to clarity which devices initiate which connections. The Client Steelhead (CSH) is the Steelhead that receives the first Naked SYN from the client machine initiating the connection. A Server Steelhead (SSH) is a Steelhead that receives the SYN+ from a Client Steelhead and is the Steelhead closest to the destination server.

The Channels

Channels denote areas where specific devices communicate with each other, and  where traffic is optimized or not optimized. There are 2 separate Outer Channels and a single Inner Channel.

Screen Shot 2013-08-18 at 10.38.49 PM

Outer Channel (local) – This channel is between the Steelhead LAN interface and the client machine sourcing of the Naked SYN. This may be a branch workstation for example.

Outer Channel (Remote) – This is the channel between the Steelhead LAN interface and the server machine that the initial SYN was intended to be received by. This may be some sort of application or file server at a data center for example.

Inner Channel – This is the network between the WAN interfaces of the Steelheads. This is a TCP connection between the IN PATH IP address where all optimized traffic between Steelheads passes.  The default for used for this connection is TCP 7800.

 The Process

Now on to the steps used for the Steelhead devices to find each other. There are 7 main steps involved, in the following example you are initiating a TCP connection from a Client workstation to a Server to copy a file.

Screen Shot 2013-08-18 at 10.54.42 PM


  1. The client machine sends a Naked SYN packet, this is a SYN with out the PROBE TCP option set. This SYN packet is received on the LAN interface of the CSH and is intercepted by the CSH. At this stage the Steelhead checks licenses to make sure this traffic is entitled, if it is not the traffic is just passed through unchanged. If all is well the CSH will add the TCP header Option number 76. This option header is named AUTO-DISCOVERY PROBE and will include such information as the IN PATH IP address and what role this Steelhead is assuming. A packet with an TCP option is noted by a ‘+’ in documentation, for example SYN+. After the header is set the packet is forwarded out the WAN interface.
  2. This SYN+ is received by the Steelhead that will become the SSH (in this example) on the WAN interface. Again at this stage licenses on this Steelhead are checked to make sure this traffic is entitled, if not the traffic is just passed through untouched. If the SYN does not for the option headers set it is also passed through untouched.
  3. Since this packets is a SYN+ the Steelhead then sends a SYN/ACK+ with FWD Negotiation option back towards Client machine. This SYN/ACK+ is then intercepted by the CSH on the return path.
  4. The Steelhead near the Server machine starts a negotiates 3 way handshake with Server machine. If this Steelhead does not encounter another Steelhead between itself and the Server machine it will assume the role of SSH and complete an 3 way handshake with the Server machine. If a Steelhead is encountered this process repeats down the line towards the server.
  5. The newly declared SSH will then send a SYN/ACK+ with PROBE RESPONSE option in headers towards CSH.
  6. The SYN/ACK+ is intercepted by the CSH and the CSH initiates the Inner channel between the CSH to the SSH.
  7. Once the Inner Channel is formed the CSH completes the 3 way between itself and the Client machine.

Now that the above steps are completed the traffic between the Client machines and Server machines is on its way and the Steelhead is transparently (to the client and server) doing its optimization magic. All of the traffic in this flow, after it has been optimized is transmitted, over the Inner Channel. If a new traffic flow needs to communicate between the Client and Server the process starts again for that new traffic flow and this is the case for every TCP connection that occurs from site to site.

Where Trouble can Occur

There are a few basic items that can easily stop this process from occurring, therefor blocking optimization from occurring.

  • If there is something strips the Option headers out of TCP packets such as a firewall or IDS.
  • The IN PATH interfaces do not have Layer 3 reachability between each other. This will prevent the Inner Channel from forming.
  • The traffic does not follow from the LAN to WAN and WAN to LAN interfaces on both the SSH and CSH.
  • The boxes are improperly licensed or the amount of traffic/tcp session is exceeding the license

Over all this is a pretty simple process and not to different from how other vendors handle things such as Cisco WAAS. Even though they use technologies such as WCCPv2. There are other scenarios that  such as Virtual In Path and Server Side Out of Path that are a bit different, but this is the Riverbed recommended way of doing things.

8static 34 – April 2013 Photos

So I have some catching up to do, so here are some photos from April!

8static 34
April 13th, 2013 7:00pm

Br1ght Pr1mate (BOS)
Note! (NYC)
Dauragon (DC)
Environmental Sound Collapse (CHI)

Environmental Sound Collapse (CHI)

Animal Style on soldering for modding / circuit bending


8static - Dauragon
8static - Br1ght Pr1mates
Br1ght Pr1mates
8static - Note!


Cisco Live 2013 – My (late) wrap up.

This past month I attended Cisco Live in Orlando, FL with 20,000(?) of my fellow Network/Collaboration/Service Provider/Data Center engineers from all around the world. This was my first time attending, and I had a blast! There are a few themes I won’t talk much about in this post that were big topics at Cisco Live one of which is the Internet of Everything (IoE) as that is well covered, and well is really just Market-ecture-tastic. New gear like the Catalyst 6800 or Nexus 7700 and new ASICs all of which are neat, powerful, and that will enable a lot of the future technologies, but Better, Faster, Stronger hardware comes all the time. In the end SDN/Network Virtualization for me was the most discussed topic in through all of the Network-centric sessions and “hallway” conversations throughout the entire week.

CLUS 2013 Schedule I was able to attended many great sessions, but even with a packed schedule I still wanted to be in two places at once most of the time. There were a few standout sessions including “BRKRST-3114 The Art of Network Architecture” and “BRKRST-3045 LISP – A Next Generation Networking Architecture”. “The Art of Network Architecture” was a very business forward discussion of network architecture, and I believe attempted to change the discussion around designing a network. Wheras “LISP – A Next Generation Networking Architecture” got me excited about LISP in a way that I had not been before. All the previous information I had read about LISP left me wanting for a tangible use case. This presentation at CLUS started to describe some good use cases for LISP, I am still left wanting for more wide spread production implementations.

CLUS Tweetup

Another great event I attended was the Tweetup organized by Tom Hollingsworth. I met a lot of people I follow on Twitter there, and it was nice to put a face with a Twitter handle and have some good conversations about networking and well just about anything else.


When listening to the discussions and presentations a few trends and themes struck me. First, there is a trend towards the flat network; when I look at the fabric technologies or the affinity networking coming out of Plexxi or potentially Insieme, this all puts a large exclamation point at the end of the need to move to IPv6 or at least implement dual stack sooner rather than later. It will be key in the success of these technologies in the data center. Next there was a constant argument going on about the death of the CLI and that the GUI will reign supreme. I believe both the CLI users and the GUI user can be accommodated, both types of interfaces can be used to manipulate some back end software and logic. An example of this is tail-f NCS which has both, while not “SDN” by some definitions, but an example of the 2 UIs co-existing. The real augment that needs to be had concerns the designs of the system needed to support the applications.

This one is more a rant and less of a theme, but I still think Cisco is missing the mark with the ASA 1000v. I think virtualized physical appliances are a transitional technology, but a needed one. Creating the ASA 1000v and not giving it the full set of features of it’s physical counterpart without a roadmap as far as I can tell to add them, along with the insane licensing scheme of a per-socket protected model does not make sense to me. This is all short changing the IaaS provider market and IMHO it should be licensed and operated similar to the CSR 1000v, full features and per appliance licensing.

Overall, I was left with two general questions from the week. First, I’m curious how the balance of systemic complexity vs configuration complexity vs structure complexity will fall as the overlay, “underlay” and the SDN glue that holds it all together sets into place. Each new technology that is introduced seems to address one of these complexity problems but not all three in one fell swoop, but this is a larger topic for another post. Second is a reoccurring theme in technology: everything old is new again; I look at the data center technologies, and some of the new IP routing technologies (LISP), and they look a lot like old telephony switching technologies, in the same way VDI looks like mainframe dumb terminals. This is not a critique, just an observation on how it’s important to know your past because it will come back Better, Faster, Stronger, or maybe just the same with a new box around it.


Cheap Dinosaurs Play Goblin Photos


This show is from last year, i’m just finally getting the images edited and posted

Cheap Dinosaurs play Goblin
Friday, October 12th, 2012
7:00pm at PhilaMOCA – All Ages!

Cheap Dinosaurs (PHL)

Tom Guycot (PHL)
The Joint Chiefs of Math (PHL)

Full set of images:

The Joint Chiefs of Math
The Joint Chiefs of Math
Tom Guycot
Tom Guycot
Cheap Dinosaurs
Cheap Dinosaurs

How I started and learned (Hello World!)

I think it makes sense to kick off this blog with a how did I get to where I’m at today. This post covers about 16 or so years (as of posting) of working in IT and Computers and well many years before them as a well….hobbyist.

My first introduction to communications, I guess you could say networking, was calling BBSs and eventually installing and playing with BBS software. I learned a lot about modems and dialup communications. Towards the end of this era (to learn more about this time in technology I highly recommend watching the BBS Documentary that was released by Jason Scott in 2005.) I leaned about 2600 Magazine and went to a 2600 Meeting in my area and starting to get interested in the Phone System and how the Phone Switching Systems work.

Then came along the internet, and I started to learn about programing in some classes in High School and on my own, eventually learning C Sockets and coding some basic Client/Server programs and showing me how applications worked across a network. At this time I didn’t know anything about the 7 layer model or another really about networking theory, but I do know now this knowledge has shaped how I have always approached troubleshooting.

Eventually I decided it was time to get a job working on computers and  I found a job working for an outsourced call center for a major home computer manufacturer. This was a very valuable job in my career, here I learned one of the toughest skills to learn, customer service. There were many difficult customers, for example, “where the Start Button?”, Sorry your hard drive is clicking and your data is gone, ‘Yes, click the mouse button in the right” are just some examples. I learned to handle and lead non-expert users, and resolve there issues to the best of my ability. Doing it all with a smile in my voice no matter how frustrating they were.

Eventually that job ended (ahem laid off) and I moved into a job and small computer integration company (see Thoughts on Working as a Consultant for a VAR and THE VAR-Y GOOD UPSIDES TO BEING A CONSULTANT! both are true in a lot of ways about a 13+ years ago as they are today). On my first day I walked to a stack of Novell NetWare books. So I learned NetWare, then Windows NT, Windows 2000, the desktop OS of the month, how to run cable (badly), and just about anything you could imagine in a small company would use at that time. This made me a  very well-rounded technician but not too deep any topic. I did always have a feeling I had an interest in the networking, so I made every opportunity I could to learn how to program routers and learn networks.

After enough time I studied and got my CCNA at the same time as my boss/owner of the company at the time. This is when I fell off the deep end and started to become a specialist in networking and connectivity, learning about T1s, factional T1s, 56k circuits  working with phone companies, still running cabling, routers, switches and still doing all the server work when needed. Then the craze of IP Telephony hit and we started to install Cisco CallManager and Unity. Learning more about PBXs, call routing, dial plans, and auto attendants then I’d ever cared too. Some where in this time period I’d say i could be calling a Network Engineer and moved up form just being a Technician.

After long enough I decided it was time to get my CCIE, then I procrastinated a while and then I actually did my Written exam and then Lab exam. It was a lot of work but I got it done with a stack of Cisco 2600s, and a lot of late nights. Not long after this my work role changed a lot moving into a role as an IT Director, where I still did my fair share of engineering, but also dived into security and operational policy for internal IT along with our virtual co-location data center (what we would call IaaS now) that we built from the ground up which was very fun. This more or less leads me to today where after more role changes I now focus on Network Architecture for on site customer networks as a Managed Service Provider and the network for our Cloud Service we provide. Working more in diagrams, word, excel and documenting now then working in actual commands lines.

Not to wax poetic too much but it was been a long winding road, and all the change along the way has kept me on my toes and kept it interesting.

8static 2F – November 2012 Photos

8static 2F
November 10th, 2012
at PhilaMOCA

Full Set a Flickr:

Animal Style (PHL)
exileFaker (NYC)
Dream Fox (STL)

noteNdo (NYC)

SKGB – No-Input Mixing and Creative Use of Distortion

Dream Fox



Animal Style

8static 2E – October 2012

8static 2E
October 13th, 2012
4th Anniversary Show!!

My full Flickr Set:

Bit Shifter (NYC)
Radlib (CT)
an0va (PHL)
Nikola Whallon (DET)

Chromacle (PHL)
Animal Style (PHL)

8static Alumni Showcase

One hour of music by 12 veteran 8static performers plus surprises!!
animal style – cheap dinosaurs
AdamGetsAwesome – kris keyser
bubblyfish – ro-bear
br1ght pr1mate – chipocrite
alex mauer – inverse phase
saint – exilefaker
void vision – decktonic

Nikola Whallon




Bit Shifter