Networking on Ben's ideas and projects

How to set up a Meraki API Test environment

locutus@the-collective.net (Ben Mason) — Wed, 19 Jun 2019 19:18:14 -0400

I needed to set up and Meraki API key to test, well an Meraki API that was in beta. This is the process I used to get started with some of the basics of the Meraki API and getting a test environment up and running. There are lots of great references covering the basics of REST APIs like the REST API Tutorial. These resources will do a much better job then I can of explaining REST APIs. I found there was a lack of guide for the initial steps of building the data you need to get started with the Meraki API.

Note: The screenshots are from late 2018 and may have changed over time.

API Key Generation

First things first you will need to login to the Meraki Dashboard. Once there, you will navigate using the menu on the left to Organization -> Settings.

On the settings screen scroll down to Dashboard API Access, and check “Enable access to the Meraki Dashboard API” and click Save at the bottom. Once the general access is enabled you will need to click the “profile” link to go to the screen where you generate an API Key to use than making REST API calls.

On the API access click the “Generate new API Key” button. If the button is not there I found with my account I can only have a maximum of two API keys generated at any point in time. Once I deleted one key the button came back.

After clicking the button a dialogue similar to this showing you your new key, this key is only shown once so make a note of it since you will use it to authenticate your API calls.

Now that you have a key what to do with it?

Meraki has an extensive API with many calls and you will want a tool to start to test some of the calls. A good utility to start testing with is Postman. This tool allows you to make REST API calls using a convenient GUI. I won’t go into complete detail on how to use Postman but cover some highlights to getting it setup to test some Meraki API calls.

A useful feature of Postman is the ability to import collections of API calls. The collection of Meraki Dashboard calls is at https://create.meraki.io/postman. Once there click “Run in Postman” in the upper right and it will ask to open the Postman client. Once you import the collection there will need to be some variables you will need to discover and fill in:

X-Cisco-Meraki-API-Key
organizationId
networkId
baseUrl

To set these variables you will need to edit the newly imported Postman collection, you can right click on the collection and select “Edit.”

Then select the Variables tab, I have populated these variables already in the screenshot, you will need to type them in.

![](../2019-06-19-how-to-set-up-a-meraki-api-test-environment-images/Screen-Shot-2018-10-29-at-21.27.59.png) Now you ask where do I find the values for these variables. I’ll cover the calls that are made to collect the values you need in the next few sections.

Meraki API URL (baseUrl) and API Key

baseURL
The first variable you will set it the baseUrl this is the URL that Postman will use to send REST API calls to. In general for testing you can the use URL:

<pre class="wp-block-preformatted">https://dashboard.meraki.com/api/v0

This will work for testing and non-production. Once you go to production you will want to point to the specific shard you are hosted on such as:

<pre class="wp-block-preformatted">https://n466.meraki.com/api/v0

X-Cisco-Meraki-API-Key
We will also need to set the API key which we generated earlier. This is stored in the X-Cisco-Meraki-API-Key variable. This variable sets the header also named X-Cisco-Meraki-API-Key in REST calls. This is used to authenticate the REST calls.

With these two variables set you can start to discover the organizationId and networkIds.

Finding the “organizationId”

To find the organizationId, in Postman navigate to “Organizations -> List organizations this user has access to” in the sidebar on the left.

The query in Postman looks like:

The full REST URL to retrieve the Meraki Organizations you have access to is: https://dashboard.meraki.com/api/v0/organizations
The data returned shows the organizations you have access to. The “id” number is the field used to select the organization you wish to query

<pre class="wp-block-preformatted">[ { "id": 1234567, "name": "Organization name" }

Finding the “networkIds”

Many calls I have worked will use either the organizationID ora networkId. In most organizations, there are multiple networks in the organization you are querying. Each of the networks is identified by the networkId.

The full REST URL to retrieve the Meraki Networks in an Organization you have access to is: https://dashboard.meraki.com/api/v0/organizations/1234567/networks

The output below will list all of the networks in the Organization, the field labeled “id” is what you will use to query data for a specific network.

<pre class="wp-block-preformatted">[ <br></br>{ "id": "L_234567890", <br></br>  "organizationId": "1234567", <br></br>  "name": "Test Network", <br></br>  "timeZone": "US/Eastern", <br></br>  "tags": null, "type": <br></br>  "combined", <br></br>  "disableMyMerakiCom": false }, <br></br>{ "id": "N_678901234", <br></br>  "organizationId": "1234567", <br></br> "name": "Systems Manager", <br></br> "timeZone": "America/Detroit", <br></br> "tags": null, <br></br> "type": "systems manager" } <br></br>]

Done?

Now there is a test environment to play and learn how the various API calls work and what data can be collected, set, or deleted. Postman is just the start for experimentation and to

https://create.meraki.io/

March 2019 NX-OS Vulnerability Dump

locutus@the-collective.net (Ben Mason) — Thu, 07 Mar 2019 10:42:24 -0500

On March 6th Cisco released 29 high and medium rated PSIRT notices for NX-OS based platforms. These platforms include the Cisco Nexus 3000 – 9000 series and Nexus adjacent platforms FX-OS and UCS Fabric Interconnect platforms. Not all advisories affect all platforms but all platforms are affected by at least one high rated vulnerability. The vulnerabilities range from command and code execution, privilege escalation, denial of service, and arbitrary file read vulnerabilities. This is just about everything bad that could affect core infrastructure devices.

If you haven’t updated your switch in a while this is probably the time too. Within some of the advisories Cisco notes that they are providing free updates:

Cisco has released free software updates that address the vulnerability described in this advisory. Customers may only install and expect support for software versions and feature sets for which they have purchased a license.

I’ve included a table of the fixed in versions notes as of the writing of this post. I would recommend looking at the advisories to assist in selecting the best version as there are other code versions that have integrated the fixes.

Platform	Version
Nexus 1000v	5.2(1)SM3(2.1) (Hyper-V) 5.2(1)SV3(4.1a) (VMWare)
Nexus 3000 Nexus 3500 Nexus 3600	9.2(2)
Nexus 5500, 5600, and 6000 Nexus 7000 and 7700	8.3(3)
Nexus 9000 and 9500	9.2(2)
UCS 6200 and 6300 Series Fabric Interconnects UCS 6400 Series Fabric Interconnects	4.0(2a)

Cisco has a bundled advisory for all of the high rated notices at the following link,

Cisco Event Response: March 2019 Cisco FXOS and NX-OS Software Security Advisory Bundled Publication

I have also included a laundry list of notices including both high and medium rated vulnerabilities for your reference.

Happy patching!

Vyatta 5400 and interface inbound discards

locutus@the-collective.net (Ben Mason) — Tue, 16 Dec 2014 10:30:02 -0500

Recently I was investigating alerts that were being generated for inbound interface discards on multiple interfaces and multiple Vyatta 5400 devices. There were not any noticeable performance issues on traffic passing through the devices. The discards would report in SNMP, show interface ethernet ethX, and ifconfig outputs. An example show interface ethernet ethX output I was reviewing is below.

vyatta@FW01:~$ sh int ethernet eth0
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:50:56:0x:0x:0x brd ff:ff:ff:ff:ff:ff
inet 172.x.x.x/24 brd 172.x.x.x scope global eth0
inet6 fe80::250:56ff:0x:0x/64 scope link
valid_lft forever preferred_lft forever
Last clear: Wed Oct 29 10:55:13 GMT 2014
Description: MGMT
RX: bytes packets errors dropped overrun mcast
   242863    3664      0     163       0     0
TX: bytes packets errors dropped carrier collisions
   128065     701      0       0       0          0

I was not finding any other statistics that would match up with the quantity of discards being reported. Here are a few of the commands I looked at to look for matching discard counters.

vyatta@FW01:~$ sh int ethernet eth0 queue
vyatta@FW01:~$ sh int ethernet eth0 statistics
vyatta@FW01:~$ sh queueing
vyatta@FW01:~$ sudo netstat -s

While researching where to go next I was reminded that the Vyatta 5400 is at it’s heart a Linux device server. I found a few references that beginning in the Linux kernel version 2.6.36 there were more error conditions added to this counter in the kernel.

The rx_dropped counter shows statistics for dropped frames because of: (Beginning with kernel 2.6.37)
(http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=caf586e5f23cebb2a68cbaf288d59dbbf2d74052)
Softnet backlog full — (Measured from /proc/net/softnet_stat)
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6
If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

via http://www.novell.com/support/kb/doc.php?id=7007165

When taking a look I found that the version of Vyatta code in use contains the Linux kernel version 3.3.8. The only way to verify if these conditions are causing the counter to increment is to put the interface into promiscuous mode. Since this was a production system I instead looked for neighboring Linux systems in the same subnet, and found they do not report the same level of discards. It appears I found my the reason behind this counter incrementing. This issue looked more urgent as we measure this counter in percentage of packets discarded and this interface does not have much traffic flowing through it. This made the percentages very high which the discarded frames where non-production impacting frames. This issue was a reminder that it is good to remember the underlying Operating System even if it is masked by a custom CLI.

A meditation on the interface discard counter

locutus@the-collective.net (Ben Mason) — Mon, 08 Dec 2014 10:15:18 -0500

I find the interface discard counter a deceptively complex counter. When you ask people what the counter means the usual answer is that you are over running the throughput capability of an interface. Which matched pretty closely to the definition in the IF-MIB SNMP MIB.

The number of inbound packets which were chosen
to be discarded even though no errors had been
detected to prevent their being deliverable to a
higher-layer protocol.  One possible reason for
discarding such a packet could be to free up
buffer space.

ifInDiscards : https://www.ietf.org/rfc/rfc1213.txt

The description from the MIB is often the cause of this counter incrementing, however as devices get more powerful and circuits keep increasing in size, this description is becoming less applicable. There are many other issues that have been lumped into this counter, all of these other issues are vendor, platform, and configuration dependent. Some examples I have found are,

ASA Dispatch Unit CPU over utilization
ASA ACL/ASP packet drops
QoS on an IOS interface can cause an elevated (purposeful) number of frames dropped
An ASIC shared between ports on a switch is being over utilized
L2/L3 packet handling on some Linux kernels and some virtual network platforms

Looking at this list the interface discard counter starts to look more like a check engine light for a device or interface. As with the check engine light it is important to understand all of the that data your devices are presenting, and build good baselines of the statistics for your system. Ethan Banks has some good thoughts on data baselines in a post titled The Importance of Knowing Baselines.

Nexus 7000 and a systematic Bug

locutus@the-collective.net (Ben Mason) — Mon, 20 Oct 2014 10:43:45 -0400

I have been thinking about an old issue that a customer encountered with an pair of Nexus 7000 switches about a year and half ago. When the issue first came onto my radar it was in a bad place, this customer had Nexus 2000 Fabric Extenders that would go offline and eventually the Nexus 7000 would go offline causing some single homed devices to be come in reachable, and in the process broader reachability issues. This is occurred intermittently which always causes data collection to be complicated. After working with TAC and finally collecting all of the information the the summary of the multiple causes came down the these 5 items.

1. Fabric extender link become error disabled due to CSCtz01813
2. Nexus is reloaded to recover from Fabric Extender issue.
3. After reload ports get stuck in INIT state due to CSCty8102 and failed to come online.
4. Peer Link, Peer Keep-Alive and VPCs fail to come online since ports are err-disabled from sequence time out.
5. VPC would go into a split brain state causing SVIs to go into shutdown mode.
6. Network connectivity is lost until reload of module and ports are brought online.

The summary is two bugs that would get triggered at random times causing a firestorm of confusing outages. The two temporary work arounds to mitigate the problem before we could upgrade the code on the switches was to,

VPC keep alive link to Admin port on Supervisor.
Use EEM script to reset a register when a module comes on line.

When thinking about what occurred it is important to remember the Nexus 7000 platform consists of many line cards that each contain an independent “brain” (Forwarding Engine(s) and supporting systems on the line cards) that are connected and orchestrated by the Supervisor module. It is true previous statement was a bit of a simplification, however I find it enigmatic of some of the design challenges you can on the Nexus 7000 platform. For example there are many limitations with Layer 3 routing features and VPC. In the example above it could be said that this sort of complexity can cause safety features such as those build into VPC to cause more harm then good when they encounter an in planned failure scenario. This is different from the Catalyst platform where (for the most part) everything is processed through an central processor.

Over all the Nexus 7000 system design allows for tightly coupled interactions between the modules, supervisors and even more loosely coupled interactions between chassises. These interactions can allow for the high speed and throughput that can be delivered, however is adds to the complexity of troubleshooting and complex designs. In the end what makes this issue so interesting to me and and why I keep mentally revisiting it is that it is an example of a system failure. Every single cause if occurred individually would have been as greatly problematic but their interactions together caused the observed issue to be many times worse.

Some great Nexus 7000 references

My First OpenDaylight

locutus@the-collective.net (Ben Mason) — Sun, 10 Aug 2014 16:18:07 -0400

Over the last few days I’ve started to the play with the OpenDaylight Test VM Image. This image is was easy to get up and running and have a playground with mininet and a pre-baked OpenDaylight (ODL) controller to play with. After deploying the OVA file in Virtualbox poking around the file system I got down to “business” with getting a test topology in place. I made some changes to initial mininet configuration startup file to make the topology more complex and changing the startup command to look like the following,

sudo mn --controller 'remote,ip=127.0.0.1,port=6633' --topo tree,3

This yielded a 8 hosts and 7 switches topology. At one point I have 63 hosts and some number of switches things broke pretty hard so I dialed it back a little bit. I want over to the webui for the controller and after some fiddling Names and Tiers on the switches. My test topology in the ODL console is show in the following screenshot.

I also had full reachability from all of the mininet hosts.

mininet> pingall
*** Ping: testing ping reachability
h1 > h2 h3 h4 h5 h6 h7 h8
h2 > h1 h3 h4 h5 h6 h7 h8
h3 > h1 h2 h4 h5 h6 h7 h8
h4 > h1 h2 h3 h5 h6 h7 h8
h5 > h1 h2 h3 h4 h6 h7 h8
h6 > h1 h2 h3 h4 h5 h7 h8
h7 > h1 h2 h3 h4 h5 h6 h8
h8 > h1 h2 h3 h4 h5 h6 h7
*** Results: 0% dropped (56/56 received)

Now that I had things working it was time to find ways to break it. Diving into the flow rules I threw together a basic Drop rule on one of the transit links.

As expected the network was split into two.

mininet> pingall
*** Ping: testing ping reachability
h1 > h2 h3 h4 X X X X
h2 > h1 h3 h4 X X X X
h3 > h1 h2 h4 X X X X
h4 > h1 h2 h3 X X X X
h5 > X X X X h6 h7 h8
h6 > X X X X h5 h7 h8
h7 > X X X X h5 h6 h8
h8 > X X X X h5 h6 h7
*** Results: 57% dropped (24/56 received)

Lets see about black holing a single host now.

This drops all traffic from the host connected to port 1 on the switch which happens to be h1

mininet> pingall
*** Ping: testing ping reachability
h1 > X X X X X X X
h2 > X h3 h4 h5 h6 h7 h8
h3 > X h2 h4 h5 h6 h7 h8
h4 > X h2 h3 h5 h6 h7 h8
h5 > X h2 h3 h4 h6 h7 h8
h6 > X h2 h3 h4 h5 h7 h8
h7 > X h2 h3 h4 h5 h6 h8
h8 > X h2 h3 h4 h5 h6 h7
*** Results: 25% dropped (42/56 received)

OpenDaylight has always peaked my interested, I’ve been trying to follow the mailing lists and some of the discussions out there and the Test VM is a nice way to start to get under the hood. I have a lot more to learn and there are a ton of other plugins to start to explore. Not to mention to start to think about the API and writing some code against it.

Notes

If you do not set switch roles properly end hosts my not show up on the topology.
Flow rule names can not have spaces in them.
The controller had the Access switches properly classified in the Tier however the transit switches were not set to either Distribution or Core.

IBM PURE systems networking

locutus@the-collective.net (Ben Mason) — Mon, 02 Dec 2013 13:00:04 -0500

To start off I’ll cut past some of the marketing and state that PURE Systems are IBM BladeCenters with some predefined hardware configurations that support both x86 and POWER work loads.

With that being said the advantage to the PURE architecture is the software that IBM has assembled to orchestrate deployments of workloads across all of the integrated platforms. The orchestrator is named Flex System Manager (FSM). The FSM plugs into VMWare for x86, HMC for Power systems and other management system for virtualization platforms. The FSM will use these connections to automate deployment of systems and monitoring of the hardware, physical and virtual systems within the PURE System.

There are many details about the hardware I will not cover but one of the details IBM discusses is the increased speeds and feeds. This is accomplished by interconnections between the Nodes and the I/O Bays, each Node has multiple connection to the I/O Bays. The number of paths grow or shrink by the number of licenses, or as IBM says Pay as you Grow.

Image copied from (http://www.redbooks.ibm.com/abstracts/tips0864.html)

The portfolio of IO Modules is similar to any BladeCenter you may have seen in the past, with options for in Network Switches (BNT Switches, some supporting OpenFlow 1.0), Fiber Channel switches and passthrough modules (All the options can be found here: http://www-03.ibm.com/systems/flex/networking/).

Where I see the need for great improvement is the POWER Series networking. POWER utilizes a Virtual IO Server (VIOS) to connect the LPARs to each other and the outside world. Essentially the VIOS is a AIX server that acts as a layer 2 bridge. The VIOS lacks the ability most network switches have had to do private VLAN configurations and layer 3 inspection. There also currently is no support at this time for next generation such as OpenFlow, IBM DOVE, or VXLAN.

This brings many complications in a multi-hypervisor environment. For example locating an IBM LPAR next to a VMWare workload you will need glue it together with VLANs and legacy networking. This will require networking teams maintain network controls separately from how you may treat the rest of your virtualized work loads on the VMWare platform.

Even though I have a bit of a beef with the VIOS, the PURE system is a good approach for IBM shops to consolidate their workloads into a single Private Cloud style configuration.

How does Riverbed Steelhead Auto Discovery work?

locutus@the-collective.net (Ben Mason) — Sat, 14 Sep 2013 15:41:18 -0400

Riverbed Steelhead devices have a method they use to find each other on the network. Riverbed has named this Enhanced Auto Discovery. This is intended to reduce time to deployment and simplify the configuration on the devices. The core of this method uses setting Options in the TCP headers within the initial 3 way handshake. There are a few concepts to go over to fully understand the process of Steelhead Auto Discovery.

The Steelhead is a Layer 2 Bridge

Every Riverbed is a layer 2 bridge, for traffic to enter the optimization engine it must be bridged through the Steelhead. In the appliance there are 2 interfaces that make of the bridge, they are named the LAN and WAN interfaces. The LAN interface connects the network where the client machines, or server machines are located. The WAN interfaces connects to the external network where the router for the VPN, MPLS, P2P, 3G Radio, or whatever medium may be in use resides.

These interfaces together are called the IN PATH interfaces. In a Cisco device that is doing IRB style bridging the IN PATH interface would be similar to a BVI interface. The IN PATH interface is required to have an IP address assigned to it, this is the IP Steelheads use to communicate to each other on. For example Inner Channel is negotiated between IP addresses of IN PATH interfaces, this is one reason that IP reachability between the IN PATH IP addresses is important.

Clients and Servers

There are a couple of designations that help to clarity which devices initiate which connections. The Client Steelhead (CSH) is the Steelhead that receives the first Naked SYN from the client machine initiating the connection. A Server Steelhead (SSH) is a Steelhead that receives the SYN+ from a Client Steelhead and is the Steelhead closest to the destination server.

The Channels

Channels denote areas where specific devices communicate with each other, and where traffic is optimized or not optimized. There are 2 separate Outer Channels and a single Inner Channel.

Outer Channel (local) – This channel is between the Steelhead LAN interface and the client machine sourcing of the Naked SYN. This may be a branch workstation for example.

Outer Channel (Remote) – This is the channel between the Steelhead LAN interface and the server machine that the initial SYN was intended to be received by. This may be some sort of application or file server at a data center for example.

Inner Channel – This is the network between the WAN interfaces of the Steelheads. This is a TCP connection between the IN PATH IP address where all optimized traffic between Steelheads passes. The default for used for this connection is TCP 7800.

The Process

Now on to the steps used for the Steelhead devices to find each other. There are 7 main steps involved, in the following example you are initiating a TCP connection from a Client workstation to a Server to copy a file.

The client machine sends a Naked SYN packet, this is a SYN with out the PROBE TCP option set. This SYN packet is received on the LAN interface of the CSH and is intercepted by the CSH. At this stage the Steelhead checks licenses to make sure this traffic is entitled, if it is not the traffic is just passed through unchanged. If all is well the CSH will add the TCP header Option number 76. This option header is named AUTO-DISCOVERY PROBE and will include such information as the IN PATH IP address and what role this Steelhead is assuming. A packet with an TCP option is noted by a ‘+’ in documentation, for example SYN+. After the header is set the packet is forwarded out the WAN interface.
This SYN+ is received by the Steelhead that will become the SSH (in this example) on the WAN interface. Again at this stage licenses on this Steelhead are checked to make sure this traffic is entitled, if not the traffic is just passed through untouched. If the SYN does not for the option headers set it is also passed through untouched.
Since this packets is a SYN+ the Steelhead then sends a SYN/ACK+ with FWD Negotiation option back towards Client machine. This SYN/ACK+ is then intercepted by the CSH on the return path.
The Steelhead near the Server machine starts a negotiates 3 way handshake with Server machine. If this Steelhead does not encounter another Steelhead between itself and the Server machine it will assume the role of SSH and complete an 3 way handshake with the Server machine. If a Steelhead is encountered this process repeats down the line towards the server.
The newly declared SSH will then send a SYN/ACK+ with PROBE RESPONSE option in headers towards CSH.
The SYN/ACK+ is intercepted by the CSH and the CSH initiates the Inner channel between the CSH to the SSH.
Once the Inner Channel is formed the CSH completes the 3 way between itself and the Client machine.

Now that the above steps are completed the traffic between the Client machines and Server machines is on its way and the Steelhead is transparently (to the client and server) doing its optimization magic. All of the traffic in this flow, after it has been optimized is transmitted, over the Inner Channel. If a new traffic flow needs to communicate between the Client and Server the process starts again for that new traffic flow and this is the case for every TCP connection that occurs from site to site.

Where Trouble can Occur

There are a few basic items that can easily stop this process from occurring, therefor blocking optimization from occurring.

If there is something strips the Option headers out of TCP packets such as a firewall or IDS.
The IN PATH interfaces do not have Layer 3 reachability between each other. This will prevent the Inner Channel from forming.
The traffic does not follow from the LAN to WAN and WAN to LAN interfaces on both the SSH and CSH.
The boxes are improperly licensed or the amount of traffic/tcp session is exceeding the license

Over all this is a pretty simple process and not to different from how other vendors handle things such as Cisco WAAS. Even though they use technologies such as WCCPv2. There are other scenarios that such as Virtual In Path and Server Side Out of Path that are a bit different, but this is the Riverbed recommended way of doing things.