I know it's cliche and I know I'm biased because I have an @cisco.com email address, but I've truthfully never seen anything like CPOC before. And the customer's I've worked with at CPOC haven't either. It's extremely gratifying to take something you built "on paper" and prove that it works; to take it to the next level and work those final kinks out that the paper design just didn't account for.

Anyways, on to the point of this post. When I was building the topology for the customer, I kept notes about random things I ran into that I wanted to remember later or those "oh duh!" moments that I probably should've known the answer to but had forgotten or overlooked at the time. This post is just a tidy-up of those notes, in no particular order.

ASA CLI prompt doesn't show HA state or context⌗

After I configured the ASA HA pair and setup multiple contexts, the CLI still looked like this:

ralph(config)#

I assumed that the prompt would change to reflect the HA state and active context. This actually needs to be configured though using the prompt command.

ralph(config)# prompt ?

configure mode commands/options:
  cluster-unit     Display the cluster unit name in the session prompt
  context          Display the context in the session prompt (multimode only)
  domain           Display the domain in the session prompt
  hostname         Display the hostname in the session prompt
  management-mode  Display management mode
  priority         Display the priority in the session prompt
  state            Display the traffic passing state in the session prompt

I chose this order for the prompt options: prompt hostname priority state context which resulted in a prompt that looked like this:

ralph/sec/act/context1(config)#

Modify Nexus reserved VLAN range⌗

Nexus switches (and Cat 6500/6800, too) reserve a few VLANs for use by the system. VLANs in this reserved range cannot be used by front-facing ports and you'll get an error if you try to do so.

euston# show vlan internal usage

VLANs                   DESCRIPTION
-------------------     -----------------
3968-4031               Multicast
4032-4035               Online Diagnostic
4036-4039               ERSPAN
4042                    Satellite
4046                    Fabric scale
3968-4047,4094          Current

The reserved range can be modified but I always forget the command.

In NX-OS:

euston(config)# system vlan 2500 reserve
This will delete all configs on vlans 2500-2579. Continue anyway? (y/n) [no] y
Note: After switch reload, VLANs 2500-2579 will be reserved for internal use.
 This requires copy running-config to startup-config before
 switch reload. Creating VLANs within this range is not allowed.

In 15.xSY on Cat6k:

adidas(config)# vlan internal reserved met vlan 2500

The VLAN number given in the command will be the starting VLAN for the reserved range. And yeah, you do have to reload the box for the changes to take affect.

I have permit any any but ASA still blocks my traffic!⌗

The ASA is mostly foreign territory to me so this one was a head scratcher for a bit.

The testing we were doing was focused on network reconvergence after a failure event occurred. The ASAs were participating in dynamic routing so they were a key part of the test bed, however we were not doing security testing so I had given each ASA a global ACL with a single entry that permitted "ip any any". However, with this config I was having reachability problems through the ASA. Even worse, when I broke down and used the ASDM Packet Tracer (oh the humanity!) it was telling me the traffic was blocked due to the implicit deny rule.

ASA Implicit Packet Drop

How was the implicit "deny" rule taking precedence over my explicit "permit" rule??

Hat tip to 3cvguy.com which reminded me that the ASA, by default, does not allow traffic to traverse between interfaces at the same security level. Turns out that's exactly what I was trying to do.

This command will override the default behavior and allow same-security traffic between interfaces.

manchester(config)# same-security-traffic permit inter-interface

Boot issues with ASR 1002⌗

Trivia time. What's wrong with this picture?

ASR1002-RP1 platform with 4194303 Kbytes of main memory

rommon 1 > dir usb0:
File System: FAT32

3 511356216 -rw- asr1001x-universal.03.16.02.S.155-3.S2-ext.SPA.bin
rommon 2 > boot usb0:/asr1001x-universal.03.16.02.S.155-3.S2-ext.SPA.bin
Located asr1001x-universal.03.16.02.S.155-3.S2-ext.SPA.bin, start cluster is 3

##########################################################################################[..]
*** Instruction Access Exception ***
PC = 0xbaddaddc, Vector = 0x400, SP = 0x6d3738
CPU context of the most recent exception:
R0 = 0xbaddaddd R1 = 0x006d3738 R2 = 0xffe3ffe3 R3 = 0x00000000
R4 = 0x0000000c R5 = 0x006d3740 R6 = 0x006d3718 R7 = 0x00000000
R8 = 0x00000001 R9 = 0x388d70d6 R10 = 0x00000001 R11 = 0x00000000
R12 = 0x00000001 R13 = 0x8000001c R14 = 0xff6aebf1 R15 = 0xcf7beff7
R16 = 0xbb7bdbff R17 = 0x6fd7fef7 R18 = 0x1c97f974 R19 = 0xb77fb3ef
R20 = 0xfbf00000 R21 = 0x00000000 R22 = 0x000f3ff0 R23 = 0x00000001
R24 = 0x00000200 R25 = 0x1e000000 R26 = 0x1e009c60 R27 = 0x00000200
R28 = 0x002b0000 R29 = 0x00000001 R30 = 0x00000004 R31 = 0x00000000
CR = 0x20000004 LR = 0xffe6f85c CTR = 0xbaddaddd XER = 0x00000000
TBU = 0x00000001 TBL = 0x9177375c DEAR = 0x00000000 IVPR = 0x00000000
PVR = 0x00000000 DBCR0 = 0x00000000 DBCR1 = 0x00000000 DBCR2 = 0x00000000
IAC1 = 0x00000000 IAC2 = 0x00000000 DAC1 = 0x00000000 DAC2 = 0x00000000
CSRR0 = 0x00000000 CSRR1 = 0x00000000 MCSRR0 = 0x00000000 MCSRR1 = 0x00000000
PC = 0xbaddaddc MSR = 0x02009200
<router halts, no more output, no response on console>

I was troubleshooting this after 11/12 hours in the lab and it took my tired brain a while to figure this out.

This is a 1002 with an RP1. I'm trying to boot an image for a 1001-X. They are not compatible 😲. Make sure you boot the correct image on your ASRs. Look for the right ASR model (1002) and the correct RP (RP-1 in this case).

asr1000rp1-adventerprise.03.16.02b.S.155-3.S2b-ext.bin
Redistributing into EIGRP not working⌗

This never would have escaped my notice when I was studying for my CCIE but things get a little foggy over time so it took me a bit to realize my mistake here.

Wrong:

router eigrp 2000
 redistribute bgp 65100

Right:

router eigrp 2000
 redistribute bgp 65100 route-map set-eigrp-metric
OR
router eigrp 2000
  redistribute bgp 65100 metric 10000 1 255 1 1500

When redistributing a dynamic protocol into EIGRP, you must specify the seed metric either via a route-map that does set metric or manually as part of the redistribute command.

Where is this extra 30 seconds coming from?⌗

As I said above, the focus of this CPOC was on network convergence measurements so there was lots of simulated link and node failures. Since the network was routed everywhere except right at the edge, a lot of attention was paid to how the IGP was handling the failures.

In one particular failure scenario, we were seeing an extra 30 seconds for convergence which was an eternity considering the other tests were a few seconds at worst, and sub-second at best.

We spent a lot of time scratching out heads wondering why the IGP was taking so long. We were basically out of ideas when one of the engineers suggested we look at Layer 2. Sure enough, it was Spanning Tree going through Listen/Learn phases. There's our 30 seconds.

So lesson learned. Even though you might not have Layer 2 loops in the network, STP still runs and can still be a factor!

If you're seeing 30 seconds for something, think STP!

Capture ASA packet drops⌗

This one was brand new to me; I had no idea ASA would do this. Hat tip to RH for showing me this.

ASA can log and display the packets that it's dropping in the fast path. This makes troubleshooting so much easier.

ralph# capture MYCAP type asp-drop all buffer 15000 circular-buffer

By running show capture MYCAP, the contents of the buffer can be examined. The full syntax of the capture command is here: cisco.com.

As a supplement to examining packet drops, this document gives a detailed explanation of the ASA's drop reasons (eg, l2_acl, no-route, or invalid-ethertype) and what each counter indicates.

No debug output from OSPF in a VRF on Nexus 5k⌗

I expect lots of log messages when turning on these debugs... where are my log messages??

camden# show debug

Debug level is set to Minor(1)
 default for new sessions logging level: 3

debug ip ospf adjacency terse
debug ip ospf graceful-restart detail
debug ip ospf packets
camden#
camden#
camden#
camden# show ip ospf neigh vrf wan
 OSPF Process ID 10 VRF WAN
 Total number of neighbors: 3
Neighbor ID  Pri State        Up Time  Address      Interface
10.2.60.3    0   FULL/DROTHER 00:11:25 10.2.60.129  Vlan2999
10.2.60.253  1   FULL/ -      19:38:20 10.2.60.177  Vlan2995
10.22.60.254 1   FULL/ -      19:21:36 10.2.60.186  Eth1/6

Third and final hat tip to Tim Stevenson at Cisco for pointing out that you need to tell NX-OS to debug the OSPF process in the VRF by configuring a debug filter.

camden# debug-filter ?
 ip IP events
 ip Display IS-IS IPv4 information
 ipv6 IPv6 events
 otv IS-IS debug-filter
 pktmgr Pm debug-filter
 routing Routing events

camden# debug-filter ip ?
 arp ARP events
 detail Transport layer protocol detailed info
 icmp ICMP packet information
 igmp IGMP related events
 mfwd IP multicast forwarding packet information
 mpacket IP multicast packet information
 mrouting IP Multicast routing table events
 ospf Debug OSPF events
 otv IS-IS debug-filter
 packet IP packet information
 raw Dump [first 60] bytes of IP packet

camden# debug-filter ip ospf ?
 10 Process tag
 interface Interface filter
 lsa-advrtr LSA Advertising router filter
 lsa-lsid LSA LSID filter
 lsa-lstype LSA LS type filter
 neighbor Neighbor filter
 prefix Prefix filter
 spf-area SPF Area filter
 spf-type SPF type filter
 spf-vertex SPF Vertex filter
 vrf Display per-VRF information
camden#
camden#
camden# debug-filter ip ospf vrf WAN
camden# 2016 Jun 8 14:58:28.678312 ospf: 10 [4950] (WAN) rcvd: prty:5 ver:2 t:HELLO len:48 rid:10.2.60.253 area:0.0.0.0 crc:0x6d9f aut:0 aukid:0 from 10.2.60.177/Vlan2995
2016 Jun 8 14:58:28.882686 ospf: 10 [4950] (WAN) sent: prty:6 HELLO to 224.0.0.5/Vlan2999

Disclaimer: The opinions and information expressed in this blog article are my own and not necessarily those of Cisco Systems.