I was lucky enough (volunteering for very challenging work is luck, right? 😁) to finish my third tour through Cisco CPOC last week. CPOC is Cisco’s Customer Proof of Concept facility where customer’s can bring their network design, build it in Cisco’s lab, and beat the hell out of it. CPOC has tons of network and compute gear, all the right testing tools and processes, and excellent work areas that cater to collaborative work and information sharing. It’s also staffed by very senior and experienced engineers.
I know it’s cliche and I know I’m biased because I have an @cisco.com email address, but I’ve truthfully never seen anything like CPOC before. And the customer’s I’ve worked with at CPOC haven’t either. It’s extremely gratifying to take something you built “on paper” and prove that it works; to take it to the next level and work those final kinks out that the paper design just didn’t account for.
If you want more information about CPOC, get in touch with me or leave a comment below. Or ask your Cisco SE (and if they don’t know, have them get in touch with me).
Anyways, on to the point of this post. When I was building the topology for the customer, I kept notes about random things I ran into that I wanted to remember later or those “oh duh!” moments that I probably should’ve known the answer to but had forgotten or overlooked at the time. This post is just a tidy-up of those notes, in no particular order.
ASA CLI prompt doesn’t show HA state or context
After I configured the ASA HA pair and setup multiple contexts, the CLI still looked like this:
ralph(config)# prompt ? configure mode commands/options: cluster-unit Display the cluster unit name in the session prompt context Display the context in the session prompt (multimode only) domain Display the domain in the session prompt hostname Display the hostname in the session prompt management-mode Display management mode priority Display the priority in the session prompt state Display the traffic passing state in the session prompt
I chose this order for the prompt options:
Modify Nexus reserved VLAN range
Nexus switches (and Cat 6500/6800, too) reserve a few VLANs for use by the system. VLANs in this reserved range cannot be used by front-facing ports and you’ll get an error if you try to do so.
euston# show vlan internal usage VLANs DESCRIPTION ------------------- ----------------- 3968-4031 Multicast 4032-4035 Online Diagnostic 4036-4039 ERSPAN 4042 Satellite 4046 Fabric scale 3968-4047,4094 Current
The reserved range can be modified but I always forget the command.
euston(config)# system vlan 2500 reserve This will delete all configs on vlans 2500-2579. Continue anyway? (y/n) [no] y Note: After switch reload, VLANs 2500-2579 will be reserved for internal use. This requires copy running-config to startup-config before switch reload. Creating VLANs within this range is not allowed.
In 15.xSY on Cat6k:
adidas(config)# vlan internal reserved met vlan 2500
The VLAN number given in the command will be the starting VLAN for the reserved range. And yeah, you do have to reload the box for the changes to take affect.
The ASA is mostly foreign territory to me so this one was a head scratcher for a bit.
The testing we were doing was focused on network reconvergence after a failure event occurred. The ASAs were participating in dynamic routing so they were a key part of the test bed, however we were not doing security testing so I had given each ASA a global ACL with a single entry that permitted “ip any any”. However, with this config I was having reachability problems through the ASA. Even worse, when I broke down and used the ASDM Packet Tracer (oh the humanity!) it was telling me the traffic was blocked due to the implicit deny rule.
How was the implicit “deny” rule taking precedence over my explicit “permit” rule??
Hat tip to http://3cvguy.com/cisco-asa/ which reminded me that the ASA, by default, does not allow traffic to traverse between interfaces at the same security level. Turns out that’s exactly what I was trying to do.
This command will override the default behavior and allow same-security traffic between interfaces.
manchester(config)# same-security-traffic permit inter-interface
Boot issues with ASR 1002
Trivia time. What’s wrong with this picture?
ASR1002-RP1 platform with 4194303 Kbytes of main memory rommon 1 > dir usb0: File System: FAT32 3 511356216 -rw- asr1001x-universal.03.16.02.S.155-3.S2-ext.SPA.bin rommon 2 > boot usb0:/asr1001x-universal.03.16.02.S.155-3.S2-ext.SPA.bin Located asr1001x-universal.03.16.02.S.155-3.S2-ext.SPA.bin, start cluster is 3 ##########################################################################################[..] *** Instruction Access Exception *** PC = 0xbaddaddc, Vector = 0x400, SP = 0x6d3738 CPU context of the most recent exception: R0 = 0xbaddaddd R1 = 0x006d3738 R2 = 0xffe3ffe3 R3 = 0x00000000 R4 = 0x0000000c R5 = 0x006d3740 R6 = 0x006d3718 R7 = 0x00000000 R8 = 0x00000001 R9 = 0x388d70d6 R10 = 0x00000001 R11 = 0x00000000 R12 = 0x00000001 R13 = 0x8000001c R14 = 0xff6aebf1 R15 = 0xcf7beff7 R16 = 0xbb7bdbff R17 = 0x6fd7fef7 R18 = 0x1c97f974 R19 = 0xb77fb3ef R20 = 0xfbf00000 R21 = 0x00000000 R22 = 0x000f3ff0 R23 = 0x00000001 R24 = 0x00000200 R25 = 0x1e000000 R26 = 0x1e009c60 R27 = 0x00000200 R28 = 0x002b0000 R29 = 0x00000001 R30 = 0x00000004 R31 = 0x00000000 CR = 0x20000004 LR = 0xffe6f85c CTR = 0xbaddaddd XER = 0x00000000 TBU = 0x00000001 TBL = 0x9177375c DEAR = 0x00000000 IVPR = 0x00000000 PVR = 0x00000000 DBCR0 = 0x00000000 DBCR1 = 0x00000000 DBCR2 = 0x00000000 IAC1 = 0x00000000 IAC2 = 0x00000000 DAC1 = 0x00000000 DAC2 = 0x00000000 CSRR0 = 0x00000000 CSRR1 = 0x00000000 MCSRR0 = 0x00000000 MCSRR1 = 0x00000000 PC = 0xbaddaddc MSR = 0x02009200 <router halts, no more output, no response on console
I was troubleshooting this after 11/12 hours in the lab and it took my tired brain a while to figure this out.
This is a 1002 with an RP1. I’m trying to boot an image for a 1001-X. They are not compatible 😱. Make sure you boot the correct image on your ASRs. Look for the right ASR model (1002) and the correct RP (RP-1 in this case).
Redistributing into EIGRP not working
This never would have escaped my notice when I was studying for my CCIE but things get a little foggy over time so it took me a bit to realize my mistake here.
router eigrp 2000 redistribute bgp 65100
router eigrp 2000 redistribute bgp 65100 route-map set-eigrp-metric OR router eigrp 2000 redistribute bgp 65100 metric 10000 1 255 1 1500
When redistributing a dynamic protocol into EIGRP, you must specify the seed metric either via a route-map that does
set metric or manually as part of the redistribute command.
Where is this extra 30 seconds coming from?
As I said above, the focus of this CPOC was on network convergence measurements so there was lots of simulated link and node failures. Since the network was routed everywhere except right at the edge, a lot of attention was paid to how the IGP was handling the failures.
In one particular failure scenario, we were seeing an extra 30 seconds for convergence which was an eternity considering the other tests were a few seconds at worst, and sub-second at best.
We spent a lot of time scratching out heads wondering why the IGP was taking so long. We were basically out of ideas when one of the engineers suggested we look at Layer 2. Sure enough, it was Spanning Tree going through Listen/Learn phases. There’s our 30 seconds.
So lesson learned. Even though you might not have Layer 2 loops in the network, STP still runs and can still be a factor!
If you’re seeing 30 seconds for something, think STP!
Capture ASA packet drops
This one was brand new to me; I had no idea ASA would do this. Hat tip to RH for showing me this.
ASA can log and display the packets that it’s dropping in the fast path. This makes troubleshooting so much easier.
show capture MYCAP, the contents of the buffer can be examined. The full syntax of the
capture command is here: cisco.com.
As a supplement to examining packet drops, this document gives a detailed explanation of the ASA’s drop reasons (eg,
invalid-ethertype) and what each counter indicates.
I expect lots of log messages when turning on these debugs… where are my log messages??
camden# show debug Debug level is set to Minor(1) default for new sessions logging level: 3 debug ip ospf adjacency terse debug ip ospf graceful-restart detail debug ip ospf packets camden# camden# camden# camden# show ip ospf neigh vrf wan OSPF Process ID 10 VRF WAN Total number of neighbors: 3 Neighbor ID Pri State Up Time Address Interface 10.2.60.3 0 FULL/DROTHER 00:11:25 10.2.60.129 Vlan2999 10.2.60.253 1 FULL/ - 19:38:20 10.2.60.177 Vlan2995 10.22.60.254 1 FULL/ - 19:21:36 10.2.60.186 Eth1/6
Third and final hat tip to Tim Stevenson at Cisco for pointing out that you need to tell NX-OS to debug the OSPF process in the VRF by configuring a debug filter.
camden# debug-filter ? ip IP events ip Display IS-IS IPv4 information ipv6 IPv6 events otv IS-IS debug-filter pktmgr Pm debug-filter routing Routing events camden# debug-filter ip ? arp ARP events detail Transport layer protocol detailed info icmp ICMP packet information igmp IGMP related events mfwd IP multicast forwarding packet information mpacket IP multicast packet information mrouting IP Multicast routing table events ospf Debug OSPF events otv IS-IS debug-filter packet IP packet information raw Dump [first 60] bytes of IP packet camden# debug-filter ip ospf ? 10 Process tag interface Interface filter lsa-advrtr LSA Advertising router filter lsa-lsid LSA LSID filter lsa-lstype LSA LS type filter neighbor Neighbor filter prefix Prefix filter spf-area SPF Area filter spf-type SPF type filter spf-vertex SPF Vertex filter vrf Display per-VRF information camden# debug-filter ip ospf vrf WAN camden# 2016 Jun 8 14:58:28.678312 ospf: 10  (WAN) rcvd: prty:5 ver:2 t:HELLO len:48 rid:10.2.60.253 area:0.0.0.0 crc:0x6d9f aut:0 aukid:0 from 10.2.60.177/Vlan2995 2016 Jun 8 14:58:28.882686 ospf: 10  (WAN) sent: prty:6 HELLO to 22.214.171.124/Vlan2999