Problem:
It was one of those cold February winter nights, when I sat down on my comfy couch, late in the night determined to finish my slide deck for Cisco Live Berlin session on Deploying FirePOWER Threat Defense for ISR (BRKSEC-2057). I know I had been putting it off for a while but the time has come for me to buckle down and get it done that day.
The whole neighborhood had just gotten quiet around 11:30 PM. I let the dogs out, checked all the doors, sat back down and starred at the topology to see what tests I can run in order to gather some useful outputs and screen shots for my slides.
It was around midnight, I made that terrible mistake of adding an incorrect route on the ISR 881. I lost my test bed. Can't ssh or telnet back to the ISR 881. I meant to add the route on the laptop (to check connectivity through the ISR 4451) that I had setup at my desk at work that I was VNC-ed into via VPN from my laptop at home. Oh no!! What do I do?
- All vty lines on the ISR 881 were locked except 1 and I had used that to manage the router from the laptop behind it. Router denied any telnet or ssh sessions.
- ISR 881 was managed by Prime at one point. I remembered that very vividly.
- I knew exactly the route statement that I added that broke connectivity to the laptop behind the 881. Added a host route to the next-hop on the Corporate network via ISR 4451 172.16.1.3. Doh! who does that? Call me stupid...and I'd blame it totally on the long day and exhaustion...
- laptop GW was pointing to .1, the ISR 881. IP address was statically configured on the laptop.
- I had access to the ISR 4451 and the switch that connected all the devices in the 172.16.1.0/24network
- None of the equipment was connected to a term server or power management server to reload them remotely.
Topology:
Incorrect route added on the ISR 881: ip route 10.150.217.1 255.255.255.255 172.16.1.3
Well, RTP, North Carolina had gone to bed many hours ago so, I had no one to ping who was sitting at work who could just reload the router for me. San Jose too had long gone home for the night. My only thought was Australia must be awake and decided to ping Phil Petty, our software engineer based in Australia. It was a wonderful morning time for him and responded immediately to my jabber ping.
With so much hope, I asked him for Prime Infrastructure credentials so I could add the IP address 10.150.217.107 of the 881 to Prime and somehow figure out a way to reboot the router. Though the device got added using SNMP, CLI access failed as telnet and ssh from Prime failed as the lines were locked up (I cursed myself for not looking at the issue when the router denied telnet or ssh when another person was already using a line). Device discovery worked as the SNMP string (read/write) was in place from previous tests with Prime Infrastructure.
Curious Phil asked me what I was working on and when I got done explaining what I was doing over jabber, he too got hooked on to the problem. There's got to be a way to undo the route statement that I added and gain VNC back to the laptop. I didn't want to drive to work so late in the night even thought it is only a 10 min drive for me to get to work.
I was thinking about one of my buddies in TAC who decided to write a script that would answer a survey for him instead of clicking the radio/square buttons himself. Who does that?? Here I am, instead of driving to work and being done in 20 min. I am breaking my head to find a way to do this without having to drive and without having to reload the router. This is what makes us great engineers; never give up and find all options to solve the problem.
I read Phil typing, "Kureli, I think I can make this work...." He sounded pretty confident. Woo Hoo...! I was on board with his idea. Within minutes I was able to establish VNC back to the laptop behind the ISR 881 from home and continued working until the wee hours of the morning.
You all hate me don't you? For not telling you what exactly we did to undo the incorrect route on the ISR 881?
Solution:
I will update this blog and add the solution to the problem in about a week...keep guessing until then.....
If you can't wait, unicast me your answer and I will tell you if you are headed in the right direction.
Here is the most awaited solution:
Phil simply googled and found this link and told me, "Kureli, I think I can make this work."
http://www.cisco.com/c/en/us/support/docs/ip/simple-network-management-protocol-snmp/15217-copy-configs-snmp.html
My jabber conversation with Phil went sorta like this:
Phil: Kureli, you have access to the 4451 right?
Me: Yes.
Phil: Then, could you negate the incorrect route that you added, put it in a text file and upload it to the ISR 4451's flash?
Me: Yes absolutely.
I wrote a one liner file below and tftp-ed it to the 4451 router's flash from my home laptop:
ISR-4451#more flash:no-route.txt
no ip route 10.150.217.1 255.255.255.255 172.16.1.3
Next, we made the ISR 4451 the TFTP server. Then Phil ran an SNMP command against the ISR 881 to go and download the "one liner" from the ISR 4451 and apply it to its running config.
mcp-bld-syd-01:~>snmpset -v2c -c public 10.150.217.107 .1.3.6.1.4.1.9.9.96.1.1.1.1.2.52 i 1 .1.3.6.1.4.1.9.9.96.1.1.1.1.3.52 i 4 .1.3.6.1.4.1.9.9.96.1.1.1.1.4.52 i 1 .1.3.6.1.4.1.9.9.96.1.1.1.1.5.52 a "10.150.217.132" .1.3.6.1.4.1.9.9.96.1.1.1.1.6.52 s "bootflash:no-route.txt" .1.3.6.1.4.1.9.9.96.1.1.1.1.14.52 i 4
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.2.52 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.3.52 = INTEGER: 4
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.4.52 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.5.52 = IpAddress: 10.150.217.132
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.6.52 = STRING: "bootflash:no-route.txt"
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.14.52 = INTEGER: 4
mcp-bld-syd-01:~>
Voila! The incorrect route got deleted and I got VNC back to the laptop behind the ISR 881.
10 Comments
