It was one of those cold February winter nights, when I sat down on my comfy couch, late in the night determined to finish my slide deck for Cisco Live Berlin session on Deploying FirePOWER Threat Defense for ISR (BRKSEC-2057). I know I had been putting it off for a while but the time has come for me to buckle down and get it done that day.
The whole neighborhood had just gotten quiet around 11:30 PM. I let the dogs out, checked all the doors, sat back down and starred at the topology to see what tests I can run in order to gather some useful outputs and screen shots for my slides.
It was around midnight, I made that terrible mistake of adding an incorrect route on the ISR 881. I lost my test bed. Can't ssh or telnet back to the ISR 881. I meant to add the route on the laptop (to check connectivity through the ISR 4451) that I had setup at my desk at work that I was VNC-ed into via VPN from my laptop at home. Oh no!! What do I do?
All vty lines on the ISR 881 were locked except 1 and I had used that to manage the router from the laptop behind it. Router denied any telnet or ssh sessions.
ISR 881 was managed by Prime at one point. I remembered that very vividly.
I knew exactly the route statement that I added that broke connectivity to the laptop behind the 881. Added a host route to the next-hop on the Corporate network via ISR 4451 172.16.1.3. Doh! who does that? Call me stupid...and I'd blame it totally on the long day and exhaustion...
laptop GW was pointing to .1, the ISR 881. IP address was statically configured on the laptop.
I had access to the ISR 4451 and the switch that connected all the devices in the 172.16.1.0/24network
None of the equipment was connected to a term server or power management server to reload them remotely.
Incorrect route added on the ISR 881: ip route 10.150.217.1 255.255.255.255 172.16.1.3
Well, RTP, North Carolina had gone to bed many hours ago so, I had no one to ping who was sitting at work who could just reload the router for me. San Jose too had long gone home for the night. My only thought was Australia must be awake and decided to ping Phil Petty, our software engineer based in Australia. It was a wonderful morning time for him and responded immediately to my jabber ping.
With so much hope, I asked him for Prime Infrastructure credentials so I could add the IP address 10.150.217.107 of the 881 to Prime and somehow figure out a way to reboot the router. Though the device got added using SNMP, CLI access failed as telnet and ssh from Prime failed as the lines were locked up (I cursed myself for not looking at the issue when the router denied telnet or ssh when another person was already using a line). Device discovery worked as the SNMP string (read/write) was in place from previous tests with Prime Infrastructure.
Curious Phil asked me what I was working on and when I got done explaining what I was doing over jabber, he too got hooked on to the problem. There's got to be a way to undo the route statement that I added and gain VNC back to the laptop. I didn't want to drive to work so late in the night even thought it is only a 10 min drive for me to get to work.
I was thinking about one of my buddies in TAC who decided to write a script that would answer a survey for him instead of clicking the radio/square buttons himself. Who does that?? Here I am, instead of driving to work and being done in 20 min. I am breaking my head to find a way to do this without having to drive and without having to reload the router. This is what makes us great engineers; never give up and find all options to solve the problem.
I read Phil typing, "Kureli, I think I can make this work...." He sounded pretty confident. Woo Hoo...! I was on board with his idea. Within minutes I was able to establish VNC back to the laptop behind the ISR 881 from home and continued working until the wee hours of the morning.
You all hate me don't you? For not telling you what exactly we did to undo the incorrect route on the ISR 881?
I will update this blog and add the solution to the problem in about a week...keep guessing until then.....
If you can't wait, unicast me your answer and I will tell you if you are headed in the right direction.
Hello All, Have two query it would be great help if anyone helps 1) My company management wants to configure 'syslog logging facility' is equal to '23' is it possible to do that in ASA ?. Also need to reconfigure rsa key pair equal or ...
I have an internet speed of 140-150 Mbps but when I connect to the Cisco Anyconnect Secure Mobile client it kills my internet speed to 500-1000 kbps which is too slow. What possible solution I can try to increase my speed on VpnOS - Ubuntu 18.04.5 LTS
Hello everyone.I'm currently setting up a FMC and while attempting to use external authentication via LDAP, for some reason the FMC is not querying properly.Basically whenever I attempt the test the query, it only finds user machines and groups CNs , not ...
I have added the CRL URL link in the FMC (Ver 6.6.4) But after adding the CRL url link FMC GUI login page not coming but I m able to login through CLI. Pls suggest how to remove CRL url link from the FMC CLI.