It was one of those cold February winter nights, when I sat down on my comfy couch, late in the night determined to finish my slide deck for Cisco Live Berlin session on Deploying FirePOWER Threat Defense for ISR (BRKSEC-2057). I know I had been putting it off for a while but the time has come for me to buckle down and get it done that day.
The whole neighborhood had just gotten quiet around 11:30 PM. I let the dogs out, checked all the doors, sat back down and starred at the topology to see what tests I can run in order to gather some useful outputs and screen shots for my slides.
It was around midnight, I made that terrible mistake of adding an incorrect route on the ISR 881. I lost my test bed. Can't ssh or telnet back to the ISR 881. I meant to add the route on the laptop (to check connectivity through the ISR 4451) that I had setup at my desk at work that I was VNC-ed into via VPN from my laptop at home. Oh no!! What do I do?
All vty lines on the ISR 881 were locked except 1 and I had used that to manage the router from the laptop behind it. Router denied any telnet or ssh sessions.
ISR 881 was managed by Prime at one point. I remembered that very vividly.
I knew exactly the route statement that I added that broke connectivity to the laptop behind the 881. Added a host route to the next-hop on the Corporate network via ISR 4451 172.16.1.3. Doh! who does that? Call me stupid...and I'd blame it totally on the long day and exhaustion...
laptop GW was pointing to .1, the ISR 881. IP address was statically configured on the laptop.
I had access to the ISR 4451 and the switch that connected all the devices in the 172.16.1.0/24network
None of the equipment was connected to a term server or power management server to reload them remotely.
Incorrect route added on the ISR 881: ip route 10.150.217.1 255.255.255.255 172.16.1.3
Well, RTP, North Carolina had gone to bed many hours ago so, I had no one to ping who was sitting at work who could just reload the router for me. San Jose too had long gone home for the night. My only thought was Australia must be awake and decided to ping Phil Petty, our software engineer based in Australia. It was a wonderful morning time for him and responded immediately to my jabber ping.
With so much hope, I asked him for Prime Infrastructure credentials so I could add the IP address 10.150.217.107 of the 881 to Prime and somehow figure out a way to reboot the router. Though the device got added using SNMP, CLI access failed as telnet and ssh from Prime failed as the lines were locked up (I cursed myself for not looking at the issue when the router denied telnet or ssh when another person was already using a line). Device discovery worked as the SNMP string (read/write) was in place from previous tests with Prime Infrastructure.
Curious Phil asked me what I was working on and when I got done explaining what I was doing over jabber, he too got hooked on to the problem. There's got to be a way to undo the route statement that I added and gain VNC back to the laptop. I didn't want to drive to work so late in the night even thought it is only a 10 min drive for me to get to work.
I was thinking about one of my buddies in TAC who decided to write a script that would answer a survey for him instead of clicking the radio/square buttons himself. Who does that?? Here I am, instead of driving to work and being done in 20 min. I am breaking my head to find a way to do this without having to drive and without having to reload the router. This is what makes us great engineers; never give up and find all options to solve the problem.
I read Phil typing, "Kureli, I think I can make this work...." He sounded pretty confident. Woo Hoo...! I was on board with his idea. Within minutes I was able to establish VNC back to the laptop behind the ISR 881 from home and continued working until the wee hours of the morning.
You all hate me don't you? For not telling you what exactly we did to undo the incorrect route on the ISR 881?
I will update this blog and add the solution to the problem in about a week...keep guessing until then.....
If you can't wait, unicast me your answer and I will tell you if you are headed in the right direction.
Dear community, I have configured MAB for following UseCases: If WiredMAB -> place it to a dynamic VLAN ID. However, I noted on live logs that flows that primarily do authenticate successfully with EAP-TLS are failing back to MAB! And t...
Hello, I am new to cisco security technologies and I would like to make a question regarding anyconnect. We need to upgrade anyconnect from 4.1 to 4.10, does anyone know if there would be any issue related to licensing? or anything that I should...
From documentation, you can release stuck backup using the command app conf ise and choosing the option 24 (Force Backup Cancellation). After choosing this option, will the ISE node reload? I didn't find anything usefull in the documentation.
Hi everyoneI have this confusing problem with clustering two FTD 4100 series, when copying the cluster configuration from the first one to the other one, it stays at this state all the time and not give even an errornotice that I have both FTD have 2 port...
Hi,We noticed 'high CPU usage on the ASA 5555-X series firepower device. We checked the configuration and logs and there are no symptoms of CPU usages. Could someone please advise? Model : Cisco ASA5555-X Threat Defense (75) Version 6.6.1 (Build 91)C...