cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Uh oh!! Did I lose connectivity to my router??!!

2024
Views
10
Helpful
10
Comments
Kureli Sankar
Cisco Employee

Problem:

It was one of those cold February winter nights, when I sat down on my comfy couch, late in the night determined to finish my slide deck for Cisco Live Berlin session on Deploying FirePOWER Threat Defense for ISR (BRKSEC-2057).  I know I had been putting it off for a while but the time has come for me to buckle down and get it done that day. 

The whole neighborhood had just gotten quiet around 11:30 PM.  I let the dogs out, checked all the doors, sat back down and starred at the topology to see what tests I can run in order to gather some useful outputs and screen shots for my slides.

It was around midnight, I made that terrible mistake of adding an incorrect route on the ISR 881. I lost my test bed. Can't ssh or telnet back to the ISR 881.  I meant to add the route on the laptop (to check connectivity through the ISR 4451) that I had setup at my desk at work that I was VNC-ed into via VPN from my laptop at home. Oh no!! What do I do?

  • All vty lines on the ISR 881 were locked except 1 and I had used that to manage the router from the laptop behind it. Router denied any telnet or ssh sessions.
  • ISR 881 was managed by Prime at one point. I remembered that very vividly.
  • I knew exactly the route statement that I added that broke connectivity to the laptop behind the 881. Added a host route to the next-hop on the Corporate network via ISR 4451 172.16.1.3. Doh! who does that? Call me stupid...and I'd blame it totally on the long day and exhaustion...
  • laptop GW was pointing to .1, the ISR 881. IP address was statically configured on the laptop.
  • I had access to the ISR 4451 and the switch that connected all the devices in the 172.16.1.0/24network
  • None of the equipment was connected to a term server or power management server to reload them remotely.

Topology:

Incorrect route added on the ISR 881: ip route 10.150.217.1 255.255.255.255 172.16.1.3

Well, RTP, North Carolina had gone to bed many hours ago so, I had no one to ping who was sitting at work who could just reload the router for me.  San Jose too had long gone home for the night.  My only thought was Australia must be awake and decided to ping Phil Petty, our software engineer based in Australia.  It was a wonderful morning time for him and responded immediately to my jabber ping.

With so much hope, I asked him for Prime Infrastructure credentials so I could add the IP address 10.150.217.107 of the 881 to Prime and somehow figure out a way to reboot the router. Though the device got added using SNMP, CLI access failed as telnet and ssh from Prime failed as the lines were locked up (I cursed myself for not looking at the issue when the router denied telnet or ssh when another person was already using a line).  Device discovery worked as the SNMP string (read/write) was in place from previous tests with Prime Infrastructure.

Curious Phil asked me what I was working on and when I got done explaining what I was doing over jabber, he too got hooked on to the problem.  There's got to be a way to undo the route statement that I added and gain VNC back to the laptop.  I didn't want to drive to work so late in the night even thought it is only a 10 min drive for me to get to work. 

I was thinking about one of my buddies in TAC who decided to write a script that would answer a survey for him instead of clicking the radio/square buttons himself.  Who does that?? Here I am, instead of driving to work and being done in 20 min. I am breaking my head to find a way to do this without having to drive and without having to reload the router.  This is what makes us great engineers; never give up and find all options to solve the problem.

I read Phil typing, "Kureli, I think I can make this work...." He sounded pretty confident. Woo Hoo...! I was on board with his idea.  Within minutes I was able to establish VNC back to the laptop behind the ISR 881 from home and continued working until the wee hours of the morning.

You all hate me don't you?  For not telling you what exactly we did to undo the incorrect route on the ISR 881? 

Solution:

I will update this blog and add the solution to the problem in about a week...keep guessing until then.....

If you can't wait, unicast me your answer and I will tell you if you are headed in the right direction. 

Here is the most awaited solution:

Phil simply googled and found this link and told me, "Kureli, I think I can make this work."
http://www.cisco.com/c/en/us/support/docs/ip/simple-network-management-protocol-snmp/15217-copy-configs-snmp.html

My jabber conversation with Phil went sorta like this:

Phil: Kureli, you have access to the 4451 right? 

Me: Yes.

Phil: Then, could you negate the incorrect route that you added, put it in a text file and upload it to the ISR 4451's flash?

Me: Yes absolutely.

I wrote a one liner file below and tftp-ed it to the 4451 router's flash from my home laptop:
ISR-4451#more flash:no-route.txt
no ip route 10.150.217.1 255.255.255.255 172.16.1.3

Next, we made the ISR 4451 the TFTP server. Then Phil ran an SNMP command against the ISR 881 to go and download the "one liner" from the ISR 4451 and apply it to its running config.

mcp-bld-syd-01:~>snmpset -v2c -c public 10.150.217.107 .1.3.6.1.4.1.9.9.96.1.1.1.1.2.52 i 1 .1.3.6.1.4.1.9.9.96.1.1.1.1.3.52 i 4 .1.3.6.1.4.1.9.9.96.1.1.1.1.4.52 i 1 .1.3.6.1.4.1.9.9.96.1.1.1.1.5.52 a "10.150.217.132" .1.3.6.1.4.1.9.9.96.1.1.1.1.6.52 s "bootflash:no-route.txt" .1.3.6.1.4.1.9.9.96.1.1.1.1.14.52 i 4
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.2.52 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.3.52 = INTEGER: 4
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.4.52 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.5.52 = IpAddress: 10.150.217.132
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.6.52 = STRING: "bootflash:no-route.txt"
SNMPv2-SMI::enterprises.9.9.96.1.1.1.1.14.52 = INTEGER: 4
mcp-bld-syd-01:~>

Voila! The incorrect route got deleted and I got VNC back to the laptop behind the ISR 881.

10 Comments
estadlercisco
Beginner

Kureli I just want to say thank you. Without your expert web cast on FirePOWER for ISR and your Cisco Live BRKSEC-2507 Presentation I would have never figured out how to get FirePOWER configured on an ISR with a "fail-open" mechanism. thank you so much! I'm looking forward to seeing a full guide that explains how to do this with VRF.

p.s. how can I unicast you? I'm pretty sure I know the answer to this. The answer is in the puzzle :)

Kureli Sankar
Cisco Employee

Glad to hear. A full guide explaining how to do IPS with VRF is in the works.  However, you can easily implement it with the slides from BRKSEC-2057.  

Take a stab at solving the puzzle in the blog please... 

-Kureli

cepera1987
Beginner

Well NAT'ing VNC traffic from VPN client range (or even your specific address) to connected network of ISR and 3560 (172.16.1.x) would solve this. Asuming VPN router's primary path for 172.16.1.x network is via ISR and not 881, also assuming you have access to that router too to change that ;)

smandal
Cisco Employee

Quote: "This is what makes us great engineers; never give up and find all options to solve the problem".

and Together we win!  :)  

This was an instance of user created Chaos. In IT world, we have talked in length about unleashing Chaos Monkey to simulate intentional and unintentional user created chaos. Is there an equivalent of this tool in the Networking world?   I'm also involved in the ACI adoption from IT side so I do understand the traditional silos between various teams like Network, Storage and IT is slowly evaporating with the advents of SDN (Software Defined Networking) and ACI (Application Centric Infrastructure) is Cisco's answer to SDN.  

Would you like to be the engineer solving problems created by synthetic chaos system?  :)

Kureli Sankar
Cisco Employee

My VPN connection from home to Cisco network is via our VPN clusters in RTP, North Carolina. Once connected to Cisco network then, I am able to access the ISR 4451 sitting on my desk as it's outside address in the 10.150.217.32 is on the network.  Only way I can reach the laptop is if I changed the laptop's GW to ISR 4451 instead of the ISR 881 and provide proper STATIC NAT or STATIC PAT for VNC traffic on the ISR 4451. 

I need to be able to get back to the ISR 881....

-Kureli

Kureli Sankar
Cisco Employee

Why not? Seems like too much fun. 

You know I did that for 6 1/2 years as a TAC engineer working in the Firewall TAC team in RTP. The first best job I have ever had.  Extremely stressful but highly rewarding....

-Kureli

cepera1987
Beginner

Yes, so if you NAT'ed / PAT'ed your VPN range to 172.16.1.x subnet you wouldn't need to change gateway on your laptop as traffic was sourced locally from connected network on the ISR.

Kureli Sankar
Cisco Employee

One of my buddies decided to take a stab at it and he was on the right track as well but after a bit, he sent me this.... Thought I'd share.

-Kureli

Kureli Sankar
Cisco Employee

Very good idea. I could accomplish that even simpler by

  • Shutting the port on the switch connected to the ISR 881 and assigning the same 172.16.2.1 IP address to the ISR4451.
  • Configure STATIC PAT for the laptop on the ISR 4451 for VNC (tcp 5900).

I can certainly get back on the laptop that was consoled into the ISR 4451 but telnet session open with the ISR 881.

This still doesn't provide access back to the ISR 881 to get my entire topology working for my slides :(

-Kureli

cepera1987
Beginner

Shuting down the LAN wont give you access to 881 but NAT will as once again you are able to get to laptop which has telnet session to 881

Content for Community-Ad

This widget could not be displayed.