No access to inside Interface, but restart fixes access

[gcarson73]

I have a Cisco 2951 Router, Version 15.0(1)M3.  It has been in production for 8 months, with no issues.

In the past week, I have had to reboot it twice.  Every single server, laptop, and VoiP phone went down (no internet), I was not able to ping the inside interface.

I was able to console into it, and I could ping both Outside and inside interface, and I could ping 8.8.8.8.

I could find no issue with the router, and only way I have found to fix it, was to reboot the router.

Anyone have an idea?

[Georg Pauwen]

Hello,

can you enable 'debug ip nat detailed' ?

You might want to try and lower the NAT translation timeout to something like 60 seconds:

2951(config)#ip nat translation timeout 60

[Richard Burts]

If I am understanding the original post correctly it says that devices inside were not able to ping the router interface. I am not clear how that would be a nat issue.

The original post says the router has been in production for 8 months. Have there been any changes recently (changes on the router or changes in devices connected to the router)?  Please be sure that logging is enabled and if the problem happens again please check the log to see if there are any messages that shed light on the problem.

If the problem happens again would you be able to test and see if the router can ping anything that is connected on the inside network? If the problem happens again would you post the output of show arp (or perhaps show ip arp depending on the platform).

HTH

Rick

[gcarson73]

Yes, logging is enabled.  The system I have logging set to,  just happens to be my laptop.  I was unable to ping the router, INT G0/1, which is the Gateway for the network.

I am pretty sure I tried to ping my Domain Controllers from the router with no success, I will attempt again if it happens again.  

I am looking for any ideas, since no changes have been made to the router (small business and I am the only IT guy).  I will get the output of "show arp" or "show ip arp", if this happens again.

Anyone else have any ideas, please share.

[Richard Burts]

It is good to know that logging is enabled. So one of the questions would be whether your laptop was receiving log messages during the time that this issue was going on? If so were there any unexpected log messages? But if you can not ping the router I would not be surprised if the router were not able to send log messages. So if your laptop was not receiving log messages then the next time that the problem occurs and you are logged in on the console do a show log and look for unexpected messages (this assumes that logging buffered is enabled).

When I read the original post I was thinking about things that might allow the router to run for a period of time and then to interrupt connectivity. I thought about things such as a memory leak. But issues like that should also impact the ability of the router to ping to Internet resources. So I am thinking about things that might impact the inside connectivity but not the outside. And arp seems to be the most likely thing. So it is what we should investigate first.

HTH

Rick

[gcarson73]

sorry, I looked at the log and did not see anything odd.  I only see the "coldstart"; router reboot.  I have attached a screen shot of the log. 

I have turned on "logging buffered informational", with a screen shot from Router.

[Richard Burts]

Thanks for the screenshots. With logging buffered enabled you should be able to use the show log command from your console session while the problem is going on and to see log messages while the problem is happening.

It is interesting that the log from your laptop shows only the two reboots of the router and not anything in between. This does not prove anything though, since the problem that prevents ping to the router from inside may very well also impact attempts by the router to send packets to devices inside.

HTH

Rick

[gcarson73]

My issue "seems" to have been a switch!  A Netgear JGS524PE

*The network went down and a Router restart would NOT fix the issue.  

I eventually re-moved the above switch and the network came back up.  I moved the switch over to my "test" network, and have not had any issues with it.

*as an FYI, I have the following gigabit switches in my Production environment

24 Port Netgear-Model GS728TP (all switches plugged into this switch)

24 Port Cisco-Model 3750

8 Port Dell-Model 2708

4 Port Wireless Access point-Model FIOS-G1100

[Richard Burts]

Thanks for the update. Clearly you had some recent incident and perhaps that incident was caused by the Netgear switch. But I wonder if this incident is related to the previous ones. If I am reading the thread correctly in the previous instances a reboot of the router did fix the problem. but this time reboot of the router did not fix the problem. So perhaps this time it was a slightly different problem?

HTH

Rick

[paul driver]

Hello

Is this rtr connecting through a fw ?

Have you checked for licensing issues

Also without rebooting try clearing the arp cache and see if connections reestablished

regards 

[gcarson73]

No Firewall, I know...I know.  Small business and next year I will be adding an ASA or SonicWall.

if/when it happens again, I will first get a copy of "show arp" and "show ip arp".  I will then issue a "clear arp int g0/1" and "clear ip cache" to see if this resolves it.  

[Richard Burts]

What you have identified here are reasonable steps. I would be very surprised if clear ip cache made any difference. But I see no harm in trying it. I would also suggest that before you do the clear commands that you do show log and post any messages found around the time that the problem started and while it was happening.

HTH

Rick