Cisco ASA 5505-trouble after weekend

[brettpenza]

We've got a Cisco ASA 5505 firewall in a high school setting.  About 600 users internally.  The network is usually running fine.  However, on Mondays, or this week Tuesday (after a 3 day weekend) the firewall gets very hard to find.  Since it is the only access to the internet, most users are denied access.  I can't even ping the firewall on most of the clients, though it is occasionally accessible.   All the equipment appears fine.  I try rebooting the switches and firewall.   Sometimes it gets me a few minutes of up time.   Through all of this difficulty, there is no problem accessing information internally (server folders and files, client signons, all fine).  

Strangely,  the next day (after I'm ready to start buying replacement equipment)  everything is working fine again.  

Someone told me that I should check the logs and go from there.  I will do that tomorrow (if I can connect again), but does this make any sense to anyone?   

How could a problem which appears so dire, become completely resolved the next day?  

Thanks.

[johnlloyd_13]

hi,

could you briefly provide your network setup? diagram?

is internet directly terminated on the 5505? what's the speed?

it could also due to internet bandwidth congestion. try monitoring the interface BW usage.

[brettpenza]

Hi, 

The network is pretty simple.  

We have a dedicated 100mg internet service from XO/Communication.

On the Cisco, we run a single external address in. 

This line goes right into the Cisco 5505 Firewall.  

From the Cisco (our internal 10.0.0.1) we go into a Cisco switch.  

We have several Cisco 25 port switches and all the internal clients are wired into them. 

All clients are validated through our Windows 2008 Servers (10.0.0.2 & 10.0.0.4) through DHCP  and then the servers point them to the gateway (10.0.0.1 -asa5505). 

I did run the log on the asa 5505 last night and received no errors. Of course, I was probably too late on that.   

This morning (a day later)  everything is working fine again.  

I also ran wireshark yesterday during the internet downtime,  collected stats and did not see any glaring DDOS or anything like that. 

The baffling thing is that if the cisco was configured incorrectly, why would it run perfectly all the time (except for the day back from a short break).

Thanks. 

[Richard Burts]

Are there any time based policies on the ASA? Do you have console access to the ASA? If so, you could use it to do show log while the issue is going on. It might also be helpful to get output for commands show arp, show route, and show xlate.

One possibility might be that over the weekend that the ASA has learned some invalic table entries and the invalid entries must time out before you have normal connectivity.

HTH

Rick

[Dean Romanelli]

This sounds to me like your ASA could be host-locked. The 5505's are sold by number of host machines. They sold them with unlimited hosts as well but you have to really know what you are looking for package-wise to ensure you get an unlimited host license. 

Once you max out the number of hosts your ASA supports, it will deny all other connections.  Once the host count drops below the host limit maximum, everything works again.  I am betting you have a few minutes of uptime after a reboot because it takes a few minutes for all of your hosts to pop back into ARP.

Do the following on your ASA and please post output:

show local-host

What you want to see is:

FW68Pittsburgh-5505# show local-host
Licensed host limit: Unlimited.

If you see anything other than that, you are probably host locked.  Below is an example of one of my host locked units:

FW31Basingstoke-5505# show local-host
Detected interface 'outside' as the Internet interface. Host limit applies to all other interfaces.
Current host count: 7, towards licensed host limit of: 50

Fortunately this site only has 20 client machines, but if the host count ever reached 50, the ASA would deny all new connections. The end result is that some clients at your site are fine and others are dead, which can send you on a wild goose chase thinking you have a bad switch somewhere.  If your ASA shows this verbiage, you need to upgrade the license to UL-BUN-K9.

[brettpenza]

Thanks, 

A good thought, but we've got unlimited on this ASA  and over 90% of the time there is no problem on our network accessing the internet.  It is only on the return days back from low usage (weekends, holidays).

Thanks.

[brettpenza]

We've got licenses: unlimited. 

But it is a good thought.  Thanks.. 

[brettpenza]

Don't think there are any time based policies.  I will be checking logs next time this happens though. 

Thanks. 

[Richard Burts]

It is interesting that you can usually find a PC on the same switch that still has connectivity to the ASA while others do not. This would seem to eliminate an interface problem from being the issue. If an interface was not working then it should impact all devices and not just most devices.

Here is another thing that you can do during the next time that the problem is occurring - from a PC that is not able to access the ASA do a ping to the ASA and then immediately do arp -a then look to see if there is an entry for the address of the ASA. This is one way to find out if there is layer 2 connectivity between the PC and the ASA.

HTH

Rick

[Georg Pauwen]

Hello,

on a side note, I would start a ping -t on one of the client machines to keep the Internet connection up:

ping -t 8.8.8.8

This will ensure that the LAN and WAN interface on the ASA stay up all the time...maybe the problem is related to the inactivity you experience during those days...

[Richard Burts]

When I read the original post the first issue that I thought about was the limitation on number of hosts. The reason that I believe this is not the issue is that the network is described as running fine most of the time, with 600 users, which certainly should be an issue unless the ASA does have the unlimited license. It will be good to have the output from the original poster to confirm the license information.

The possibility of a problem with interfaces is an interesting observation. If it is an interface issue then I am guessing that it would be the inside interface since the original post indicates that it is not possible to ping the ASA while the problem in going on. If the original poster does have console access it should be possible to verify interface status the next time that this issue occurs.

HTH

Rick

[brettpenza]

Original poster here. 

Yes, the ASA has unlimited internal hosts. 

It is difficult to ping or access the Cisco ASA during the internet downtime.  However, I can usually find a p.c on the same switch that will let me in to the ASA  through the ASDM software.   I will check the interface and logs when this happens to us again.  Hopefully, I'll be able to make the access and come up with some clues. Thanks. 

[brettpenza]

Yes, I have set a steady ping on a client p.c. during the internet "outage".   It will show replies but then will drop.  

I can assure everyone that there is a flood of activity.  As previously stated, there are about 600 students and 50 faculty that are regularly attempting to access the internet.   Not all at once of course, but the potential is there. 

[Georg Pauwen]

Hello,

I didn't mean to set the ping DURING the outage, but before the outage, to check if that prevents the outage.

That said, which code are you running on the ASA ? There could be a bug involved. And what is the uptime of the device ?

[brettpenza]

We're running on ASA :version 9.1(4)

Thanks.

I will give the pinging idea a try.

-Brett