Random loss of internet access

[Stanley Ramsey]

I have a domain with about 100 users connected via Cisco 2960 switches to a Cisco 5510 ASA for internet access. We use static IP addresses. Randomly, users will lose internet access. They can get to all local resources inside the network and can even ping the ASA address, but can't get to anything outside the network. I can change their IP address and everything works again.

It's weird and it's random. Any ideas??

[lukeoxley1]

I will attempt to help Stanley...

My first point, which may not be entirely helpful, is what benefits is using static IP addresses for end user devices giving you? My understanding is that all 100+/- end user's devices are given static IP addresses. Obviously servers/controllers etc. need static IPs to be permanently accessible but it is best practice to give roaming or end user client dynamic IPs. Reason being is if they are going offsite to their home or a hotel then that static IP will not work unless the subnet is overlapping. So they will switch to DHCP. Then when they return to work it will have to be set back up again with a static.

Also, running static IPs on end users devices (end users devices normally making up the bulk of your network) is just cat-nip to end up with duplicate IP addresses. Check your layer 3 device debugging and see if there are any ‘MAC-FLAP’ or duplicate/IP conflict messages. You'll probably also see the ARP table constantly updating the MAC for one IP over and over.

I'm guessing you break out to the internet via a NAT overload on the 5510, in which case having two of the same IPs on your inside LAN will enable them to be able to access local resources on the layer 2 infrastructure, they will also be able to access resources (allbeit a bit choppy) over the layer 3 infrastructure to your DMZ for example, but the NAT translations will get really messed up if it is trying to translate two flows to two machines with one IP and will probably just end up dropping most the packets.

I would suggest firstly checking for IP conflicts on your layer 3 and verifying the ARP table. Thereafter, enabling DHCP on the inside of your 5510 and getting rid of all the static IPs for a few days and seeing how it goes would be a good start.

A few questions if I am to help you further though.

- What is handling layer 3 for your inside network?

- Do all the users lose internet at the same time, or just at random times for random users?

- If you change their static IP from one in the range to another to enable it to work, what happens if you change it back to the original non-working one?

Look forward to hearing back and hope I can help.

Many thanks,

Luke

[Stanley Ramsey]

I'll try to address all your questions...

First, reason for static IP's is security and the policy predates me, so I just continue it. I keep very accurate records of which PC has which IP, so as to not duplicate, but when a PC is replaced, I'm thinking the ARP table might get confused as it is a new PC with an already used IP address. These PC's are permanently onsite and no one ever takes one home or to a hotel.

I'm not sure how to check the arp table (little bit new at this). And as far as layer 3, the only device that handles layer 3 on my network is the ASA.

As for internet loss, it is just random users individually, not everyone. And after I change his address, I can change it back the next day to what it originally was and it works fine.

Thanks for any help you can provide!

[lukeoxley1]

Thanks for that - a strange one, no doubt. If you replace a PC and give it the same IP as the old unit the ARP table should update as appropriate providing there is only one device with this IP.

It only gets messy when you have two devices with the same IP address. The ARP table is just a table that holds each devices MAC address and what IP it has, it gathers this data when packets are traversing the L3 device, much like a MAC address table on a switch. When two devices have the same IP, each time they transmit data the L3 device keeps getting updated with a new MAC address for that IP... almost like the following conversation of two different computers talking to the L3 device.

Computer A: I've got the IP X.X.X.X and my MAC is Y.Y.Y.Y

Computer B: NO, I've got the IP X.X.X.X and my MAC is B.B.B.B.

And so on...

Depending on how often this happens to each user, I would suggest doing a packet capture on one of the affected PC's NICs that is on the inside LAN to see exactly what is happening when it occurs. I would also not mind seeing the running configuration of your ASA - but please remember to remove all sensitive data. Don't worry my man, we will get it sorted.