Client's losing Internet connectivity intermittently and at random intervals

blaine.robbins · ‎04-02-2015

Hello Cisco Community!

My two colleagues and I manage over 500 devices across the greater part of North Georgia. In one of the county governments that we manage, we are having a strange problem.

A large number of users are reporting that their machines lose Internet connectivity at random times during the day and for only an hour or so, until the issue resolves itself. Now, we have done a copious amount of troubleshooting on the machines themselves and haven't found a single issue with their network cards, configuration, or even their Layer 1 backbone. It's also important to mention that the client's have total LAN connectivity while the problem is occurring. They can communicate with their DNS server, Gateway, and other machines within their local subnet. The issue seems to be within our network infrastructure.

My colleagues and I all have our own idea of what is causing this issue. However, none of us can seem to come to an agreement on where to actually begin to devote time and resources into troubleshooting.

In my opinion, the firewall that all of the users go through for their Internet access is experiencing oversubscription. As simple as it is to check the firewall to determine if this happening, my colleagues are certain that this isn't the issue because of the size of the firewall we have installed.

Has anybody ran into an issue like this before? Anyone have an opinion on what may be our problem?

Thank you all for your feedback!

Blaine Robbins

IT Technician

blaine.robbins · ‎04-02-2015

Also, we haven't had any recent changes to the network infrastructure recently, besides a new VM server, and all of the network equipment was installed in 2010.

Leo Laohoo · ‎04-02-2015

A large number of users are reporting that their machines lose Internet connectivity at random times during the day and for only an hour or so

Does this event happen daily? If it does, can you get a few clients to constantly ping the default gateway?

You'll need to provide more information than this:

1. How often does this happen during the day? What times? During lunchtime?

2. What is your edge WAN equipment? What is the WAN bandwidth?

paul driver · ‎04-03-2015

Hello

Can you inticiate a wireshark capture on a problematic pc and post the pacap file

res

paul

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Tinashe Ndhlovu · ‎04-07-2015

Probably congestion on the internet link.. what time does this happen? check to see if no other bandwidth hogging processes are also occurring at the same time like backups or something, and do these machines completely lose internet connectivity or the web browsing is slow... if it completely drops then u might want to check with your ISP just in case they are running some scheduled task.. or do you have a proxy that blocks internet connectivity at particular times.. or maybe a time-based ACL ? look at usage on the firewall at the time internet goes down and find out the top users if any

blaine.robbins · ‎04-10-2015

We have a 20 Mbps up/down fiber backbone for this particular network. That's the thing that is so puzzling.. The entire network infrastructure is designed to where it could handle 10x more traffic.

We don't have a proxy.. We've checked the usage on the firewall.

NJC-ASA5520# sh cpu

CPU utilization for 5 seconds = 3%; 1 minute: 4%; 5 minutes: 4%

NJC-ASA5520#

NJC-ASA5520# sh process cpu-hog

Process: snmp, PROC_PC_TOTAL: 2, MAXHOG: 2, LASTHOG: 2

LASTHOG At: 11:40:19 EST Dec 2 2014

PC: 8c63b18 (suspend)

Process: snmp, NUMHOG: 2, MAXHOG: 2, LASTHOG: 2

LASTHOG At: 11:40:19 EST Dec 2 2014

PC: 8c63b18 (suspend)

Call stack: 8b877b3 8b67aed 8b6684c 8063bb3

Process: Unicorn Admin Handler, PROC_PC_TOTAL: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 15:10:53 EST Dec 16 2014

PC: 8c3f47b (suspend)

Process: Unicorn Admin Handler, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 15:10:53 EST Dec 16 2014

PC: 8c3f47b (suspend)

Call stack: 8c3f882 938386d 84426cb 8439959 8439d8e 843a0b4 8440b7d

8063bb3

Process: ssh, PROC_PC_TOTAL: 2, MAXHOG: 5, LASTHOG: 4

LASTHOG At: 15:22:27 EST Dec 16 2014

PC: 8befc0c (suspend)

Process: ssh, NUMHOG: 2, MAXHOG: 5, LASTHOG: 4

LASTHOG At: 15:22:27 EST Dec 16 2014

PC: 8befc0c (suspend)

Call stack: 8befc0c 8bfe989 8bf4545 8bf49e5 8bf4fa3 88acb90 88ace5e

9386bd1 8550cc2 89cba88 89cbb8d 88b4ccf 88ade50 898b7ec

Process: Dispatch Unit, PROC_PC_TOTAL: 9, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 09:04:07 EST Feb 12 2015

PC: 81aab94 (suspend)

Process: Dispatch Unit, NUMHOG: 9, MAXHOG: 3, LASTHOG: 3

LASTHOG At: 09:04:07 EST Feb 12 2015

PC: 81aab94 (suspend)

Call stack: 81aab94 8063bb3

Process: NIC status poll, PROC_PC_TOTAL: 23, MAXHOG: 16, LASTHOG: 12

LASTHOG At: 16:11:08 EST Feb 25 2015

PC: 89212e0 (suspend)

Process: NIC status poll, NUMHOG: 23, MAXHOG: 16, LASTHOG: 12

LASTHOG At: 16:11:08 EST Feb 25 2015

PC: 89212e0 (suspend)

Call stack: 89212e0 8063bb3

Process: Dispatch Unit, PROC_PC_TOTAL: 83020, MAXHOG: 9, LASTHOG: 3

LASTHOG At: 10:40:52 EDT Apr 3 2015

PC: 81aad7f (suspend)

Process: Dispatch Unit, NUMHOG: 41009, MAXHOG: 9, LASTHOG: 3

LASTHOG At: 10:40:52 EDT Apr 3 2015

PC: 81aad7f (suspend)

Call stack: 81aad7f 8063bb3

Process: Dispatch Unit, PROC_PC_TOTAL: 155486, MAXHOG: 122, LASTHOG: 3

LASTHOG At: 14:41:49 EDT Apr 3 2015

PC: 81aae69 (suspend)

Process: Dispatch Unit, NUMHOG: 118438, MAXHOG: 122, LASTHOG: 3

LASTHOG At: 14:41:49 EDT Apr 3 2015

PC: 81aae69 (suspend)

Call stack: 81aae69 8063bb3

CPU hog threshold (msec): 2.844

Last cleared: None

NJC-ASA5520#

NJC-ASA5520# sh inter

NJC-ASA5520# sh interface | inc errors

5768 input errors, 0 CRC, 0 frame, 5768 overrun, 0 ignored, 0 abort - The dropped packets are most likely do drops on power to the other devices. NJC-ASA5520 up 162 days 20 hours

0 output errors, 0 collisions, 1 interface resets

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 output errors, 0 collisions, 1 interface resets

1780 input errors, 0 CRC, 0 frame, 1780 overrun, 0 ignored, 0 abort

0 output errors, 0 collisions, 25 interface resets

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 output errors, 0 collisions, 1 interface resets

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 output errors, 0 collisions, 0 interface resets

NJC-ASA5520# sh traffic

INSIDE:

received (in 1184790.032 secs):

5035710554 packets 1121324725788 bytes

4000 pkts/sec 946001 bytes/sec

transmitted (in 1184790.032 secs):

5918415604 packets 3111059086530 bytes

4002 pkts/sec 2625001 bytes/sec

1 minute input rate 247 pkts/sec, 28576 bytes/sec

1 minute output rate 289 pkts/sec, 130385 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 395 pkts/sec, 84281 bytes/sec

5 minute output rate 464 pkts/sec, 303575 bytes/sec

5 minute drop rate, 0 pkts/sec

OUTSIDE:

received (in 1184790.042 secs):

5932388107 packets 3354415166779 bytes

5003 pkts/sec 2831003 bytes/sec

transmitted (in 1184790.042 secs):

5036315015 packets 1346610744749 bytes

4000 pkts/sec 1136001 bytes/sec

1 minute input rate 294 pkts/sec, 143256 bytes/sec

1 minute output rate 253 pkts/sec, 40286 bytes/sec

1 minute drop rate, 1 pkts/sec

5 minute input rate 465 pkts/sec, 316605 bytes/sec

5 minute output rate 396 pkts/sec, 95572 bytes/sec

5 minute drop rate, 1 pkts/sec

management:

received (in 1184790.912 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 1184790.912 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

----------------------------------------

Aggregated Traffic on Physical Interface

----------------------------------------

GigabitEthernet0/0:

received (in 1184790.922 secs):

5035717154 packets 1218192879514 bytes

4000 pkts/sec 1028000 bytes/sec

transmitted (in 1184790.922 secs):

5918415932 packets 3222855268097 bytes

4002 pkts/sec 2720000 bytes/sec

1 minute input rate 247 pkts/sec, 33516 bytes/sec

1 minute output rate 289 pkts/sec, 135880 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 395 pkts/sec, 92375 bytes/sec

5 minute output rate 464 pkts/sec, 312384 bytes/sec

5 minute drop rate, 0 pkts/sec

GigabitEthernet0/1:

received (in 1184791.352 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 1184791.352 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

GigabitEthernet0/2:

received (in 1184791.352 secs):

5932392393 packets 3462385869560 bytes

5003 pkts/sec 2922000 bytes/sec

transmitted (in 1184791.352 secs):

5036294721 packets 1441725565833 bytes

4000 pkts/sec 1216001 bytes/sec

1 minute input rate 294 pkts/sec, 148600 bytes/sec

1 minute output rate 253 pkts/sec, 45224 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 465 pkts/sec, 325200 bytes/sec

5 minute output rate 396 pkts/sec, 103560 bytes/sec

5 minute drop rate, 0 pkts/sec

GigabitEthernet0/3:

received (in 1184791.752 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 1184791.752 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

Management0/0:

received (in 1184791.762 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

transmitted (in 1184791.762 secs):

0 packets 0 bytes

0 pkts/sec 0 bytes/sec

1 minute input rate 0 pkts/sec, 0 bytes/sec

1 minute output rate 0 pkts/sec, 0 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 0 pkts/sec, 0 bytes/sec

5 minute output rate 0 pkts/sec, 0 bytes/sec

5 minute drop rate, 0 pkts/sec

NJC-ASA5520# sh conn count

571 in use, 6019 most used

NJC-ASA5520# sh resour

NJC-ASA5520# sh resource usage

Resource Current Peak Limit Denied Context

SSH 1 2 5 0 System

Syslogs [rate] 16 10677 N/A 0 System

Conns 572 6019 280000 0 System

Xlates 508 9756 N/A 0 System

Hosts 2027 7942 N/A 0 System

Conns [rate] 4 2083 N/A 0 System

Inspects [rate] 1 2331 N/A 0 System

One interesting thing to note is that we're now having an issue with VoIP phones at a location that is connected to the main network through a VPN tunnel.. The phones intermittently go down while all of the machines at the said location still have internet access..

One theory that I've thought of is that there may be an issue with our QoS configuration.. While we have data and voip separated with VLAN's, they're still on the same switch. Could that be causing machines and voip phones to lose Internet access for 30-45 min intervals intermittently at random times of the day?

Also I should note that when the machines are losing internet connectivity they can ping literally anything within our LAN, but absolutely nothing outside of that. We've managed to run a tracert on one of the machines while the problem was occurring..

Connection-specific DNS Suffix . : lcsrvpdc.****
Link-local IPv6 Address . . . . . : fe80::50d3:3483:2fa4:ab33%11
IPv4 Address. . . . . . . . . . . : 10.134.10.118
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.134.10.1

C:\Users\administrator>tracert www.bing.com

Tracing route to any.edge.bing.com [204.79.197.200]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 10.134.10.1
2 1 ms 1 ms 2 ms 192.168.134.1
3 * * * Request timed out.
4 * * ^C

Any ideas guys? Also, thank you all so much for your correspondence so far! Sorry for not getting back with a response sooner!

s.aun_ali · ‎04-19-2015

Well your outside interface has traffic more than 20 Megabits per second in the stats ....

If this is for internet traffic, then no wonder you are chocking your internet bandwidth ....

HTH