Client's losing Internet connectivity intermittently and at random intervals

[blaine.robbins]

Hello Cisco Community!

My two colleagues and I manage over 500 devices across the greater part of North Georgia. In one of the county governments that we manage, we are having a strange problem.

A large number of users are reporting that their machines lose Internet connectivity at random times during the day and for only an hour or so, until the issue resolves itself. Now, we have done a copious amount of troubleshooting on the machines themselves and haven't found a single issue with their network cards, configuration, or even their Layer 1 backbone. It's also important to mention that the client's have total LAN connectivity while the problem is occurring. They can communicate with their DNS server, Gateway, and other machines within their local subnet. The issue seems to be within our network infrastructure. 

My colleagues and I all have our own idea of what is causing this issue. However, none of us can seem to come to an agreement on where to actually begin to devote time and resources into troubleshooting.

In my opinion, the firewall that all of the users go through for their Internet access is experiencing oversubscription. As simple as it is to check the firewall to determine if this happening, my colleagues are certain that this isn't the issue because of the size of the firewall we have installed. 

Has anybody ran into an issue like this before? Anyone have an opinion on what may be our problem?

Thank you all for your feedback!

Blaine Robbins

IT Technician

[blaine.robbins]

Also, we haven't had any recent changes to the network infrastructure recently, besides a new VM server, and all of the network equipment was installed in 2010. 

[Leo Laohoo]

A large number of users are reporting that their machines lose Internet connectivity at random times during the day and for only an hour or so

Does this event happen daily?  If it does, can you get a few clients to constantly ping the default gateway?  

You'll need to provide more information than this: 

1.  How often does this happen during the day?  What times?  During lunchtime? 

2.  What is your edge WAN equipment?  What is the WAN bandwidth?

[paul driver]

Hello

Can you inticiate a wireshark capture on a problematic pc and post the pacap file

res

paul

[Tinashe Ndhlovu]

Probably congestion on the internet link.. what time does this happen? check to see if no other bandwidth hogging processes are also occurring at the same time like backups or something, and do these machines completely lose internet connectivity or the web browsing is slow... if it completely drops then u might want to check with your ISP just in case they are running some scheduled task.. or do you have a proxy that blocks internet connectivity at particular times.. or maybe a time-based ACL ? look at usage on the firewall at the time internet goes down and find out the top users if any

[blaine.robbins]

We have a 20 Mbps up/down fiber backbone for this particular network. That's the thing that is so puzzling.. The entire network infrastructure is designed to where it could handle 10x more traffic. 

We don't have a proxy.. We've checked the usage on the firewall.

NJC-ASA5520# sh cpu

CPU utilization for 5 seconds = 3%; 1 minute: 4%; 5 minutes: 4%

NJC-ASA5520#

NJC-ASA5520#

NJC-ASA5520#

NJC-ASA5520#

NJC-ASA5520# sh process cpu-hog

Process:      snmp, PROC_PC_TOTAL: 2, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   11:40:19 EST Dec 2 2014

PC:           8c63b18 (suspend)

Process:      snmp, NUMHOG: 2, MAXHOG: 2, LASTHOG: 2

LASTHOG At:   11:40:19 EST Dec 2 2014

PC:           8c63b18 (suspend)

Call stack:   8b877b3  8b67aed  8b6684c  8063bb3

Process:      Unicorn Admin Handler, PROC_PC_TOTAL: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   15:10:53 EST Dec 16 2014

PC:           8c3f47b (suspend)

Process:      Unicorn Admin Handler, NUMHOG: 1, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   15:10:53 EST Dec 16 2014

PC:           8c3f47b (suspend)

Call stack:   8c3f882  938386d  84426cb  8439959  8439d8e  843a0b4  8440b7d

              8063bb3

Process:      ssh, PROC_PC_TOTAL: 2, MAXHOG: 5, LASTHOG: 4

LASTHOG At:   15:22:27 EST Dec 16 2014

PC:           8befc0c (suspend)

Process:      ssh, NUMHOG: 2, MAXHOG: 5, LASTHOG: 4

LASTHOG At:   15:22:27 EST Dec 16 2014

PC:           8befc0c (suspend)

Call stack:   8befc0c  8bfe989  8bf4545  8bf49e5  8bf4fa3  88acb90  88ace5e

              9386bd1  8550cc2  89cba88  89cbb8d  88b4ccf  88ade50  898b7ec

Process:      Dispatch Unit, PROC_PC_TOTAL: 9, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   09:04:07 EST Feb 12 2015

PC:           81aab94 (suspend)

Process:      Dispatch Unit, NUMHOG: 9, MAXHOG: 3, LASTHOG: 3

LASTHOG At:   09:04:07 EST Feb 12 2015

PC:           81aab94 (suspend)

Call stack:   81aab94  8063bb3

Process:      NIC status poll, PROC_PC_TOTAL: 23, MAXHOG: 16, LASTHOG: 12

LASTHOG At:   16:11:08 EST Feb 25 2015

PC:           89212e0 (suspend)

Process:      NIC status poll, NUMHOG: 23, MAXHOG: 16, LASTHOG: 12

LASTHOG At:   16:11:08 EST Feb 25 2015

PC:           89212e0 (suspend)

Call stack:   89212e0  8063bb3

Process:      Dispatch Unit, PROC_PC_TOTAL: 83020, MAXHOG: 9, LASTHOG: 3

LASTHOG At:   10:40:52 EDT Apr 3 2015

PC:           81aad7f (suspend)

Process:      Dispatch Unit, NUMHOG: 41009, MAXHOG: 9, LASTHOG: 3

LASTHOG At:   10:40:52 EDT Apr 3 2015

PC:           81aad7f (suspend)

Call stack:   81aad7f  8063bb3

Process:      Dispatch Unit, PROC_PC_TOTAL: 155486, MAXHOG: 122, LASTHOG: 3

LASTHOG At:   14:41:49 EDT Apr 3 2015

PC:           81aae69 (suspend)

Process:      Dispatch Unit, NUMHOG: 118438, MAXHOG: 122, LASTHOG: 3

LASTHOG At:   14:41:49 EDT Apr 3 2015

PC:           81aae69 (suspend)

Call stack:   81aae69  8063bb3

CPU hog threshold (msec):  2.844

Last cleared: None

NJC-ASA5520#

NJC-ASA5520#

NJC-ASA5520# sh inter

NJC-ASA5520# sh interface | inc errors

        5768 input errors, 0 CRC, 0 frame, 5768 overrun, 0 ignored, 0 abort  - The dropped packets are most likely do drops on power to the other devices. NJC-ASA5520 up 162 days 20 hours

        0 output errors, 0 collisions, 1 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 1 interface resets

        1780 input errors, 0 CRC, 0 frame, 1780 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 25 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 1 interface resets

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 output errors, 0 collisions, 0 interface resets

NJC-ASA5520# sh traffic

INSIDE:

        received (in 1184790.032 secs):

                5035710554 packets      1121324725788 bytes

                4000 pkts/sec   946001 bytes/sec

        transmitted (in 1184790.032 secs):

                5918415604 packets      3111059086530 bytes

                4002 pkts/sec   2625001 bytes/sec

      1 minute input rate 247 pkts/sec,  28576 bytes/sec

      1 minute output rate 289 pkts/sec,  130385 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 395 pkts/sec,  84281 bytes/sec

      5 minute output rate 464 pkts/sec,  303575 bytes/sec

      5 minute drop rate, 0 pkts/sec

OUTSIDE:

        received (in 1184790.042 secs):

                5932388107 packets      3354415166779 bytes

                5003 pkts/sec   2831003 bytes/sec

        transmitted (in 1184790.042 secs):

                5036315015 packets      1346610744749 bytes

                4000 pkts/sec   1136001 bytes/sec

      1 minute input rate 294 pkts/sec,  143256 bytes/sec

      1 minute output rate 253 pkts/sec,  40286 bytes/sec

      1 minute drop rate, 1 pkts/sec

      5 minute input rate 465 pkts/sec,  316605 bytes/sec

      5 minute output rate 396 pkts/sec,  95572 bytes/sec

      5 minute drop rate, 1 pkts/sec

management:

        received (in 1184790.912 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 1184790.912 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

----------------------------------------

Aggregated Traffic on Physical Interface

----------------------------------------

GigabitEthernet0/0:

        received (in 1184790.922 secs):

                5035717154 packets      1218192879514 bytes

                4000 pkts/sec   1028000 bytes/sec

        transmitted (in 1184790.922 secs):

                5918415932 packets      3222855268097 bytes

                4002 pkts/sec   2720000 bytes/sec

      1 minute input rate 247 pkts/sec,  33516 bytes/sec

      1 minute output rate 289 pkts/sec,  135880 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 395 pkts/sec,  92375 bytes/sec

      5 minute output rate 464 pkts/sec,  312384 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/1:

        received (in 1184791.352 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 1184791.352 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/2:

       received (in 1184791.352 secs):

                5932392393 packets      3462385869560 bytes

                5003 pkts/sec   2922000 bytes/sec

        transmitted (in 1184791.352 secs):

                5036294721 packets      1441725565833 bytes

                4000 pkts/sec   1216001 bytes/sec

      1 minute input rate 294 pkts/sec,  148600 bytes/sec

      1 minute output rate 253 pkts/sec,  45224 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 465 pkts/sec,  325200 bytes/sec

      5 minute output rate 396 pkts/sec,  103560 bytes/sec

      5 minute drop rate, 0 pkts/sec

GigabitEthernet0/3:

        received (in 1184791.752 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 1184791.752 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

Management0/0:

        received (in 1184791.762 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

        transmitted (in 1184791.762 secs):

                0 packets       0 bytes

                0 pkts/sec      0 bytes/sec

      1 minute input rate 0 pkts/sec,  0 bytes/sec

      1 minute output rate 0 pkts/sec,  0 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 0 pkts/sec,  0 bytes/sec

      5 minute output rate 0 pkts/sec,  0 bytes/sec

      5 minute drop rate, 0 pkts/sec

NJC-ASA5520#   sh conn count

571 in use, 6019 most used

NJC-ASA5520# sh resour

NJC-ASA5520# sh resource usage

Resource              Current         Peak      Limit        Denied Context

SSH                         1            2          5             0 System

Syslogs [rate]             16        10677        N/A             0 System

Conns                     572         6019     280000             0 System

Xlates                    508         9756        N/A             0 System

Hosts                    2027         7942        N/A             0 System

Conns [rate]                4         2083        N/A             0 System

Inspects [rate]             1         2331        N/A             0 System

One interesting thing to note is that we're now having an issue with VoIP phones at a location that is connected to the main network through a VPN tunnel.. The phones intermittently go down while all of the machines at the said location still have internet access.. 

One theory that I've thought of is that there may be an issue with our QoS configuration.. While we have data and voip separated with VLAN's, they're still on the same switch. Could that be causing machines and voip phones to lose Internet access for 30-45 min intervals intermittently at random times of the day?

Also I should note that when the machines are losing internet connectivity they can ping literally anything within our LAN, but absolutely nothing outside of that. We've managed to run a tracert on one of the machines while the problem was occurring..

Connection-specific DNS Suffix . : lcsrvpdc.****
Link-local IPv6 Address . . . . . : fe80::50d3:3483:2fa4:ab33%11
IPv4 Address. . . . . . . . . . . : 10.134.10.118
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.134.10.1

C:\Users\administrator>tracert www.bing.com

Tracing route to any.edge.bing.com [204.79.197.200]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 10.134.10.1
2 1 ms 1 ms 2 ms 192.168.134.1
3 * * * Request timed out.
4 * * ^C

Any ideas guys? Also, thank you all so much for your correspondence so far! Sorry for not getting back with a response sooner!

[s.aun_ali]

Well your outside interface has traffic more than 20 Megabits per second in the stats ....

If this is for internet traffic, then no wonder you are chocking your internet bandwidth ....

HTH