06-06-2007 06:57 PM - edited 03-03-2019 05:20 PM
Hi everybody,
We are currently experiencing problems at our floors in Melbourne.
I will try to describe the topology of the network in brief,
Building A
OPTUS Ethernet 5M (Internet) --> fas0/0(3700)fas0/1 --> Switch Stack 3550 --> PC
Building B
(A) (3700)fas2/0 --> Ethernet 4M --> (B) fas0/0(3700)fas0/1 --> Switch Stack 3550 --> PC
IOS on all Cisco hardware have been updated.
The distance between the floors are 1.5 KM apart. Basic terms building A which is connected to the OPTUS Internet is hub and building B is spoke.
Building B is currently having Internet browsing issue which affects some users at a time and not all user. The time and users affected change day after day. The affected user is unable to browse to any URL but another user on the same floor can. The problem has affected all browsing, outlook, msn, terminal service. Building A is affected aswell but not as much as the other
Here are the tasks I have completed to diagnose this issue:
* Provide support to team and clients for Internet related problem during business hours. When traceroute to www or IP, it stops at the Building A 3700 fas2/0.
* Replace all trunk cables to and from the Internet Gateway after hours
* Liaise with Optus to test the Internet (A) and point to point circuit (B)
* Apply new DNS on the gateway identical for both A and B
* Check event viewer during the time it occur, nothing
* Ping internal IP (OK) and external IP (stops at Building A fas2/0) address during the event
* Another situation is ping internal IP (OK) and ping/tracert external IP (www.google.com) OK. Very strange.
Thanks. Sorry for the lengthy body. His forum has help me in the pass and thought Id give it a try.
BTW, release /renew IP or restart of the users PC fixes the problem.
DHCP is a /25 and theres only 10 of us in the network using the DHCP pool.
It happens constantly and varies on user. One user would be affected and the rest is fine browsing the Internet, the same day or next a different user.
What I also found is the Internet Link is congested peaking to max limit speed. But this should only cause the users to have slow Internet connection but not drop it?
06-06-2007 07:22 PM
Building A#sh ver
Cisco IOS Software, 3700 Software (C3725-ADVSECURITYK9-M), Version 12.4(7c), RELEASE SOFTWARE (fc2)
Building B#sh ver
Cisco IOS Software, 3700 Software (C3725-ADVSECURITYK9-M), Version 12.4(7c), RELEASE SOFTWARE (fc2)
System image file is "flash:c3725-advsecurityk9-mz.124-7c.bin"
06-06-2007 09:02 PM
Hi Valdesp,
can you giv ur topology diagram, so that i can understand better based on ur post.
06-06-2007 09:18 PM
06-06-2007 09:32 PM
Any firewall or Access-list configured on the network? lemme know the switch configuration also posted.
if i am not wrong, only one side the internet is there & there side users access internet via the main switch isn't? so as you mentioned, only one side users are affected in accessing the internet,outlook,msn etc... or both the side users faces the similar kind of problem.
06-06-2007 09:50 PM
When this situtation happens only the Building B users are affected and some of the building A users as well. What happens when you try to ping the external IP from the building A ?
When this problem happens, did you check the link status/untilization of both the 5MB link and 4MB link on both the routers. Do you see some errors, drops etc...
During the problem, Did you try monitoring the local traffic on the switch in BuildingB. It could be a worm/virus attack which is creating the problem and chokes the WAN link and effects all the users on siteB. Please setup a local port mirroring session during the problem and it should reveal problem.
Did you check the switch utilization and logs for the switch at SiteB during the problem.
I believe that setting up the local port/vlan span sesssion on switches during the problem will help us diagnosing the issue.
HTH,
-amit singh
06-06-2007 10:49 PM
Hi,
Thanks for your replies.
From what I have seen this week, users from both buildings would get affected but not at the same time. 20% combine users from both buildings would get this problem and the rest of the 80% would work fine on both application.
On the show logging, "%IP_VFR-4-FRAG_TABLE_OVERFLOW: FastEthernet0/0: the fragment table has reached its maximum threshold 16" is shown almost every day.
Could this cause the internet drop?
06-06-2007 10:51 PM
%IP_VFR-4-FRAG_TABLE_OVERFLOW (x1): [chars]: the fragment table has reached
its maximum threshold [dec]
Explanation: The number of datagrams being reassembled at any one time has reached
it maximum limit.
Recommended Action: Increase the maximum number of datagrams that can be reassembled
by entering the ip virtual-reassembly max-reassemblies number command, with number
being the maximum number of datagrams that can be reassembled at any one time.
06-06-2007 10:53 PM
Ive set the max number to 32.
Could this cause the internet drop affecting only some users at both sites?
06-06-2007 11:18 PM
Please read the link below for virtual assembly feature:
http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/products_feature_guide09186a00802299fb.html
When this feature is enabled it puts a litlle overhead on the router and could result the sysmptoms that you having but not 100% sure.
I would reuqest to you monitor the CPU utilization when this happens on the routers.
Also Set up a sniffer off of the interface that's reporting the error to capture the fragmented packets and find out where they are coming from. This could also be some type of buffer overflow attack generated by some of the host on the network. To troubleshoot it futther try setting up the sniffer on the network.
-amit singh
06-07-2007 10:18 PM
Hi, Thanks guys for the support.
So far the processes cpu has been low. I will continue to monitor this router duing peak time to make sure it doesn't cause any problems on the routers performance. Cisco.com did state that this will cause a performance impact once enabled.
SVC-M40-GW1#sh processes cpu | ex 0.00
CPU utilization for five seconds: 14%/13%; one minute: 13%; five minutes: 12%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
2 132 13632 9 0.08% 0.02% 0.02% 0 Load Meter
74 820924 2870622 285 0.24% 1.81% 1.49% 0 IP Input
116 3256 140121 23 0.08% 0.03% 0.02% 0 DHCPD Receive
186 84 1232 68 0.32% 0.06% 0.01% 99 Virtual Exec
Also I want to make sure this is a permanent fix for the intermittent Internet drops affected a number of clients at one time, which is a very strange problem. I would still like to know how this problem has anything to do with the fragmentation?
06-08-2007 10:23 PM
Before I made the additional config on the interface, I also cleared the counters.
After that, I've got 163 input errors. The fastEthernet is hard set to 10M/Full connected to a 5M link to the Internet. ISP can't see anything that relates to this errors on their switch but are seeing alot of congestion. Their switch which connects to this interface is also hard set to 10M/Full.
Any ideas?
GW1#sh int fastEthernet 0/0
FastEthernet0/0 is up, line protocol is up
Hardware is Gt96k FE, address is 0012.0158.b550 (bia 0012.0158.b550)
Description: Link To Optus - 5Mbps Ethernet
Internet address is x.x.x.x/30
MTU 1500 bytes, BW 5120 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/8523/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 22000 bits/sec, 17 packets/sec
30 second output rate 15000 bits/sec, 14 packets/sec
20918313 packets input, 491881396 bytes
Received 16 broadcasts, 0 runts, 0 giants, 0 throttles
163 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
18459838 packets output, 1142112889 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide