06-01-2006
11:25 AM
- last edited on
03-25-2019
03:12 PM
by
ciscomoderator
I'm having a strange and frustrating problem with excessive latency (slow response time) to a regional office.
The connection is a 256k frame link with a 128k CIR. At the far end is a 2612 router and a 2950 switch with about a dozen connected nodes. During no-traffic hours our round-trip ping times from the home office to any device on the far end average about 50ms with peaks no higher than 70ms, which makes sense for about 1000 miles.
During times of traffic, when people are working on those workstations, we get occasional "bursts of slow", where round-trip times from the home office to the Vlan1 management interface of the far-end switch or to any workstation attached to the switch hit peaks of over 7,000ms (yes, that's seven seconds). At the same time, pings to either the serial interface or the Ethernet interface of the far-end router hit peaks not higher than 200ms. WAN bandwidth utilization during those times is under 60%.
We have a half-dozen other regional offices connected with similar or identical equipment and similar or identical configs, and don't see this problem in any others. We have replaced the router and the switch in this office (at different times) to no avail. We have built and re-built configs on the router and the switch, and have compared them line-by-line to those in other regional offices, to no avail. We capture syslogs from both the router and the switch and there are no interesting events in either. Neither the router nor the switch are memory-bound nor cpu-bound. Interface error counters on the switch are unremarkable. Interface stats on the router are unremarkable. We have had the telco check the WAN circuit up as far as our CSU/DSU, to rule out problems in the local loop, at the demarc or in the riser wiring (which we do not control), and they reported everything clean. We are scheduling a cabling contractor to come in and certify all the in-office runs, but beyond that, we have pretty well emptied the idea jar.
What would you check next?
06-01-2006 12:14 PM
i would check the frame-relay statistics....
how many discard eligible packets, FECNs, BECNs...check LMI stats as well while your at it.
these can show issues with the FR and/or the CIR if the problem is there. if so, an increase in the CIR and/or total FR link capacity could be in order.
as this is not a constant problem but only during rather heavy peak usuage times, it may not be a cabling issue.
also, did you state that the switch(s) stats show no collisions, runts, giants, xmit/rcv errors? (for any ports?) as well, check for deferred packets on the router LAN & WAN interface, could mean an oversubscribed interface capacity.
06-01-2006 12:53 PM
Do you have traffic shapeing enabled on the frame-relay interface(s) - both sides - HQ and remotes?
It's possible that you are either hitting "Committed Burst" and the the frames hit congestion and are discarded, and/or, you are pushing above Comitted Burst into Excessive Burst, and the frames are being policed (i.e., dropped) off the network.
Either of these can show as latency because the dropped frames cause re-transmission, which takes some time (first to realize the missing data, then to re-send it). Furthermore, the re-transmissions can add to the already congested circuit and cause more dropping ... which causes more re-transmissions, which causes more dropping ...(etc.).
With Frame-Relay traffic Shaping, the traffic is paced such that you do not violate Excess Burst (usually an automatic discard in the FR network) and "usually" don't violate Committed Burst to the level that you'd lose much traffic (but still get a little more than CIR from the circuit).
A quick search of Google or Cisco's main site should provide you with the documentation necessary to implement FRTS.
There's also a "dynamic" traffic shape that keys of of reception of FECN and BECN flags (scales back the traffic for a while if congestion is reported).
Good Luck
Scott
06-02-2006 05:51 AM
In response to Greg first.
FR stats are unremarkable. I reset all counters yesterday afternoon, and have had an "incident" already this morning. Here are the counters from the Ethernet and serial interfaces on the router.
Ethernet0/0 is up, line protocol is up
Hardware is AmdP2, address is
Internet address is
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:01:46
Input queue: 3/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 22000 bits/sec, 32 packets/sec
5 minute output rate 33000 bits/sec, 25 packets/sec
161826 packets input, 14918851 bytes, 0 no buffer
Received 4981 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
130781 packets output, 22939761 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial0/0 is up, line protocol is up
Hardware is PowerQUICC Serial
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 5/255, rxload 11/255
Encapsulation FRAME-RELAY IETF, loopback not set
Keepalive set (10 sec)
LMI enq sent 6130, LMI stat recvd 6130, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 1023 LMI type is CISCO frame relay DTE
Broadcast queue 0/64, broadcasts sent/dropped 14276/0, interface broadcasts 13
254
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:01:46
Input queue: 5/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 1/1000/64/0 (size/max total/threshold/drops)
Conversations 1/8/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
30 second input rate 72000 bits/sec, 57 packets/sec
30 second output rate 34000 bits/sec, 75 packets/sec
179668 packets input, 26780853 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
190904 packets output, 14953044 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
Serial0/0.100 is up, line protocol is up
Hardware is PowerQUICC Serial
Description:
Internet address is
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 5/255, rxload 11/255
Encapsulation FRAME-RELAY IETF
Continued in the next message...
06-02-2006 05:52 AM
Here are the interface stats on the switch port that the router connects to:
FastEthernet0/1 is up, line protocol is up
Hardware is Fast Ethernet, address is
MTU 1500 bytes, BW 100000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:31, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:03:30
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
136199 packets input, 24459293 bytes, 0 no buffer
Received 3826 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1023 multicast, 0 pause input
0 input packets with dribble condition detected
197890 packets output, 17738272 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Interface errors on the switch ports are negligible (sh int cou err), meaning mostly zeros with a couple of ports showing single-digit numbers. This is over approximately 18 hours since reset. Zero collisions, runts, giants.
As I mentioned earlier, bandwidth utilization during those times is not excessive, and pings to the far-end router are within normal expectations. That's why I think it's not link congestion, discards, etc.
Replying to Scott, this problem has occurred with FRTS turned on and with it turned off. It is currently off. But as noted above, bandwidth measurements during those times do not show congestion. Also, identically configured PVCs terminating in the same HQ circuit servicing other regional offices which have the same staffing and run the same mix of applications have never experienced this problem.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide