Sporadically excessive latency

gkuzmowycz · ‎06-01-2006

I'm having a strange and frustrating problem with excessive latency (slow response time) to a regional office.

The connection is a 256k frame link with a 128k CIR. At the far end is a 2612 router and a 2950 switch with about a dozen connected nodes. During no-traffic hours our round-trip ping times from the home office to any device on the far end average about 50ms with peaks no higher than 70ms, which makes sense for about 1000 miles.

During times of traffic, when people are working on those workstations, we get occasional "bursts of slow", where round-trip times from the home office to the Vlan1 management interface of the far-end switch or to any workstation attached to the switch hit peaks of over 7,000ms (yes, that's seven seconds). At the same time, pings to either the serial interface or the Ethernet interface of the far-end router hit peaks not higher than 200ms. WAN bandwidth utilization during those times is under 60%.

We have a half-dozen other regional offices connected with similar or identical equipment and similar or identical configs, and don't see this problem in any others. We have replaced the router and the switch in this office (at different times) to no avail. We have built and re-built configs on the router and the switch, and have compared them line-by-line to those in other regional offices, to no avail. We capture syslogs from both the router and the switch and there are no interesting events in either. Neither the router nor the switch are memory-bound nor cpu-bound. Interface error counters on the switch are unremarkable. Interface stats on the router are unremarkable. We have had the telco check the WAN circuit up as far as our CSU/DSU, to rule out problems in the local loop, at the demarc or in the riser wiring (which we do not control), and they reported everything clean. We are scheduling a cabling contractor to come in and certify all the in-office runs, but beyond that, we have pretty well emptied the idea jar.

What would you check next?

gpulos · ‎06-01-2006

i would check the frame-relay statistics....

how many discard eligible packets, FECNs, BECNs...check LMI stats as well while your at it.

these can show issues with the FR and/or the CIR if the problem is there. if so, an increase in the CIR and/or total FR link capacity could be in order.

as this is not a constant problem but only during rather heavy peak usuage times, it may not be a cabling issue.

also, did you state that the switch(s) stats show no collisions, runts, giants, xmit/rcv errors? (for any ports?) as well, check for deferred packets on the router LAN & WAN interface, could mean an oversubscribed interface capacity.

scottmac · ‎06-01-2006

Do you have traffic shapeing enabled on the frame-relay interface(s) - both sides - HQ and remotes?

It's possible that you are either hitting "Committed Burst" and the the frames hit congestion and are discarded, and/or, you are pushing above Comitted Burst into Excessive Burst, and the frames are being policed (i.e., dropped) off the network.

Either of these can show as latency because the dropped frames cause re-transmission, which takes some time (first to realize the missing data, then to re-send it). Furthermore, the re-transmissions can add to the already congested circuit and cause more dropping ... which causes more re-transmissions, which causes more dropping ...(etc.).

With Frame-Relay traffic Shaping, the traffic is paced such that you do not violate Excess Burst (usually an automatic discard in the FR network) and "usually" don't violate Committed Burst to the level that you'd lose much traffic (but still get a little more than CIR from the circuit).

A quick search of Google or Cisco's main site should provide you with the documentation necessary to implement FRTS.

There's also a "dynamic" traffic shape that keys of of reception of FECN and BECN flags (scales back the traffic for a while if congestion is reported).

Good Luck

Scott

gkuzmowycz · ‎06-02-2006

In response to Greg first.

FR stats are unremarkable. I reset all counters yesterday afternoon, and have had an "incident" already this morning. Here are the counters from the Ethernet and serial interfaces on the router.

Ethernet0/0 is up, line protocol is up

Hardware is AmdP2, address is

Internet address is

MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive not set

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 17:01:46

Input queue: 3/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue :0/40 (size/max)

5 minute input rate 22000 bits/sec, 32 packets/sec

5 minute output rate 33000 bits/sec, 25 packets/sec

161826 packets input, 14918851 bytes, 0 no buffer

Received 4981 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

130781 packets output, 22939761 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

Serial0/0 is up, line protocol is up

Hardware is PowerQUICC Serial

MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,

reliability 255/255, txload 5/255, rxload 11/255

Encapsulation FRAME-RELAY IETF, loopback not set

Keepalive set (10 sec)

LMI enq sent 6130, LMI stat recvd 6130, LMI upd recvd 0, DTE LMI up

LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0

LMI DLCI 1023 LMI type is CISCO frame relay DTE

Broadcast queue 0/64, broadcasts sent/dropped 14276/0, interface broadcasts 13

254

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 17:01:46

Input queue: 5/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: weighted fair

Output queue: 1/1000/64/0 (size/max total/threshold/drops)

Conversations 1/8/256 (active/max active/max total)

Reserved Conversations 0/0 (allocated/max allocated)

Available Bandwidth 1158 kilobits/sec

30 second input rate 72000 bits/sec, 57 packets/sec

30 second output rate 34000 bits/sec, 75 packets/sec

179668 packets input, 26780853 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

190904 packets output, 14953044 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

DCD=up DSR=up DTR=up RTS=up CTS=up

Serial0/0.100 is up, line protocol is up

Hardware is PowerQUICC Serial

Description:

Internet address is

MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,

reliability 255/255, txload 5/255, rxload 11/255

Encapsulation FRAME-RELAY IETF

Continued in the next message...

gkuzmowycz · ‎06-02-2006

Here are the interface stats on the switch port that the router connects to:

FastEthernet0/1 is up, line protocol is up

Hardware is Fast Ethernet, address is

MTU 1500 bytes, BW 100000 Kbit, DLY 1000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s

input flow-control is off, output flow-control is off

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:31, output 00:00:00, output hang never

Last clearing of "show interface" counters 17:03:30

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue :0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

136199 packets input, 24459293 bytes, 0 no buffer

Received 3826 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 1023 multicast, 0 pause input

0 input packets with dribble condition detected

197890 packets output, 17738272 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Interface errors on the switch ports are negligible (sh int cou err), meaning mostly zeros with a couple of ports showing single-digit numbers. This is over approximately 18 hours since reset. Zero collisions, runts, giants.

As I mentioned earlier, bandwidth utilization during those times is not excessive, and pings to the far-end router are within normal expectations. That's why I think it's not link congestion, discards, etc.

Replying to Scott, this problem has occurred with FRTS turned on and with it turned off. It is currently off. But as noted above, bandwidth measurements during those times do not show congestion. Also, identically configured PVCs terminating in the same HQ circuit servicing other regional offices which have the same staffing and run the same mix of applications have never experienced this problem.