I'm using a MOE circuit here in Phoenix to connect several locations in a mesh configuration with my Main DC. I have one location that has 3-5% packet loss but I see no errors inbound or outbound on the interface. CEF is running and none of the other sites are dropping packets either to eachother or to the DC, However when I ping the troubled site I see the same behavior. Here are the statistics...
NOTE: Each Remote Site has a collapsed Core with a 6509 running the latest IOS and Firmware.
Remote-Site1#p
Protocol [ip]:
Target IP address: [Troubled-Site]
Repeat count [5]: 1000
Datagram size [100]: 1500
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to [Troubled-Site], timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!.!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!.!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!.!.!!!!!!!!!!!!!!.!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!.!!!.!!.!!!!!
!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!.!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
Success rate is 96 percent (969/1000), round-trip min/avg/max = 1/3/12 ms
Troubled-Site#sh ip int GigabitEthernet1/24
GigabitEthernet1/24 is up, line protocol is up
Internet address is [omitted]/28
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Multicast reserved groups joined: 224.0.0.10 224.0.0.1 224.0.0.2 224.0.0.22
224.0.0.13
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is enabled
IP CEF switching is enabled
IP Flow switching turbo vector
IP Flow CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow cache, CEF, Full Flow
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
BGP Policy Mapping is disabled
Sampled Netflow is disabled
IP multicast multilayer switching is enabled
Troubled-Site#sh int GigabitEthernet1/24
GigabitEthernet1/24 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 0017.0f97.f000 (bia 0017.0f97.f000)
Description: QMOE CKT#14/XXXX/XXXXXXX to Central DC
Internet address is [omitted]/28
MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec,
reliability 255/255, txload 14/255, rxload 33/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is SX
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 6w4d
Input queue: 0/75/31/31 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 13244000 bits/sec, 2122 packets/sec
5 minute output rate 5598000 bits/sec, 2011 packets/sec
L2 Switched: ucast: 17662597 pkt, 1982838385 bytes - mcast: 24075638 pkt, 12600715488 bytes
L3 in Switched: ucast: 7321139380 pkt, 6113001010979 bytes - mcast: 457819 pkt, 137273358 bytes mcast
L3 out Switched: ucast: 7154731761 pkt, 2792326024317 bytes mcast: 45289477 pkt, 22070735402 bytes
7365636806 packets input, 6128170808390 bytes, 0 no buffer
Received 26647616 broadcasts (5918633 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
7156492485 packets output, 2774754690600 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Please Share your expertise and any insight into this. I've never seen so much packet loss with no Errors whatsoever before. That's just odd...
Sincerely,
Michael
