WS-X6704-10GE packet loss

dcundiff · ‎08-06-2013

Hello,

I have a really strange problem. I'm trying to setup a 10G 802.1q trunk port between 2 buildings and I'm getting a strange packet loss issue. The connections will start fine and then just stop. Router 1 will send packets to router 2 that are received but the packets from router 1 will start disappearing on the way to router 2. I've even setup a tftp-server on both routers and copied files between them. The same results. The problem lies somewhere between the 2 routers.

This only occurs on the TE ports. The GE ports I've been using are fine. I've also put a light meter on the fiber and it checks out. It's single mode with LR transceivers. The run is about 800 feet. My equipment is OLD so direct TAC support is unavailable.

Hardware

Building 1:

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 16 16 port 1000mb GBIC ethernet WS-X6416-GBIC

2 48 48-port 10/100 mb RJ45 WS-X6148-RJ-45

3 48 48-port 10/100 mb RJ45 WS-X6148-RJ-45

4 48 48-port 10/100 mb RJ45 WS-X6148-RJ-45

5 48 48-port 10/100 mb RJ45 WS-X6148-RJ-45

6 48 48-port 10/100 mb RJ45 WS-X6148-RJ-45

7 2 Supervisor Engine 720 (Active) WS-SUP720-BASE

8 2 Supervisor Engine 720 (Hot) WS-SUP720-BASE

9 48 48-port 10/100 mb RJ45 WS-X6148-RJ-45

10 48 48-port 10/100 mb RJ45 WS-X6148-RJ-45

11 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0005.740c.59f8 to 0005.740c.5a07 2.0 5.4(2) 12.2(33)SXI9 Ok

2 000c.ce27.3f18 to 000c.ce27.3f47 1.2 5.4(2) 12.2(33)SXI9 Ok

3 0012.4304.5e90 to 0012.4304.5ebf 3.1 5.4(2) 12.2(33)SXI9 Ok

4 0012.0182.2228 to 0012.0182.2257 3.1 5.4(2) 12.2(33)SXI9 Ok

5 0015.c667.a944 to 0015.c667.a973 5.0 8.3(1) 12.2(33)SXI9 Ok

6 000c.ce27.41b8 to 000c.ce27.41e7 1.2 5.4(2) 12.2(33)SXI9 Ok

7 0013.7f0d.a5c0 to 0013.7f0d.a5c3 3.3 8.1(3) 12.2(33)SXI9 Ok

8 0011.92e6.fa08 to 0011.92e6.fa0b 3.1 8.1(3) 12.2(33)SXI9 Ok

9 0016.9d53.aeb0 to 0016.9d53.aedf 5.1 8.3(1) 12.2(33)SXI9 Ok

10 0012.016a.f6e8 to 0012.016a.f717 3.1 5.4(2) 12.2(33)SXI9 Ok

11 8843.e1aa.b978 to 8843.e1aa.b97b 3.1 12.2(14r)S5 12.2(33)SXI9 Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

7 Policy Feature Card 3 WS-F6K-PFC3A 2.5 Ok

7 MSFC3 Daughterboard WS-SUP720 2.5 Ok

8 Policy Feature Card 3 WS-F6K-PFC3A 2.4 Ok

8 MSFC3 Daughterboard WS-SUP720 2.3 Ok

11 Distributed Forwarding Card WS-F6700-DFC3A 2.2 Ok

Building 2:

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 48 48 port 10/100 mb RJ21 Enhanced QoS WS-X6248A-TEL

2 48 48 port 10/100 mb RJ21 Enhanced QoS WS-X6248A-TEL

3 48 48 port 10/100 mb RJ21 Enhanced QoS WS-X6248A-TEL

4 48 48 port 10/100 mb RJ-21 ethernet WS-X6248-TEL

5 2 Supervisor Engine 720 (Active) WS-SUP720-BASE

6 2 Supervisor Engine 720 (Hot) WS-SUP720-BASE

7 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE

8 16 16 port 1000mb GBIC ethernet WS-X6416-GBIC

9 96 96-port 10/100 Mbps RJ45 WS-X6196-21AF

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0030.b63c.df90 to 0030.b63c.dfbf 1.0 5.1(1)CSX 8.5(0.46)RFW Ok

2 0005.9a8a.9118 to 0005.9a8a.9147 2.0 5.4(2) 8.5(0.46)RFW Ok

3 0001.9754.6d80 to 0001.9754.6daf 1.0 5.1(1)CSX 8.5(0.46)RFW Ok

4 00d0.d331.f118 to 00d0.d331.f147 1.0 4.2(0.24)VAI 8.5(0.46)RFW Ok

5 0011.21a1.587c to 0011.21a1.587f 3.2 8.1(3) 12.2(18)SXF1 Ok

6 0013.7f0a.7004 to 0013.7f0a.7007 3.3 8.1(3) 12.2(18)SXF1 Ok

7 0019.aaf1.dcf8 to 0019.aaf1.dcfb 2.5 12.2(18r)S1 12.2(18)SXF1 Ok

8 0009.b794.1400 to 0009.b794.140f 2.3 5.4(2) 8.5(0.46)RFW Ok

9 0019.306d.d796 to 0019.306d.d7f5 1.4 8.2(2) 8.5(0.46)RFW Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

5 Policy Feature Card 3 WS-F6K-PFC3A 2.4 Ok

5 MSFC3 Daughterboard WS-SUP720 2.4 Ok

6 Policy Feature Card 3 WS-F6K-PFC3A 2.5 Ok

6 MSFC3 Daughterboard WS-SUP720 2.5 Ok

7 Distributed Forwarding Card WS-F6700-DFC3A 2.2 Ok

9 IEEE Voice Daughter Card WS-F6K-FE48X2-AF 1.4 Ok

Below is the relavant configuration from both routers. I only pasted 1 copy as its identical on both except for slot differences on the ports. The only thing I do to cause the packet loss is move the fiber from the GE ports to the TE ports and everything falls apart. Moving it back to the GE ports puts everything back to normal.

spanning-tree mode rapid-pvst

spanning-tree portfast default

interface GigabitEthernet5/1

switchport

switchport trunk encapsulation dot1q

switchport mode trunk

mtu 9216

no ip address

mls qos trust dscp

end

interface TenGigabitEthernet7/3

switchport

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 1-999

switchport mode trunk

mtu 9216

no ip address

shutdown

mls qos trust dscp

end

I'm excluding vlan 1000 on the TE port since it has storage server traffic on it. The other vlans can handle some packet loss for short periods while I'm testing. The storage servers cannot.

The really strange thing about this is I get ZERO error counters.

Router 1:

show int

TenGigabitEthernet11/3 is down, line protocol is down (notconnect)

Hardware is C6k 10000Mb 802.3, address is 8843.e1aa.b97a (bia 8843.e1aa.b97a)

MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec,

reliability 255/255, txload 0/255, rxload 0/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 10Gb/s, media type is 10Gbase-LR

input flow-control is off, output flow-control is off

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:39:30, output 00:39:30, output hang never

Last clearing of "show interface" counters 11w1d

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

1756250870 packets input, 276893128198 bytes, 0 no buffer

Received 4652610 broadcasts (2814099 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

2287076473 packets output, 1940759519804 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Show fabric errors

Module errors:

slot channel crc hbeat sync DDR sync

7 0 0 0 0 0

8 0 0 0 0 0

11 0 0 0 0 0

11 1 0 0 0 0

Fabric errors:

slot channel sync buffer timeout

7 0 0 0 0

8 0 0 0 0

11 0 0 0 0

11 1 0 0 0

Router 2:

Show int

TenGigabitEthernet7/3 is administratively down, line protocol is down (disabled)

Hardware is C6k 10000Mb 802.3, address is 0019.aaf1.dcfa (bia 0019.aaf1.dcfa)

MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec,

reliability 255/255, txload 0/255, rxload 0/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 10Gb/s

input flow-control is off, output flow-control is off

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:37:27, output 00:36:55, output hang never

Last clearing of "show interface" counters 12w2d

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

2287095723 packets input, 1940873532153 bytes, 0 no buffer

Received 312018 broadcasts (298729 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

1756240845 packets output, 276904112065 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Show fabric errors

Module errors:

slot channel crc hbeat sync DDR sync

5 0 0 0 0 0

6 0 0 0 0 0

7 0 0 0 0 0

7 1 0 0 0 0

Fabric errors:

slot channel sync buffer timeout

5 0 0 0 0

6 0 0 0 0

7 0 0 0 0

7 1 0 0 0

Hopefully someone can help me shed some light on this as its the strangest problem I've ever seen.

Thanks,

Dave