Errors and link flaps on fibre (fiber) trunk

Andrew Devine · ‎08-11-2011

Hi,

I have setup a trunk between a WS-X4306-GB line card and a dual-media uplink port on a WS-CBS3020-HPQ blade switch. However, when I enable the port it flaps every minute or so and after the flap reports runts on the interface.

The WS-X4306-GB line card is installed in mod 9 of a 4510R-E running IOS 12.2(46)SG (cat4500e-ENTSERVICESK9-M), a 1000BASE-SX GBIC (WS-G5484) is installed in Gi 9/1
The WS-CBS3020-HPQ blade switch is running 12.2(40)SE2 (CBS30X0-LANBASE-M) and has a 1000BASE-SX SFP (GLC-SX-MM) installed into port Gi 0/17
The fibre connection goes via a patch panel
62.5/125 and 50/125 fibre has been tried but with the same result for both types

Config:-

!

! STP port-priority configured so I can enable new uplink without affecting STP topology, copper uplinks existing

!

4510R-E#sh run int gi 9/1

!

interface GigabitEthernet9/1

switchport mode trunk

udld port aggressive

spanning-tree port-priority 144

!

! Uplinkfast enabled so I can control root port by altering 9/1 STP port priority out of hours, link state group on all uplinks to bring blade NICs down

!

3020#sh run int gi 0/17

!
interface GigabitEthernet0/17

switchport trunk encapsulation dot1q
switchport mode trunk

udld port aggressive
link state group 1 upstream

!

Outputs:-

4510R-E#sh int gi 9/1 contr

GigabitEthernet9/1 is down, line protocol is down (notconnect)

Hardware is Gigabit Ethernet Port, address is 0022.bddd.48b6 (bia 0022.bddd.48b6)

Description: 3020 Gi0/17

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, Auto-speed, link type is auto, media type is 1000BaseSX

input flow-control is off, output flow-control is off

ARP type: ARPA, ARP Timeout 04:00:00

Last input 21:03:48, output never, output hang never

Last clearing of "show interface" counters never

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

1310747 packets input, 1293130218 bytes, 0 no buffer

Received 4965 broadcasts (4504 multicasts)

2 runts, 0 giants, 0 throttles

2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 input packets with dribble condition detected

5931346 packets output, 1660789456 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

4510R-E#sh int gi 9/1 capabilities

GigabitEthernet9/1

Model: WS-X4306-GB-Gbic

Type: 1000BaseSX

Speed: 1000

Duplex: full

Auto-MDIX: no

Trunk encap. type: 802.1Q

Trunk mode: on,off,desirable,nonegotiate

Channel: yes

Broadcast suppression: percentage(0-100), hw

Multicast suppression: percentage(0-100), hw

Flowcontrol: rx-(off,on,desired),tx-(off,on,desired)

VLAN Membership: static, dynamic

Fast Start: yes

CoS rewrite: yes

ToS rewrite: yes

Inline power: no

SPAN: source/destination

UDLD: yes

Link Debounce: no

Link Debounce Time: no

Port Security: yes

Dot1x: yes

Maximum MTU: 9198 bytes (Jumbo Frames)

Multiple Media Types: no

Diagnostic Monitoring: N/A

Queuing: rx-(N/A), tx-(1p7q1t)

3020#sh int gi 0/17 controller
GigabitEthernet0/17 is administratively down, line protocol is down (disabled)
Hardware is Gigabit Ethernet, address is 0022.91af.9a11 (bia 0022.91af.9a11)
Description: 4510R-E Gi9/1
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 254/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Auto-duplex, Auto-speed, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 21:06:47, output 21:06:47, output hang never
Last clearing of "show interface" counters 21:20:17
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1133530 packets input, 783706915 bytes, 0 no buffer
Received 4791 broadcasts (12215 multicasts)
2166 runts, 0 giants, 0 throttles
2166 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 12215 multicast, 0 pause input
0 input packets with dribble condition detected
302 packets output, 35116 bytes, 0 underruns
0 output errors, 0 collisions, 5 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Transmit GigabitEthernet0/17 Receive
1474443247 Bytes 2002996525 Bytes
1316622994 Unicast frames 45790944 Unicast frames
90800633 Multicast frames 1047999 Multicast frames
71657260 Broadcast frames 410590 Broadcast frames
0 Too old frames 1693481132 Unicast bytes
2167069 Deferred frames 236924556 Multicast bytes
0 MTU exceeded frames 45180464 Broadcast bytes
862378 1 collision frames 0 Alignment errors
2398 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 63932 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 12648203 Minimum size frames
0 8 collision frames 4929545 65 to 127 byte frames
0 9 collision frames 5811136 128 to 255 byte frames
0 10 collision frames 342125 256 to 511 byte frames
0 11 collision frames 1252505 512 to 1023 byte frames
0 12 collision frames 22266019 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
163200 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 0 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
645753757 64 byte frames 63932 Valid frames, too small
296775985 127 byte frames
135004286 255 byte frames 0 Too old frames
66860049 511 byte frames 0 Valid oversize frames
134507717 1023 byte frames 0 System FCS error frames
200245716 1518 byte frames 0 RxPortFifoFull drop frame
0 Too large frames
862378 Good (1 coll) frames
2398 Good (>1 coll) frames

4510R-Esh int gi 0/17 capabilities
GigabitEthernet0/17
Model: WS-CBS3020-HPQ
Type: 1000BaseSX SFP
Speed: 1000
Duplex: full
Trunk encap. type: 802.1Q,ISL
Trunk mode: on,off,desirable,nonegotiate
Channel: yes
Broadcast suppression: percentage(0-100)
Flowcontrol: rx-(off,on,desired),tx-(none)
Fast Start: yes
QoS scheduling: rx-(not configurable on per port basis),tx-(4q2t)
CoS rewrite: yes
ToS rewrite: yes
UDLD: yes
Inline power: no
SPAN: source/destination
PortSecure: yes
Dot1x: yes
Multiple Media Types: rj45, sfp, auto-select

Aug 10 12:17:02 BST: %LINK-3-UPDOWN: Interface GigabitEthernet0/17, changed state to up

Aug 10 12:17:05 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to up

Aug 10 12:17:41 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to down

Aug 10 12:17:44 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to up

Aug 10 12:17:58 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to down

Aug 10 12:18:01 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to up

Aug 10 12:19:11 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to down

Aug 10 12:19:14 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to up

Aug 10 12:19:41 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to down

Aug 10 12:19:44 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to up

Aug 10 12:22:12 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to down

Aug 10 12:22:15 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to up

Aug 10 12:22:20 BST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/17, changed state to down

Any help appreciated, I intened to run a direct cable to rule out the patch panel once I've sourced one long enough.

Cheers,

Andrew

Leo Laohoo · ‎08-11-2011

3020#sh int gi 0/17 controller

GigabitEthernet0/17 is administratively down, line protocol is down (disabled)

Hardware is Gigabit Ethernet, address is 0022.91af.9a11 (bia 0022.91af.9a11)

Description: 4510R-E Gi9/1

MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,

reliability 254/255, txload 1/255, rxload 1/255

That't not good. Change the GBIC on the 3020 and see what happens.

Andrew Devine · ‎08-11-2011

I did try swapping the SFP, I notice the relaiability is 255 on the parallel switch in the blade chassis.

I had the exact same issue with both blade switches so implies a common issue.

3020-2#sh int gi 0/17

GigabitEthernet0/17 is administratively down, line protocol is down (disabled)

Hardware is Gigabit Ethernet, address is 0022.91af.b511 (bia 0022.91af.b511)

Description: 4510R-E Gi9/2

MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive not set

Auto-duplex, Auto-speed, link type is auto, media type is 1000BaseSX SFP

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 1d15h, output 1d15h, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

90237 packets input, 36657846 bytes, 0 no buffer

Received 1209 broadcasts (3518 multicasts)

15 runts, 0 giants, 0 throttles

15 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 3518 multicast, 0 pause input

0 input packets with dribble condition detected

82074 packets output, 75563958 bytes, 0 underruns

0 output errors, 0 collisions, 6 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Leo Laohoo · ‎08-11-2011

Which is wierd because even though the port is ADMINISTRATIVELY DOWN the reliability is 254/255. The value changes so it doesn't mean that it stays that way. It was just good that you punched the command in when the value went down.

What IOS are both switches running?

Andrew Devine · ‎08-11-2011

All defined in the bullet points above...

Leo Laohoo · ‎08-11-2011

Did you change the patch cable?

Andrew Devine · ‎08-11-2011

Tried 62.5/125 and 50/125 fibre, same result. Thinking possibly patch panel or maybe dirty GBICs/SFPs

Jean-Christophe Sicard · ‎09-01-2011

Don't believe the rx/tx load value: older IOSes on CBS3020 have a known bug (sorry don't have the bugid at hand) where the interface bandwidth is incorrectly reported as 10mbps instead of 1gig.

See from your own output:

leolaohoo wrote:

3020#sh int gi 0/17 controller

GigabitEthernet0/17 is administratively down, line protocol is down (disabled)

Hardware is Gigabit Ethernet, address is 0022.91af.9a11 (bia 0022.91af.9a11)

Description: 4510R-E Gi9/1

MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,

reliability 254/255, txload 1/255, rxload 1/255

That't not good. Change the GBIC on the 3020 and see what happens.

Since the load values are calculated based on the bandwidth value, they are wrong.

You can either upgrade to a newer IOS or set the bandwidth parameter manually on all interfaces (this only affects the reported bandwidth...).

amikat · ‎08-11-2011

Hi,

Will you please check via the "show udld" command that UDLD message timer (interval) is set to the same value at both ends.

Best regards,

Antonin

Andrew Devine · ‎08-11-2011

4510R-E#sh udld gi 9/1

Interface Gi9/1
---
Port enable administrative configuration setting: Enabled / in aggressive mode
Port enable operational state: Enabled / in aggressive mode
Current bidirectional state: Unknown
Current operational state: Link down
Message interval: 7
Time out interval: 5
No neighbor cache information stored

3020#show udld gi 0/17

Interface Gi0/17
---
Port enable administrative configuration setting: Enabled / in aggressive mode
Port enable operational state: Enabled / in aggressive mode
Current bidirectional state: Unknown
Current operational state: Link down
Message interval: 7
Time out interval: 5
No neighbor cache information stored

Jonathancert_2 · ‎08-11-2011

Are you using "GLC-SX-MM" for the SFP on your 3020?

Jonathan,

Andrew Devine · ‎08-11-2011

Yup

3020#show int gi 0/17 transceiver properties

Diagnostic Monitoring is not implemented.

Name : Gi0/17

Administrative Speed: auto

Administrative Duplex: auto

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: auto

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: 1000BaseSX SFP

Configured Media: auto-select

Active Media: sfp

Jonathancert_2 · ‎08-11-2011

I did notice you keepalive setting don't match between devices and try hard coding your duplex on your 3020.

3020#sh int gi 0/17 controller

GigabitEthernet0/17 is administratively down, line protocol is down (disabled)

Hardware is Gigabit Ethernet, address is 0022.91af.9a11 (bia 0022.91af.9a11)

Description: 4510R-E Gi9/1

MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,

reliability 254/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive not set

Auto-duplex, Auto-speed, link type is auto, media type is 1000BaseSX SFP

Jonathan,

Andrew Devine · ‎08-11-2011

I though ethernet keepalives were only locally significant?

there is no option to configure speed/duplex on any of the interfaces, the GBIC and SFP are pinned to 1000/Full.

The default for SFP interfaces is no keepalive

Jonathancert_2 · ‎08-11-2011

To my knowledge (which does have potholes), having no keepalive on an interface will trick the device to believing the interface is up when actually there is no distant end connection. I am a fan of uniformity when it comes to Cisco configs. When possible make you P2P match. I have several 4510R-E's. My WS-X4624-SFP-E's are set to "Full" duplex and "1000" speed, no other options available. But on my WS-X4148-FX-MT's I have an option of "Full" or "Half" duplex. Strange, that your 3020 is showing "Auto" for the duplex.

Jonathan,