Cisco 3650-48PS Uplink Issue > Nexus 5548UP

[stevenle]

Trying to get a trunk uplink between a Cisco 3650 to a Nexus 5548UP up and running. It's turned into the most complicated issue ever. GBIC and fiber cable have been proven good by connecting the 3650 to a 3560 using the same equipment and configuration.

The 3650 is running Version 03.06.04.E. Using a GLC-SX-MM gbic with MM fiber. Switches are in the same rack. Configuration on the 3650 port is below.

interface GigabitEthernet1/1/1
description Uplink to 5KA
switchport trunk native vlan 7
switchport mode trunk

As soon as I plug the fiber into the Nexus switch I get the output below on the 3650

%PM-4-ERR_DISABLE: link-flap error detected on Gi1/1/3, putting Gi1/1/3 in err-disable state

The Nexus is a N5K-C5548UP version 5.2(1)N1(4. Using same GLC-SX-MM gbic with fiber. Configuration is below.

version 5.2(1)N1(4)

interface Ethernet1/31
switchport mode trunk
switchport trunk native vlan 7
spanning-tree port type normal
speed 1000

I proved out the physical layer by connecting the 3650 to another switch and it comes up right away. I verified that the transmit on the fiber strand on one end is going to the receive on the other end (light to no light). I messed with the spanning tree port type, speed non negotiate settings as well. 

A 10 minute item has now dragged on for two days and it's killing me. Configuration for the 3650 is attached. I have plenty of other model Cisco switches connected to my 5K's. I can connect this 3650 to a production 3560 but I cannot connect a 3650 to a Nexus 5548. I'm at wits end here. 

Let me know if anyone has had this issue or if I am overlooking some small item. Thanks.

[Philip D'Ath]

Any chance the SFP on the Nexus is damaged?

[stevenle]

I proved out the SFP and fiber by connecting the 3650 to another switch, a 3560 we have in production. Used the same SFP and fiber jumper. Works like a charm. Also tried this SFP on several interfaces on the Nexus side.

I can connect the 3650 to a production 3560 eight the same SFP and fiber. I can connect the Nexus to a 3560 with the same SFP and fiber. I cannot connect the Nexus to my 3650 with this SFP and fiber. Interface on the 3650 goes into err-disable immediately after hitting link flaps.

Looking to to see if anyone has successfully connected a Nexus to a 3650 uplink interface with an SFP and fiber. Very strange issue.

[Madhukrishnan Gopinathan Nair]

Hi , 

Could you please disable the link flap detection and connect the link on Nexus

" no errdisable detect cause link-flap "

If the interface stays up, enable it again and see if it continue to stay up. 

"errdisable detect cause link-flap"

Thanks,

Madhu.

[stevenle]

Thanks for the suggestion. I ran the no errdisable detect cause link-flap on the 3650 because that's where the err disable happens. Once I ran the command the interface kept going up and down and then eventually the Nexus err-disabled it. Took about 10-15 seconds. 

[Madhukrishnan Gopinathan Nair]

Lets connect this 3650 to another N5k and see what happens?

Or if you dont have another N5K, take a Maintenance window and connect it to a known working port on N5K, It could be hardware issue on the port on N5k too? 

If thats not possible, can we connect any other switch to  N5K port in question and see it works?

Regards,

Madhu,

[stevenle]

I tried several different unused ports on the 5K and ports that are currently connected to production switches and have run in to the same issue with each. I have not tried plugging this into another Nexus switch. I do have a 5K HA pair so will try to connect to the other Nexus and let you know how I make out.

[ahmedshoaib]

Hi;

Can you do 1 more test via removing Bridge assurance on Nexus 5K.

no spanning-tree port type normal (interface connected to Cisco 3560 Switch).

Thanks

[stevenle]

Thanks for the suggestion. Tried that but same condition.

[ahmedshoaib]

Hi;

Can you confirm me that below are the configuration on your switches:

Nexus:

interface Ethernet1/31

 switchport mode trunk

 switchport trunk native vlan 7

 speed 1000

3560:

no spanning-tree vlan 7 hello-time 4

no spanning-tree vlan 7 forward-time 4

no spanning-tree vlan 7 max-age 10

interface GigabitEthernet1/1/3

 switchport trunk native vlan 7

 switchport mode trunk

If yes then it should work perfectly without any issue. 

Thanks

[stevenle]

That is my exact configuration. Doesn't work when I use a GLC-SX-MM and fiber for connectivity. It does work when I use a GLC-T copper gbic and a patch cable. The optic and fiber have been proven good by connecting to another switch and I've used brand new gbics and fiber as well. No luck.

[ahmedshoaib]

Hi;

I remember I also face the similar issue, the difference was Nexus 7710 with 4948 via SFP-10G. When ever I connect the link 4948 side port was gone in error disable. When I review my config I found bridge assurance was enable (spanning-tree port type network) once I remove the command & shut - no shut interface every thing working fine.

In your case why is not working i don't know untill to see myself physically. If you have Cisco contract open a case with Cisco and try to troubleshoot the issue. Once you found the issue please share the info with us also. 

Thanks.

[stevenle]

Working with TAC now. They suggested IOS upgrade on the 3650 to 3.6.5 after hours of troubleshooting. Will update once I finish and test.

[stevenle]

Fixed. Put it just below original post. IOS bug, upgrade to later version fixed. Thanks for the assist.

[ahmedshoaib]

Hi;

Why you modify the Hello, forward and Age time for vlan 7 (native Vlan) on Cisco 3560 Switches. These values are different on Nexus 5548 & there is not option available to modify wrt rstp in nexus may be these value is creating problem and trunk become error-disable between Cisco 3560 & Nexus5548.

Thanks & Best regards