Nexus / link fail / autonegotiation

[STUART KENDRICK]

This summer, we replaced a pair of Catalyst 4500s with a pair of Nexus 5010s, dangling a FEX off each (N2K-C2248TP-1GE).  No cabling change -- just swapped electronics.

Since then, the operational staff have been struggling to persuade our APC gear to connect (APC 9606, 9617, and 9630 cards, inserted into a range of PDUs, EMUs, and UPSes).

I spent an hour in the room this morning with a Fluke NetTool (Series II), putting in-line with a couple of APC PDUs plus one of the tech's laptops, watching log messages from the Nexi. playing around.

I think I'm seeing autonegotiation issues.  But perhaps you can see something else.

Here's a PDU -- link comes up briefly (though notice the 10 Mb/s speed ... I didn't think that the 2248T supported 10Mb!?) -- and then drops.

Nov 22 09:24:26 dfsr-a-esx : 2010 Nov 22 09:24:26 pst: %ETHPORT-5-SPEED: Interface Ethernet100/1/40, operational speed changed to 10 Mbps
Nov 22 09:24:26 dfsr-a-esx : 2010 Nov 22 09:24:26 pst: %ETHPORT-5-IF_DUPLEX: Interface Ethernet100/1/40, operational duplex mode changed to Full
Nov 22 09:24:26 dfsr-a-esx : 2010 Nov 22 09:24:26 pst: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Receive Flow Control state changed to off
Nov 22 09:24:26 dfsr-a-esx : 2010 Nov 22 09:24:26 pst: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Transmit Flow Control state changed to on
Nov 22 09:24:26 dfsr-a-esx : 2010 Nov 22 09:24:26 pst: %ETHPORT-5-IF_UP: Interface Ethernet100/1/40 is up
Nov 22 09:24:30 dfsr-a-esx : 2010 Nov 22 09:24:30 pst: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet100/1/40 is down (Link failure)
Nov 22 09:24:40 dfsr-a-esx : 2010 Nov 22 09:24:40 pst: %ETHPORT-5-SPEED: Interface Ethernet100/1/40, operational speed changed to 10 Mbps
Nov 22 09:24:40 dfsr-a-esx : 2010 Nov 22 09:24:40 pst: %ETHPORT-5-IF_DUPLEX: Interface Ethernet100/1/40, operational duplex mode changed to Full
Nov 22 09:24:40 dfsr-a-esx : 2010 Nov 22 09:24:40 pst: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Receive Flow Control state changed to off
Nov 22 09:24:40 dfsr-a-esx : 2010 Nov 22 09:24:40 pst: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Transmit Flow Control state changed to on
Nov 22 09:24:40 dfsr-a-esx : 2010 Nov 22 09:24:40 pst: %ETHPORT-5-IF_UP: Interface Ethernet100/1/40 is up

Nov 22 09:24:30 dfsr-a-esx : 2010 Nov 22 09:24:30 pst: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet100/1/40 is down (Link failure)

Here's another try with the PDU -- negotiates to 1Gbps (the APC9630 card in the PDU contains a 10/100 chipsset), but in the end drops link also:

Nov 22 09:25:15 dfsr-a-esx : 2010 Nov 22 09:25:15 pst: %ETHPORT-5-SPEED: Interface Ethernet100/1/40, operational speed changed to 1 Gbps
Nov 22 09:25:15 dfsr-a-esx : 2010 Nov 22 09:25:15 pst: %ETHPORT-5-IF_DUPLEX: Interface Ethernet100/1/40, operational duplex mode changed to Full
Nov 22 09:25:15 dfsr-a-esx : 2010 Nov 22 09:25:15 pst: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Receive Flow Control state changed to off
Nov 22 09:25:15 dfsr-a-esx : 2010 Nov 22 09:25:15 pst: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Transmit Flow Control state changed to on
Nov 22 09:25:15 dfsr-a-esx : 2010 Nov 22 09:25:15 pst: %ETHPORT-5-IF_UP: Interface Ethernet100/1/40 is up
Nov 22 09:25:16 dfsr-a-esx : 2010 Nov 22 09:25:16 pst: %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet100/1/40 is down (Link failure)

Here is the laptop succeeding (the laptop succeeded reliably, although sometimes it would autonegotiate to 10/full, rather than 1000/full):

Nov 22 09:47:21 dfsr-b-esx : 2010 Nov 22 09:47:21 pst: %ETHPORT-5-SPEED: Interface Ethernet100/1/25, operational speed changed to 1 Gbps
Nov 22 09:47:21 dfsr-b-esx : 2010 Nov 22 09:47:21 pst: %ETHPORT-5-IF_DUPLEX: Interface Ethernet100/1/25, operational duplex mode changed to Full
Nov 22 09:47:21 dfsr-b-esx : 2010 Nov 22 09:47:21 pst: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet100/1/25, operational Receive Flow Control state changed to off
Nov 22 09:47:21 dfsr-b-esx : 2010 Nov 22 09:47:21 pst: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet100/1/25, operational Transmit Flow Control state changed to on
Nov 22 09:47:21 dfsr-b-esx : 2010 Nov 22 09:47:21 pst: %ETHPORT-5-IF_UP: Interface Ethernet100/1/25 is up

The NetTool reported the following:

Nexi advertised:  10/100/1000

APC advertised:  10/100

Nexi negotiated to 100/full

APC negotiated to 100/half

Link stays down / ping does not work

Nexi advertised: 10/100/1000

Laptop advertised: 10/100/1000

Nexi negotiated to 10/full

Laptop negotiated to 10/full

[Sometimes, both would negotiate to 1000/full]

Link comes up / ping works

For grins, the Ops staff have schlepped two of the PDUs and one of the EMUs to another location, plugged them into Catalyst 4500s, link comes up immediately; they can configure the appropriate IP address and then ping successfully.  Most of the NICs in the room, ~60 devices, work fine, including most of the APC NICs.  However, some of the gear has never worked (since the Nexi went in), some have worked intermittently.  I have visibility into the APC stuff, because our group manages it; I don't know how much, if any, of the more conventional end-stations (Dell, Sun, and IBM servers) have been affected.

I lived through the broken-negotiation era in the 1990s, with buggy NIC drivers, admins hard-coding one side to full-duplex and leaving the other side auto-negotiating (blech), and even trying to run 10/100 autonegotiation over Type 1 cabling using baluns (failed spectacularly -- had to hard-code both sides).  But I haven't seen auto-negotiation issues ... in many years.  All ~13,000 Ethernet ports at our company are configured to auto-negotiate, except for four feeding old Compaq Alpha machines.

Am I on the right trail?  Or can someone see another avenue of inquiry?

--sk

Stuart Kendrick

FHCRC

dfsr-a-esx# sh fex
  FEX         FEX           FEX                       FEX              
Number    Description      State            Model            Serial    
------------------------------------------------------------------------
100        FEX0100                Online     N2K-C2248TP-1GE   JAF1416BLMR
dfsr-a-esx#

Software
  BIOS:      version 1.2.0
  loader:    version N/A
  kickstart: version 4.2(1)N1(1)
  system:    version 4.2(1)N1(1)
  power-seq: version v1.0
  BIOS compile time:       06/19/08
  kickstart image file is: bootflash:/n5000-uk9-kickstart.4.2.1.N1.1.bin
  kickstart compile time:  4/29/2010 19:00:00 [04/29/2010 19:38:04]
  system image file is:    bootflash:/n5000-uk9.4.2.1.N1.1.bin
  system compile time:     4/29/2010 19:00:00 [04/29/2010 20:51:47]

Hardware
  cisco Nexus5010 Chassis ("20x10GE/Supervisor")
  Intel(R) Celeron(R) M CPU    with 2074284 kB of memory.
  Processor Board ID JAF1342AMFG

  Device name: dfsr-a-esx
  bootflash:    1003520 kB

Kernel uptime is 39 day(s), 5 hour(s), 17 minute(s), 22 second(s)

Last reset
  Reason: Unknown
  System version: 4.2(1)N1(1)
  Service:

plugin
  Core Plugin, Ethernet Plugin
dfsr-a-esx#

[STUART KENDRICK]

And if I hard-code the port to 100Mb, then link comes up:

dfsr-a-esx# config t

dfsr-a-esx(config)# int eth100/1/40

dfsr-a-esx(config-if)# speed 100

dfsr-a-esx(config-if)#

Nov 22 16:18:09 dfsr-a-esx : 2010 Nov 22 16:18:09 pst: %ETHPORT-5-SPEED: Interface Ethernet100/1/40, operational speed changed to 100 Mbps
Nov 22 16:18:09 dfsr-a-esx : 2010 Nov 22 16:18:09 pst: %ETHPORT-5-IF_DUPLEX: Interface Ethernet100/1/40, operational duplex mode changed to Full
Nov 22 16:18:09 dfsr-a-esx : 2010 Nov 22 16:18:09 pst: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Receive Flow Control state changed to off
Nov 22 16:18:09 dfsr-a-esx : 2010 Nov 22 16:18:09 pst: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet100/1/40, operational Transmit Flow Control state changed to on
Nov 22 16:18:09 dfsr-a-esx : 2010 Nov 22 16:18:09 pst: %ETHPORT-5-IF_UP: Interface Ethernet100/1/40 is up

dfsr-a-esx# sh int eth100/1/40 cap
Ethernet100/1/40
  Model:                 N2K-C2248TP-1GE
  Type (Non SFP):        --
  Speed:                 10,100,1000,auto
  Duplex:                full
  Trunk encap. type:     802.1Q
  Channel:               yes
  Broadcast suppression: percentage(0-100)
  Flowcontrol:           rx-(off/on),tx-(off/on)
  Rate mode:             none
  QOS scheduling:        rx-(6q1t),tx-(1p6q0t)
  CoS rewrite:           no
  ToS rewrite:           no
  SPAN:                  yes
  UDLD:                  no
  Link Debounce:         yes
  Link Debounce Time:    yes
  MDIX:                  no
  Pvlan Trunk capable:   yes

dfsr-a-esx#

So, I find this astonishing.  Auto-negotiation failure between modern hardware!? (the APC card in question is the latest model, just shipped this year).  But perhaps I'm naive ... have other folks seen auto-negotiation issues recently?

--sk

[robmillet]

Yep, got a similar random issue when plugging into an Avaya MGC.

Nothing else helpful I'm afraid - I've just started to look into the problem and yours is the first post I've run into.

The problem came to light when we shut an interface for testing and brought it back up but it just stayed down no matter what we did, until we altered the speed/duplex settings.

avaya   -   cisco

auto     -   auto         = 100/full - but issue where it just stays down/down

100/full     -   100/full = comes up everytime but reads as 100/half on the avaya

100/full -   auto         = comes up each time and reads the required speed on both sides

looking a bit closer, If we plugged a laptop in after the problem appeared, the port would come up straight away. Pluggin the avaya back in and that interface that previously wouldn't move from down/down came straight up.

We have 2 sites (geographically seperated) with the same setups and an MGC at both and they both exhibit the same symptoms.

Anyways, will post again if I find anything more specific.

[Jitendra Kumar]

Hi Hi !! 

Here is the Answer !!

2.3 Port Speed/Duplex
Avaya recommends Cisco ports connected to CLAN, MedPro, IPSI, S8xxx, G700, G450,
G350 and G250 are fixed at 100Mb/s full-duplex. Cisco LAN switch ports connecting IP
Phones should be configured for auto-negotiation. PCs connecting to IP Phones’ secondary
Ethernet ports should also be auto-negotiation. To avoid duplex mismatch, fixed or auto
values must be identically configured at both ends of the Ethernet link. A value of ‘auto’ at
one end and ‘fixed’ at the other will result in duplex mismatch.

https://pdf4pro.com/cdn/avaya-system-and-network-connectivity-guidelines-6e67a.pdf

[jailong]

Hello, Stuart.

Did you solve your autonegotion with speed ?

I had a very similar situaltion with N5Ks and FEX 2K ?

I just solved it by manually set the speed and duplex, after that everything is OK.

But, I want to know the exact reason. Do you know why ?

[FTBZ]

I know that's very old thread, but I've a similar issue with FEX 2348TQ-10G and APC/MGE things. Forcing the port to 100M has no effect in my case. If someone has an idea...

[Rakesh Tiwari]

any resolution of this ?