Nexus 9K C93108TC connected with 1Gig NIC is not working

[vishalpsingh]

Hello All, 

I have a EVPN VXLAN setup where Nexus 9K C93108TC is acting as a leaf. There are several servers connected with the leaf switch. We have some servers with 10gig NIC and some of them have 1Gig NIC. All the server with 10Gig NIC work properly but servers with 1Gig NIC are getting stuck/wedged few mins after reboot. If we just put dumb switch from another vendor between server w/ 1Gig NIC and the Nexus leaf switch everything starts to work again but as long as server with 1Gig NIC is directly connected with nexus sw it doesn't work. 

We have tried to toggle speed, flow control etc but that doesn't help. Anyone has seen similar issue ?  

[Francesco Molino]

Hi

 when you connect your 1G device, what is the status of the port? 
also do you see any mac address learned on that interface?

[vishalpsingh]

Hello, 

The port remains up all the time..It just stops passing traffic after few mins or so. MAC address is learned in MAC table and EVPN (MAC+IP) entries can be seen in the EVPN table but traffic stops to work. I'm wondering is there some parameter on the port level that I can try? I had tried manual speed and flow control configs but that didn't help.  

[Francesco Molino]

Also can you share the NX-OS version please?

[vishalpsingh]

NXOS: version 9.3(7)

[balaji.bandi]

we would like to know what port you are connecting.

C93108TC - is fixed Ethernet port with uplink ports fixed 40/100

since Ethernet ports are mGig port, try default the interface and set the speed to 1000 and check ?

what servers you connecting other side ?

still issue

post

show run interface ether x/x (where the server connected)

show interface ethe x/x

show logg last 50 ( if you see what is the logs when you plug the port to Server?)

[vishalpsingh]

I have checked with speed 1000 that didn't help. When I configure speed 1000 it still works in Auto-neg mode since it doesn't allow no-auto-neg on speed 1G and above. So I tried with 100Mb and no-auto-neg that also didn't help. I have also tried flow-control on but there were no pause frame or anything. Its a MAC machine

Config: 

version 9.3(7) Bios:version 05.43

interface Ethernet1/22
switchport mode trunk
switchport trunk native vlan 600
switchport trunk allowed vlan 600,616-618
spanning-tree port type edge trunk
spanning-tree bpduguard enable

show interface
xq110.iad65# show interface ethernet 1/22
Ethernet1/22 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000 Ethernet, address: 
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned on
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 00:12:15
Last clearing of "show interface" counters never
24 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 912200 bits/sec, 336 packets/sec
input rate 0 bps, 0 pps; output rate 912.20 Kbps, 336 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 0 bits/sec, 0 packets/sec
300 seconds output rate 900896 bits/sec, 301 packets/sec
input rate 0 bps, 0 pps; output rate 900.90 Kbps, 301 pps
RX
3786 unicast packets 0 multicast packets 121 broadcast packets
4056 input packets 381588 bytes
0 jumbo packets 0 storm suppression bytes
147 runts 0 giants 2 CRC 0 no buffer
149 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
9152 unicast packets 30992125 multicast packets 320328950 broadcast packets
351330227 output packets 120559108277 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Logs:

2022 Jan 24 13:32:44 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 13:32:49 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to off
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 13:32:54 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 13:33:01 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 13:34:21 xyz %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/22 is down (Link failure)
2022 Jan 24 13:34:41 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 13:41:44 xyz %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/22 is down (Link failure)
2022 Jan 24 13:42:19 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 14:10:23 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 14:10:28 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to off
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 14:10:34 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 14:10:39 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to off
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to off
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 25 08:59:29 xyz %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/22 is down (Link failure)
2022 Jan 25 08:59:50 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to off
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to off
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk

[balaji.bandi]

Maybe another suggestion,  some time in nexus the speed command not take (my observation).

shutdown the port

remove the SFP

set the speed 

connect the SFP

setup the speed and config,. no shutdown and test it ?

[vishalpsingh]

I configured the port with 1Gig speed but that didn't help. With 1Gig or above speed we cant disable the auto-neg so I disabled the auto-neg with 100Mb that also didn't help. I enabled flow control but it didn't help. There are no queuing drops. 

Surprising things is after enabling the port it works for few mins and get stuck / wedged after few mins

Config:

version 9.3(7) Bios:version 05.43

interface Ethernet1/22
switchport mode trunk
switchport trunk native vlan 600
switchport trunk allowed vlan 600,616-618
spanning-tree port type edge trunk
spanning-tree bpduguard enable
no shutdown


Interface:
 show interface ethernet 1/22
Ethernet1/22 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000 Ethernet, 
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned on
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 00:10:54
Last clearing of "show interface" counters never
29 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 860104 bits/sec, 303 packets/sec
input rate 0 bps, 0 pps; output rate 860.10 Kbps, 303 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 0 bits/sec, 0 packets/sec
300 seconds output rate 884992 bits/sec, 295 packets/sec
input rate 0 bps, 0 pps; output rate 884.99 Kbps, 295 pps
RX
4773 unicast packets 0 multicast packets 179 broadcast packets
5101 input packets 484956 bytes
0 jumbo packets 0 storm suppression bytes
147 runts 0 giants 2 CRC 0 no buffer
149 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
10832 unicast packets 31101905 multicast packets 321494091 broadcast packets
352606828 output packets 120996987714 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Logs:
2022 Jan 24 13:32:44 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 13:32:49 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to off
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:32:49 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 13:32:54 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 13:33:01 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:33:01 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 13:34:21 xyz %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/22 is down (Link failure)
2022 Jan 24 13:34:41 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:34:41 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 13:41:44 xyz %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/22 is down (Link failure)
2022 Jan 24 13:42:19 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to on
2022 Jan 24 13:42:19 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 14:10:23 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 14:10:28 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to on
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to off
2022 Jan 24 14:10:28 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 24 14:10:34 xyz %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/22 is down(Config change)
2022 Jan 24 14:10:39 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to off
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to off
2022 Jan 24 14:10:39 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk
2022 Jan 25 08:59:29 xyz %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/22 is down (Link failure)
2022 Jan 25 08:59:50 xyz %ETHPORT-5-SPEED: Interface Ethernet1/22, operational speed changed to 1 Gbps
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/22, operational duplex mode changed to Full
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/22, operational Receive Flow Control state changed to off
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/22, operational Transmit Flow Control state changed to off
2022 Jan 25 08:59:50 xyz %ETHPORT-5-IF_UP: Interface Ethernet1/22 is up in mode trunk

[Sergiu.Daniluk]

Hi @vishalpsingh 

What is curious about the output you shared is this:

300 seconds input rate 0 bits/sec, 0 packets/sec
300 seconds output rate 884992 bits/sec, 295 packets/sec

It looks like the interface on N9K is functioning just fine. The problem seems to be on the server side. Check from the server side what is happening (logs, ARP tables, pings etc)

Stay safe,

Sergiu

[Francesco Molino]

Is this the only switch you have in that specific model? If no, is the issue only on this switch or same on all switches?

 Did you test the host connectivity on this same switch without vxlan?
Is this the only server or every single device doesn't work as soon as they're 1G? Did you test with your own machine plugged in?