N7K ports are not up after server restart.

[Mehmet Can]

Hello Everyone,

I have a problem with our N7K. (We are using vPC)

Our server using two ports without Port-channel, because each port is used for different purposes.

Server Port 1: vlan 3 -> N7K-1- Eth3/35

Server Port 2: vlan 4 -> N7K-2- Eth3/35

Problem: When i restart the server, N7K1 and N7K2 ports are not up (port status are notconnected). I must to shutdown/no shutdown server ports on N7K1/N7K2. The problem is fixed when I do this.

port configuration,

N7K1

interface Ethernet 3/35

switchport mode access

switchport access vlan 3

spanning-tree port type edge

N7K2

interface Ethernet 3/35

switchport mode access

switchport access vlan 4

spanning-tree port type edge

How can i solve that problem? Can you help me?

[Mark Malone]

Hi

VPC ports should be identical on each switch in terms of configuration right down to the vlan id , they should not be set for different vlans or it will cause inconsistencies in the VPC feature, that might be whats causing your issue as when its trying to come back up it cant form VPC correctly , when it was down did you check the show vpc brief ? , you could check the sh vpc consistency-parameters vpc x too

[Mehmet Can]

Hi Mark and Thanks for reply,

vPC , works fine. Exactly our problem is not about vPC. As is said our server each port is used for different purposes. Not a backup of each ports.

[Mark Malone]

Ah ok sorry skimmed it a bit so its not using a VPC setup then each server port is single homed to each 7K orphan ports really   , are the ports down/down like hardware layer 1 issue will not come up ?

whats the SFP your using on them , and if you put a laptop or another device on say the same ports  as a test do you experience the same problem or is it just with this particular server ?

can you post the show int of one of the effected ports

[Mehmet Can]

Hi Mark,

That server using RJ-45 ports, but same problem have to our ESX servers. ESX servers are using 10GIG RJ-45 ports on N7K module.

N7K_2# sh int Ethernet3/35

Ethernet3/35 is up

admin state is up, Dedicated Interface

  Hardware: 1000/10000 Ethernet, address: XXXX.XXXX.XXXX

  Description: RPA6

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, medium is broadcast

  Port mode is access

  full-duplex, 1000 Mb/s

  Beacon is turned off

  Auto-Negotiation is turned on

  Input flow-control is off, output flow-control is off

  Auto-mdix is turned on

  Switchport monitor is off

  EtherType is 0x8100

  EEE (efficient-ethernet) : Disabled

  Last link flapped 01:41:51

  Last clearing of "show interface" counters 7w0d

  10 interface resets

  Load-Interval #1: 30 seconds

    30 seconds input rate 58936 bits/sec, 30 packets/sec

    30 seconds output rate 5747488 bits/sec, 779 packets/sec

    input rate 58.94 Kbps, 30 pps; output rate 5.75 Mbps, 779 pps

  Load-Interval #2: 5 minute (300 seconds)

    300 seconds input rate 59944 bits/sec, 9 packets/sec

    300 seconds output rate 3694936 bits/sec, 406 packets/sec

    input rate 59.94 Kbps, 9 pps; output rate 3.69 Mbps, 406 pps

  RX

    491433385 unicast packets  32 multicast packets  10406 broadcast packets

    491372763 input packets  509575907149 bytes

    0 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC/FCS  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    1056534789 unicast packets  176605422 multicast packets  174074078 broadcast packets

    1407140215 output packets  560544762871 bytes

    0 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  0 output discard

    0 Tx pause

That's output ESX server.. Restarted esx server and ports are not getting up. I did shut / no shut and worked.

N7K_2# sh int status | i 5501 ----> Server is up but ports are not connected

Eth6/9           TF2ESX5501      notconnec trunk     auto    auto    10g

N7K_2# conf t

Enter configuration commands, one per line.  End with CNTL/Z.

N7K_2(config)# int Eth6/9

N7K_2(config-if)# do sh run int Eth6/9

!Command: show running-config interface Ethernet6/9

version 6.2(16)

interface Ethernet6/9

  description TF2ESX5501

  switchport mode trunk

  spanning-tree port type edge trunk

  no shutdown

N7K_2(config-if)# shutdown

N7K_2(config-if)# no shutdown

N7K_2(config-if)#

N7K_2# sh int status | i 5501

Eth6/9           TF2ESX5501      connected trunk     full    a-10G   10g

N7K_2#

[Mark Malone]

so logically thinking about this as its a bit strange its effecting virtual and physical servers which makes me think it maybe software related , whats the code the switches are on ?

when a server starts up and is connected to a switch the fist thing that happens is an electrical connection between the pair at the physical layer and then the negotiation starts at Ethernet level , this connection is sent from the switch first I believe

the server port is obviously good as above no hits errors etc

did you try set the ports to hardset incase its something to do with negotiation ?

does it happen with standard pcs as a test , do you have to reboot the ports too for them ?

[Mehmet Can]

Yes, we have a really interesting problem..

Yes, i putting ESX cable to my pc same problem is during. Port light flash "orange" and i doing shut/no shut port light turn to "green" and problem solving.

I think problem is our 10 Gig RJ-45 Line cards :/

[Mehmet Can]

I opened case and Cisco said this is a bug. N7K 6.2.16 suggested version :)

Bug url:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva84959 

[Mark Malone]

thanks for posting back its good to know was software related , at least you didn't have to replace line cards :)

[NoPh]

Thanks for posting this. I am experiencing the same problem but on the recommended version 6.2.16 you were given. The bug link you provided states the known affected version as 6.2.16 so I'm assuming you moved to 6.2.18 or higher. I plan to move to this version and see if it resolves this issue.