Switch Port Troubleshooting.

ray_stone · ‎05-28-2012

Hi Experts:

There are two Switches connected with ESX Server and port ethernet 0/19 being used which is configured in Trunk port and now server team is reporting that the port does not seems to have been working fine.

I checked the switch port status and found the below stats:

3-admn# sh int ethernet 1/19
Ethernet1/19 is down (pauseRateLimitErrDisable)
Hardware is 10000 Ethernet, address is 000d.ecf6.235a (bia 000d.ecf6.235a)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
    reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
auto-duplex, 10 Gb/s, media type is 10g
Input flow-control is on, output flow-control is off
Auto-mdix is turned on
Rate mode is dedicated
Switchport monitor is off
Last clearing of "show interface" counters never
5 minute input rate 0 bytes/sec, 0 packets/sec
5 minute output rate 0 bytes/sec, 0 packets/sec
Rx
    238796366 input packets 238750261 unicast packets 45685 multicast packets
    420 broadcast packets 126013405 jumbo packets 0 storm suppression packets
    1039029980064 bytes
    0 No buffer 0 runt 0 Overrun
    0 crc 0 Ignored 0 Bad etype drop
    0 Bad proto drop
Tx
    353714828 output packets 38671888 multicast packets
    75897612 broadcast packets 130489679 jumbo packets
    1083376474176 bytes
    1 output CRC 0 ecc
    0 underrun 0 if down drop    0 output error 0 collision 0 deferred
    0 late collision 0 lost carrier 0 no carrier
    0 babble
    45685 Rx pause 0 Tx pause 0 reset

4-admn# sh interface ethernet 1/19
Ethernet1/19 is down (pauseRateLimitErrDisable)
Hardware is 10000 Ethernet, address is 000d.ecf6.1c9a (bia 000d.ecf6.1c9a)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
    reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
auto-duplex, 10 Gb/s, media type is 10g
Input flow-control is on, output flow-control is off
Auto-mdix is turned on
Rate mode is dedicated
Switchport monitor is off
Last clearing of "show interface" counters 86w6d
5 minute input rate 0 bytes/sec, 0 packets/sec
5 minute output rate 0 bytes/sec, 0 packets/sec
Rx
    85932103994 input packets 85931903050 unicast packets 27872 multicast packets
    173072 broadcast packets 50722478635 jumbo packets 0 storm suppression packets
    287375253166073 bytes
    0 No buffer 0 runt 0 Overrun
    0 crc 0 Ignored 0 Bad etype drop
    0 Bad proto drop
Tx
    73040264335 output packets 27839138 multicast packets
    61844640 broadcast packets 32775514649 jumbo packets
    202201808046510 bytes
    1 output CRC 0 ecc
    0 underrun 0 if down drop    0 output error 0 collision 0 deferred
    0 late collision 0 lost carrier 0 no carrier
    0 babble
    27872 Rx pause 0 Tx pause 0 reset

Need your help to understand the problem as I am not best in Switch troubleshooting. Please help to understand the cause of the problem and provide the solution.

Thanks.

Leo Laohoo · ‎05-28-2012

Ethernet1/19 is down (pauseRateLimitErrDisable)

Error disable?

ray_stone · ‎05-28-2012

what does it mean and why it caused... any specific reasons....

Tagir Temirgaliyev · ‎05-28-2012

Hi

I did connect ESX Server to switch is configured as access port

ray_stone · ‎05-28-2012

Multiple V-LAN configured on ESX Server and to communicate with each other you need to configure switch interface as a Trunk Port so that it could support multiple v-lans traffic that you want to allow.

I am just wondering as to why the error disable encountered and what is a solution to fix it.

Thanks.

nkarpysh · ‎05-28-2012

Ports are pauseRateLimitErrDisable - meaning Nexus is getting too much Pause frames from server:

27872 Rx pause 0 Tx pause 0 reset

45685 Rx pause 0 Tx pause 0 reset

So Nexus receives excessive Xoff pause frames from the server, then ports become error-disabled due to the high rate of pause frames received.

Usually the port will go into an err-disable state due o pause frames, only if the drain rate is less than 5mbps on a 10G
port. This means that the server is very slow (or misbehaving) and is sending a large
number of pause frames to the switch ports.

I am sending you the following link for your reference:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_fcoe.html#wp1030594

So the problem is most likely with slow or misbehaving server which send Pause frame again and again even on slow traffic rate. Check that with server team and then unblock Nexus port (shutdown/no shutdown if needed).

Nik

+ grade if it helped

HTH,
Niko

Leo Laohoo · ‎05-29-2012

+5 Nik.

Learn something new each day!

nkarpysh · ‎05-30-2012

Thanks Leo.

HTH,
Niko