Nexus 5K Error Logs : %ETHPORT-5-IF_DOWN_ADMIN_DOWN:

[balakrishnansenthil]

Hi Team,

We are getting some error logs like below on nexus switch 5K, Please check and let me know why we are getting this kind of error.

If you want any other logs please let me know.

Logs:

-------

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/40 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/41 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/42 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/43 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/44 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/45 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/46 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/47 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/48 is down (Administratively down)

2012 Dec 11 10:40:02 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

2012 Dec 11 10:40:02 NX5K %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-121 On-line

2012 Dec 11 10:40:02 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

NX5K#

NX5K#

NX5K# sh fex 121

FEX: 121 Description: FEX121   state: Online

  FEX version: 5.0(3)N1(1c) [Switch version: 5.0(3)N1(1c)]

  Extender Model: N2K-C2248TP-1GE,  Extender Serial: FOC16063CRQ

  Part No: 73-13232-01

pinning-mode: static    Max-links: 1

  Fabric port for control traffic: Eth1/5

  Fabric interface state:

    Po121 - Interface Up. State: Active

    Eth1/5 - Interface Up. State: Active

NX5K# `

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/44 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/45 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/46 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/47 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/48 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

2012 Nov 16 20:04:31 NX5K %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-121 On-line

2012 Nov 16 20:04:31 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

2012 Nov 19 21:07:40 NX5K %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel121: first operational port changed from Eth

ernet1/5 to none

2012 Nov 19 21:07:40 NX5K %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel121: Ethernet1/5 is down

2012 Nov 19 21:07:40 NX5K %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel121: port-channel121 is down

2012 Nov 19 21:07:40 NX5K %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 2 of Fex 121 that is connected with Ethernet1/5 ch

anged its status from Active to Disconnected

2012 Nov 19 21:07:40 NX5K %PFMA-2-FEX_STATUS: Fex 121 is offline

2012 Nov 19 21:07:40 NX5K %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-121 Off-line (Serial Number )

2012 Nov 19 21:07:40 NX5K %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet121/1/1 is down (Interface remove

d)

2012 Nov 19 21:07:40 NX5K %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet121/1/2 is down (Interface remove

d)

2012 Nov 19 21:07:40 NX5K %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet121/1/3 is down (Interface remove

d)

2012 Nov 19 21:07:40 NX5K %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet121/1/4 is down (Interface remove

d)

2012 Nov 19 21:07:40 NX5K %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet121/1/5 is down (Interface remove

d)

2012 Nov 19 21:07:40 NX5K %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet121/1/6 is down (Interface remove

NX5K# sh vpc brief

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 11

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : secondary, operational primary

Number of vPCs configured       : 99

Peer Gateway                    : Enabled

Dual-active excluded VLANs      : -

Graceful Consistency Check      : Enabled

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans

--   ----   ------ --------------------------------------------------

1    Po200  up     1,960-967,980-981,1100-1103,1105-1108,1110-1113,11

                   15-1118,1120-1123,1125-1128,1130-1133,1135-1138,11

                   40,1145,1150,1155,1160,1165,1300-1303,1305-1308,13

                   10-1313,1315-1318,1320-1323,1325-1328,1330-1333,13

                   35-1338,1340,1345,1350,1355,1360,1365,1500-1503,15   ....

vPC status

----------------------------------------------------------------------------

id     Port        Status Consistency Reason                     Active vlans

------ ----------- ------ ----------- -------------------------- -----------

21     Po21        up     success     success                    1,960-967,9

                                                                 80-981,1100

                                                                 -1103,1105-

                                                                 1108,1110-1

                                                                 113,111....

121    Po121       up     success     success                    -

122    Po122       up     success     success                    -

122880 Eth121/1/1  down*  failed      Consistency Check Not      -

                                      Performed

122881 Eth121/1/2  down*  failed      Consistency Check Not      -

                                      Performed

122882 Eth121/1/3  down*  failed      Consistency Check Not      -

                                      Performed

122883 Eth121/1/4  down*  failed      Consistency Check Not      -

                                      Performed

122884 Eth121/1/5  down*  failed      Consistency Check Not      -

                                      Performed

122885 Eth121/1/6  down*  failed      Consistency Check Not      -

                                      Performed

122886 Eth121/1/7  down*  failed      Consistency Check Not      -

                                      Performed

122887 Eth121/1/8  down*  failed      Consistency Check Not      -

                                      Performed

122888 Eth121/1/9  down*  failed      Consistency Check Not      -

                                      Performed

122889 Eth121/1/10 down*  failed      Consistency Check Not      -

                                      Performed

122890 Eth121/1/11 down*  failed      Consistency Check Not      -

                                      Performed

122891 Eth121/1/12 down*  failed      Consistency Check Not      -

                                      Performed

122892 Eth121/1/13 down*  failed      Consistency Check Not      -

                                      Performed

122893 Eth121/1/14 down*  failed      Consistency Check Not      -

                                      Performed

122894 Eth121/1/15 down*  failed      Consistency Check Not      -

                                      Performed

122895 Eth121/1/16 down*  failed      Consistency Check Not      -

                                      Performed

122896 Eth121/1/17 down*  failed      Consistency Check Not      -

                                      Performed

122897 Eth121/1/18 down*  failed      Consistency Check Not      -

                                      Performed

122898 Eth121/1/19 down*  failed      Consistency Check Not      -

                                      Performed

122899 Eth121/1/20 down*  failed      Consistency Check Not      -

                                      Performed

122900 Eth121/1/21 down*  failed      Consistency Check Not      -

                                      Performed

122901 Eth121/1/22 down*  failed      Consistency Check Not      -

                                      Performed

122902 Eth121/1/23 down*  failed      Consistency Check Not      -

                                      Performed

122903 Eth121/1/24 down*  failed      Consistency Check Not      -

                                      Performed

122904 Eth121/1/25 down*  failed      Consistency Check Not      -

                                      Performed

122905 Eth121/1/26 down*  failed      Consistency Check Not      -

                                      Performed

122906 Eth121/1/27 down*  failed      Consistency Check Not      -

                                      Performed

122907 Eth121/1/28 down*  failed      Consistency Check Not      -

                                      Performed

122908 Eth121/1/29 down*  failed      Consistency Check Not      -

                                      Performed

122909 Eth121/1/30 down*  failed      Consistency Check Not      -

                                      Performed

122910 Eth121/1/31 down*  failed      Consistency Check Not      -

                                      Performed

122911 Eth121/1/32 down*  failed      Consistency Check Not      -

                                      Performed

122912 Eth121/1/33 down*  failed      Consistency Check Not      -

                                      Performed

122913 Eth121/1/34 down*  failed      Consistency Check Not      -

                                      Performed

122914 Eth121/1/35 down*  failed      Consistency Check Not      -

                                      Performed

122915 Eth121/1/36 down*  failed      Consistency Check Not      -

                                      Performed

122916 Eth121/1/37 down*  failed      Consistency Check Not      -

                                      Performed

122917 Eth121/1/38 down*  failed      Consistency Check Not      -

                                      Performed

122918 Eth121/1/39 down*  failed      Consistency Check Not      -

                                      Performed

122919 Eth121/1/40 down*  failed      Consistency Check Not      -

                                      Performed

122920 Eth121/1/41 down*  failed      Consistency Check Not      -

                                      Performed

122921 Eth121/1/42 down*  failed      Consistency Check Not      -

                                      Performed

122922 Eth121/1/43 down*  failed      Consistency Check Not      -

                                      Performed

122923 Eth121/1/44 down*  failed      Consistency Check Not      -

                                      Performed

122924 Eth121/1/45 down*  failed      Consistency Check Not      -

                                      Performed

122925 Eth121/1/46 down*  failed      Consistency Check Not      -

                                      Performed

122926 Eth121/1/47 down*  failed      Consistency Check Not      -

                                      Performed

122927 Eth121/1/48 down*  failed      Consistency Check Not      -

                                      Performed

123904 Eth122/1/1  down*  failed      Consistency Check Not      -

                                      Performed

123905 Eth122/1/2  down*  failed      Consistency Check Not      -

                                      Performed

123906 Eth122/1/3  down*  failed      Consistency Check Not      -

                                      Performed

123907 Eth122/1/4  down*  failed      Consistency Check Not      -

                                      Performed

123908 Eth122/1/5  down*  failed      Consistency Check Not      -

                                      Performed

123909 Eth122/1/6  down*  failed      Consistency Check Not      -

                                      Performed

123910 Eth122/1/7  down*  failed      Consistency Check Not      -

                                      Performed

123911 Eth122/1/8  down*  failed      Consistency Check Not      -

                                      Performed

123912 Eth122/1/9  down*  failed      Consistency Check Not      -

                                      Performed

123913 Eth122/1/10 down*  failed      Consistency Check Not      -

                                      Performed

123914 Eth122/1/11 down*  failed      Consistency Check Not      -

                                      Performed

123915 Eth122/1/12 down*  failed      Consistency Check Not      -

                                      Performed

123916 Eth122/1/13 down*  failed      Consistency Check Not      -

                                      Performed

123917 Eth122/1/14 down*  failed      Consistency Check Not      -

                                      Performed

123918 Eth122/1/15 down*  failed      Consistency Check Not      -

                                      Performed

123919 Eth122/1/16 down*  failed      Consistency Check Not      -

                                      Performed

123920 Eth122/1/17 down*  failed      Consistency Check Not      -

                                      Performed

123921 Eth122/1/18 down*  failed      Consistency Check Not      -

                                      Performed

123922 Eth122/1/19 down*  failed      Consistency Check Not      -

                                      Performed

123923 Eth122/1/20 down*  failed      Consistency Check Not      -

                                      Performed

123924 Eth122/1/21 down*  failed      Consistency Check Not      -

                                      Performed

123925 Eth122/1/22 down*  failed      Consistency Check Not      -

                                      Performed

123926 Eth122/1/23 down*  failed      Consistency Check Not      -

                                      Performed

123927 Eth122/1/24 down*  failed      Consistency Check Not      -

                                      Performed

123928 Eth122/1/25 down*  failed      Consistency Check Not      -

                                      Performed

123929 Eth122/1/26 down*  failed      Consistency Check Not      -

                                      Performed

123930 Eth122/1/27 down*  failed      Consistency Check Not      -

                                      Performed

123931 Eth122/1/28 down*  failed      Consistency Check Not      -

                                      Performed

123932 Eth122/1/29 down*  failed      Consistency Check Not      -

                                      Performed

123933 Eth122/1/30 down*  failed      Consistency Check Not      -

                                      Performed

123934 Eth122/1/31 down*  failed      Consistency Check Not      -

                                      Performed

123935 Eth122/1/32 down*  failed      Consistency Check Not      -

                                      Performed

123936 Eth122/1/33 down*  failed      Consistency Check Not      -

                                      Performed

123937 Eth122/1/34 down*  failed      Consistency Check Not      -

                                      Performed

123938 Eth122/1/35 down*  failed      Consistency Check Not      -

                                      Performed

123939 Eth122/1/36 down*  failed      Consistency Check Not      -

                                      Performed

123940 Eth122/1/37 down*  failed      Consistency Check Not      -

                                      Performed

123941 Eth122/1/38 down*  failed      Consistency Check Not      -

                                      Performed

123942 Eth122/1/39 down*  failed      Consistency Check Not      -

                                      Performed

123943 Eth122/1/40 down*  failed      Consistency Check Not      -

                                      Performed

123944 Eth122/1/41 down*  failed      Consistency Check Not      -

                                      Performed

123945 Eth122/1/42 down*  failed      Consistency Check Not      -

                                      Performed

123946 Eth122/1/43 down*  failed      Consistency Check Not      -

                                      Performed

123947 Eth122/1/44 down*  failed      Consistency Check Not      -

                                      Performed

123948 Eth122/1/45 down*  failed      Consistency Check Not      -

                                      Performed

123949 Eth122/1/46 down*  failed      Consistency Check Not      -

                                      Performed

123950 Eth122/1/47 down*  failed      Consistency Check Not      -

                                      Performed

123951 Eth122/1/48 down*  failed      Consistency Check Not      -

                                      Performed

NX5K#

Please be make sure ASAP..........

Waiting for your answer?..

Regards,

Sen

[Mandlenkosi Nkiwane]

Do you have two Nexus 5Ks as peers and have the FEXs connected to both?

If so check to see if on both 5Ks you have same configuration for each port.

If you could provide the sh run that will be helpful as well.

[Oleksandr Nesterov]

Hi Senthilkumar

sequence of messages:

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/44 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/45 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/46 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/47 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/48 is down (Administratively down)

2012 Nov 16 20:04:31 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

2012 Nov 16 20:04:31 NX5K %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-121 On-line

2012 Nov 16 20:04:31 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

means that your fex just came online, and all ports are down yet.

If you see that your fex flaps too often please check powersupplies on it, also check fabric links.

HTH,

Alex

[balakrishnansenthil]

Hi Alex,

Thanks for your reply, We are getting same error logs on other switch like fex 121 online

PFMA-2-FEX_STATUS: Fex 121 is online

%NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-121 On-line

PFMA-2-FEX_STATUS: Fex 121 is online

Checked power supply status all everything fine.

Please give some idea to resolve this issue.

[Oleksandr Nesterov]

Hi Senthilkumar

Can you please post lost that repeats. Especially errors.

Last logs that were added just shows that FEX comes online.

Thanks

[balakrishnansenthil]

Hi

We are getting some error logs like below everyday.

Switch Details : Nexus 5K .

Logs:

--------

2013 Jan 16 10:31:15 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/44 is down (Administratively down)

2013 Jan 16 10:31:15 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/45 is down (Administratively down)

2013 Jan 16 10:31:15 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/46 is down (Administratively down)

2013 Jan 16 10:31:15 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/47 is down (Administratively down)

2013 Jan 16 10:31:15 NX5K %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet121/1/48 is down (Administratively down)

2013 Jan 16 10:31:15 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

2013 Jan 16 10:31:15 NX5K %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-121 On-line

2013 Jan 16 10:31:16 NX5K %PFMA-2-FEX_STATUS: Fex 121 is online

NX5K# sh fex

  FEX         FEX           FEX                       FEX

Number    Description      State            Model            Serial

------------------------------------------------------------------------

121         FEX121                Online     N2K-C2248TP-1GE   xxxxxxxxx

122         FEX122                Online     N2K-C2248TP-1GE   xxxxxxxxx

NX5K#

NX5K#

NX5K# sh fex 121

FEX: 121 Description: FEX121   state: Online

  FEX version: 5.0(3)N1(1c) [Switch version: 5.0(3)N1(1c)]

  Extender Model: N2K-C2248TP-1GE,  Extender Serial: xxxxxxxx

  Part No: 73-13232-01

pinning-mode: static    Max-links: 1

  Fabric port for control traffic: Eth1/5

  Fabric interface state:

    Po121 - Interface Up. State: Active

    Eth1/5 - Interface Up. State: Active

NX5K#

NX5K# sh environment fex 121

Temperature Fex 121:

-----------------------------------------------------------------

Module   Sensor     MajorThresh   MinorThres   CurTemp     Status

                    (Celsius)     (Celsius)    (Celsius)

-----------------------------------------------------------------

1        Outlet-1   63            45           24          ok

1        Outlet-2   57            45           28          ok

1        Inlet-1    65            55           26          ok

1        Die-1      100           85           36          ok

Fan Fex: 121:

------------------------------------------------------

Fan             Model                Hw         Status

------------------------------------------------------

Chassis         N2K-C2248-FAN        --         ok

PS-1            N2200-PAC-400W       --         ok

PS-2            N2200-PAC-400W       --         ok

Power Supply Fex 121:

---------------------------------------------------------------------------

Voltage: 12 Volts

-----------------------------------------------------

PS  Model                Power       Power     Status

                         (Watts)     (Amp)

-----------------------------------------------------

1   N2200-PAC-400W        396.00     33.00     ok

2   N2200-PAC-400W        396.00     33.00     ok

Mod Model                Power     Power       Power     Power       Status

                         Requested Requested   Allocated Allocated

                         (Watts)   (Amp)       (Watts)   (Amp)

--- -------------------  -------   ----------  --------- ----------  ----------

1    N2K-C2248TP-1GE     58.80     4.90        58.80     4.90        powered-up

Power Usage Summary:

--------------------

Power Supply redundancy mode:                 redundant

Total Power Capacity                              792.00 W

Power reserved for Supervisor(s)                   58.80 W

Power currently used by Modules                     0.00 W

                                                -------------

Total Power Available                             733.20 W

                                                -------------

NX5K#

[Gabriel Hill]

Would you post the following:

sh port-channel summary

sh interface port-channel 121

sh interface ethernet1/5

Also how are your FEX's setup? Are you dual homing them between two 5k's are are they singled homed to just one 5k?

[balakrishnansenthil]

Thanks Hill,

I have pasted all the requested logs below....

Yes,It's dual homed , we are using for redundancy for another 5K( Two 5k's).

Logs;

-------

NX5K# sh port-channel summary

Flags:  D - Down        P - Up in port-channel (members)

        I - Individual  H - Hot-standby (LACP only)

        s - Suspended   r - Module-removed

        S - Switched    R - Routed

        U - Up (port-channel)

--------------------------------------------------------------------------------

Group Port-       Type     Protocol  Member Ports

      Channel

--------------------------------------------------------------------------------

21    Po21(SU)    Eth      LACP      Eth1/1(P)    Eth1/2(P)

121   Po121(SU)   Eth      NONE      Eth1/5(P)

122   Po122(SU)   Eth      NONE      Eth1/6(P)

200   Po200(SU)   Eth      LACP      Eth1/3(P)    Eth1/4(P)

NX5K#

NX5K#

NX5K# sh int port-channel 121

port-channel121 is up

vPC Status: Up, vPC number: 121

  Hardware: Port-Channel, address: xxxxxxxxx

  Description: VPC Etherchannel to Nexus2K

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is fex-fabric

  full-duplex, 10 Gb/s

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Switchport monitor is off

  EtherType is 0x8100

  Members in this channel: Eth1/5

  Last clearing of "show interface" counters never

  30 seconds input rate 25344 bits/sec, 3168 bytes/sec, 1 packets/sec

  30 seconds output rate 2352 bits/sec, 294 bytes/sec, 1 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 23.13 Kbps, 2 pps; output rate 2.20 Kbps, 1 pps

  RX

    65742824 unicast packets  6938469 multicast packets  15067 broadcast packets

    72696360 input packets  72368121160 bytes

    39108278 jumbo packets  0 storm suppression bytes

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    26659449 unicast packets  7045536 multicast packets  257 broadcast packets

    33705242 output packets  7222924904 bytes

    0 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble

    0 Tx pause

  618 interface resets

NX5K#

NX5K# sh interface ethernet 1/5

Ethernet1/5 is up

  Hardware: 1000/10000 Ethernet, address: xxxxxxxxxx

  Description: Nexus 2K - NOKCPXX-NX2K021

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is fex-fabric

  full-duplex, 10 Gb/s, media type is 10G

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Rate mode is dedicated

  Switchport monitor is off

  EtherType is 0x8100

  Last link flapped 12:47:36

  Last clearing of "show interface" counters never

  30 seconds input rate 34408 bits/sec, 4301 bytes/sec, 1 packets/sec

  30 seconds output rate 2808 bits/sec, 351 bytes/sec, 1 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 34.60 Kbps, 3 pps; output rate 2.28 Kbps, 1 pps

  RX

    65742868 unicast packets  6938472 multicast packets  15067 broadcast packets

    72696407 input packets  72368173376 bytes

    39108308 jumbo packets  0 storm suppression bytes

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    26659463 unicast packets  7045539 multicast packets  257 broadcast packets

    33705259 output packets  7222928160 bytes

    0 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble

    0 Tx pause

  618 interface resets

NX5K#

[balakrishnansenthil]

When i checked " sh fex details" it's showing below logs for fex 121.

      Eth121/1/44  Down        None

      Eth121/1/45  Down        None

      Eth121/1/46  Down        None

      Eth121/1/47  Down       Po121

      Eth121/1/48  Down       Po121

Logs:

01/14/2013 15:11:51.617771: Module Online

01/14/2013 15:30:27.56599: Deleting route to FEX

01/14/2013 15:30:27.61663: Module disconnected

01/14/2013 15:30:27.62860: Offlining Module

01/14/2013 15:30:27.63524: Module Offline Sequence

01/14/2013 15:30:27.759237: Module Offline

01/14/2013 15:31:04.57251: Module register received

01/14/2013 15:31:04.63233: Registration response sent

01/14/2013 15:31:04.180789: Module Online Sequence

01/14/2013 15:31:06.304134: Module Online

01/15/2013 01:00:17.660104: Deleting route to FEX

01/15/2013 01:00:17.669093: Module disconnected

01/15/2013 01:00:17.670332: Offlining Module

01/15/2013 01:00:17.671002: Module Offline Sequence

01/15/2013 01:00:18.228589: Module Offline

01/15/2013 01:00:54.263629: Module register received

01/15/2013 01:00:54.269597: Registration response sent

01/15/2013 01:00:54.386993: Module Online Sequence

01/15/2013 01:00:56.645328: Module Online

01/15/2013 01:24:52.156439: Deleting route to FEX

01/15/2013 01:24:52.162223: Module disconnected

01/15/2013 01:24:52.163448: Offlining Module

01/15/2013 01:24:52.164126: Module Offline Sequence

01/15/2013 01:24:52.783929: Module Offline

01/15/2013 01:25:28.7248: Module register received

01/15/2013 01:25:28.13484: Registration response sent

01/15/2013 01:25:28.127899: Module Online Sequence

01/15/2013 01:25:30.368767: Module Online

01/15/2013 05:20:57.985081: Deleting route to FEX

01/15/2013 05:20:57.990061: Module disconnected

01/15/2013 05:20:57.991068: Offlining Module

01/15/2013 05:20:57.991751: Module Offline Sequence

01/15/2013 05:20:58.557559: Module Offline

01/15/2013 05:21:35.214382: Module register received

01/15/2013 05:21:35.220519: Registration response sent

01/15/2013 05:21:35.336923: Module Online Sequence

01/15/2013 05:21:37.497165: Module Online

01/15/2013 07:13:48.753789: Deleting route to FEX

01/15/2013 07:13:48.759098: Module disconnected

01/15/2013 07:13:48.760342: Offlining Module

01/15/2013 07:13:48.761069: Module Offline Sequence

01/15/2013 07:13:49.357563: Module Offline

01/15/2013 07:14:26.204863: Module register received

01/15/2013 07:14:26.210923: Registration response sent

01/15/2013 07:14:26.327326: Module Online Sequence

01/15/2013 07:14:28.444356: Module Online

01/15/2013 07:29:58.703190: Deleting route to FEX

01/15/2013 07:29:58.710180: Module disconnected

01/15/2013 07:29:58.711436: Offlining Module

01/15/2013 07:29:58.712116: Module Offline Sequence

01/15/2013 07:29:59.289830: Module Offline

01/15/2013 07:30:34.551855: Module register received

01/15/2013 07:30:34.557942: Registration response sent

01/15/2013 07:30:34.672304: Module Online Sequence

01/15/2013 07:30:36.831234: Module Online

01/16/2013 10:30:38.388497: Deleting route to FEX

01/16/2013 10:30:38.394546: Module disconnected

01/16/2013 10:30:38.395790: Offlining Module

01/16/2013 10:30:38.396473: Module Offline Sequence

01/16/2013 10:30:38.990842: Module Offline

01/16/2013 10:31:13.90472: Module register received

01/16/2013 10:31:13.96516: Registration response sent

01/16/2013 10:31:13.214606: Module Online Sequence

01/16/2013 10:31:15.813230: Module Online

Thanks....

[Gabriel Hill]

Okay. From all your logs it looks like this is only happening to FEX 121. Is that correct or is this happening to other FEX's as well.

I should have been more specific. From both 5k's post:

sh interface port-channel 121

sh interface ethernet1/5

sh running interface ethernet 1/5 (just to verify configs match from both sides)

I want to know if the FEX is losing connectivity to both or just one 5k. Also is UDLD configured on those interfaces?

sh udld ethernet 1/5

Thanks,

[balakrishnansenthil]

Thanks for update....

This one only happening on one switch only another switch is working fine.

Logs for another switch:

-------------------------------------

NX5K# sh interface port-channel 121

port-channel121 is up

vPC Status: Up, vPC number: 121

  Hardware: Port-Channel, address: xxxxxxxxxxxx

  Description: VPC Etherchannel to Nexus2K

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is fex-fabric

  full-duplex, 10 Gb/s

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Switchport monitor is off

  EtherType is 0x8100

  Members in this channel: Eth1/5

  Last clearing of "show interface" counters never

  30 seconds input rate 34664 bits/sec, 4333 bytes/sec, 3 packets/sec

  30 seconds output rate 2776 bits/sec, 347 bytes/sec, 1 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 23.98 Kbps, 2 pps; output rate 2.25 Kbps, 1 pps

  RX

    53867785 unicast packets  5689200 multicast packets  1 broadcast packets

    59556986 input packets  59357255207 bytes

    32093532 jumbo packets  0 storm suppression bytes

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    21774939 unicast packets  5781625 multicast packets  6 broadcast packets

    27556570 output packets  5850375464 bytes

    0 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble

    0 Tx pause

  1 interface resets

NX5K# sh interface ethernet1/5

Ethernet1/5 is up

  Hardware: 1000/10000 Ethernet, address: xxxxxxxxxxxx

  Description: Nexus 2K - NOKCPXX-NX2K021

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is fex-fabric

  full-duplex, 10 Gb/s, media type is 10G

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Rate mode is dedicated

  Switchport monitor is off

  EtherType is 0x8100

  Last link flapped 29week(s) 3day(s)

  Last clearing of "show interface" counters never

  30 seconds input rate 27776 bits/sec, 3472 bytes/sec, 2 packets/sec

  30 seconds output rate 2648 bits/sec, 331 bytes/sec, 0 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 22.59 Kbps, 2 pps; output rate 2.25 Kbps, 1 pps

  RX

    53867829 unicast packets  5689203 multicast packets  1 broadcast packets

    59557033 input packets  59357307423 bytes

    32093562 jumbo packets  0 storm suppression bytes

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    21774953 unicast packets  5781628 multicast packets  6 broadcast packets

    27556587 output packets  5850378720 bytes

    0 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble

    0 Tx pause

  1 interface resets

NX5K# sh running interface ethernet 1/5

!Command: show running-config interface Ethernet1/5

!Time: Thu Jan 17 22:55:39 2013

version 5.0(3)N1(1c)

interface Ethernet1/5

  description Nexus 2K - NOKCPXX-NX2K021

  switchport mode fex-fabric

  fex associate 121

  channel-group 121

  no shutdown

NX5K#

Regards,

Sen

[Gabriel Hill]

Sen,

I would start with first placing your SFP modules between the troubled 5k and 2k. If the interfaces continues to reset I would then replace the fiber cable between the two. After these steps post back if the issue is not resolved.

[balakrishnansenthil]

Sorry, You mean the Uplink cable between 2k and 5k  is it?......

Thanks.....

Regards,

Sen

[Gabriel Hill]

Yes the uplink cable and the SFP module (transceiver) the cable plugs into.

[balakrishnansenthil]

Thanks for reply..........

Please let me know how to check the redundancy on the uplink?.

We want to change the cable now how to switchover all the traffic to other switch?...

Please let me know the steps ASAP.

Regards,

Sen