Nexus 5548UP + B22HP FEX Cards - Constant Interface Up/Down Log Entries without topology change

Jonathan Bekkers · ‎10-18-2014

2x Nexus 5548UP + 2x B22HP FEX modules single-homed connected. IOS 5.2(1)N1(6). Seems to be a cosmetic issue and is an annoyance. I'm unsure whether this is normal FEX behaviour or something specific to these B22HP cards (which is basically a FEX for a HP-UX blade-chassis).

I am constantly getting log entries per the below:

2009 Mar 9 12:10:39 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/15 is down (Administratively down)
2009 Mar 9 12:10:50 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/3 is down (Administratively down)
2009 Mar 9 12:11:00 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/4 is down (Administratively down)
2009 Mar 9 12:11:10 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/5 is down (Administratively down)
2009 Mar 9 12:11:20 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/6 is down (Administratively down)
2009 Mar 9 12:11:30 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/11 is down (Administratively down)
2009 Mar 9 12:11:40 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/12 is down (Administratively down)
2009 Mar 9 12:11:50 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/13 is down (Administratively down)
2009 Mar 9 12:12:00 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/14 is down (Administratively down)
2009 Mar 9 12:12:10 N5K-SW02 %ETHPORT-3-IF_DOWN_ADMIN_DOWN: Interface Ethernet102/1/15 is down (Administratively down)

Note that the above ports are all "administratively down" on purpose and are not changing state. The above repeat constantly about once every 2mins or so for each NIC. If the ports are not administrately down we get similar issues but with a different error message and again every 2 mins.

PLEASE NOTE: These ports are all meant to be down as they relate to blade slots that are empty.

2014 Oct 18 12:05:18 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/14 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:05:28 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/15 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:05:38 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/3 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:05:48 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/4 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:05:58 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/5 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:06:08 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/6 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:06:18 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/11 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:06:28 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/12 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:06:38 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/13 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:06:48 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/14 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:06:58 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/15 is down (Error disabled. Reason:ekeying triggered)
2014 Oct 18 12:07:08 N5K-SW01%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet101/1/3 is down (Error disabled. Reason:ekeying triggered)

We don't get any repeating log entries for ports that are "up" just the ones that are "down" or "admin down".

Anyone have any ideas? This seems to be cosmetic however since this is a fresh build and as I don't like cutting over to something with constant errors I'd really like this answered.

Gurpreet Kochar · ‎10-18-2014

Hi Jonathan,

Please see this

https://tools.cisco.com/bugsearch/bug/CSCul25050/?reffering_site=dumpcr

Thanks

GK

Jonathan Bekkers · ‎10-19-2014

ON NEXUS 2

(Where "down" B22HP ports are "administratively shut")

show fex detail 102

FEX: 102 Description: HP-UX B22HP #2 state: Online
FEX version: 5.2(1)N1(6) [Switch version: 5.2(1)N1(6)]
FEX Interim version: 5.2(1)N1(6)
Switch Interim version: 5.2(1)N1(6)
Extender Serial: ***********
Extender Model: N2K-B22HP-P, Part No: 73-13780-05
Bay: 2
Rack: AR31RACK7
Enclosure: melc701
Enclosure Serial: **************
Rack Id: Default RUID
Card Id: 145, Mac Addr: 50:87:89:9c:93:c2, Num Macs: 64
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/3
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po102 - Interface Up. State: Active
Eth1/3 - Interface Up. State: Active
Eth1/4 - Interface Up. State: Active
Eth1/5 - Interface Up. State: Active
Eth1/6 - Interface Up. State: Active
Fex Port State Fabric Port
Eth102/1/1 Up Po102
Eth102/1/2 Up Po102
Eth102/1/3 Down Po102
Eth102/1/4 Down Po102
Eth102/1/5 Down Po102
Eth102/1/6 Down Po102
Eth102/1/7 Up Po102
Eth102/1/8 Up Po102
Eth102/1/9 Up Po102
Eth102/1/10 Up Po102
Eth102/1/11 Down Po102
Eth102/1/12 Down Po102
Eth102/1/13 Down Po102
Eth102/1/14 Down Po102
Eth102/1/15 Down Po102
Eth102/1/16 Up Po102
Logs:
03/09/2009 02:38:23.858178: Module register received
03/09/2009 02:38:23.865762: Registration response sent
03/09/2009 02:38:23.973288: Deleting route to FEX
03/09/2009 02:38:23.979627: Module disconnected
03/09/2009 02:38:23.981509: Module Offline
03/09/2009 02:38:23.983202: Deleting route to FEX
03/09/2009 02:38:23.989408: Module disconnected
03/09/2009 02:38:24.720: Offlining Module
03/09/2009 02:38:24.1640: Deleting route to FEX
03/09/2009 02:38:24.8140: Module disconnected
03/09/2009 02:38:24.9068: Offlining Module
03/09/2009 02:38:53.870056: Module timed out
03/09/2009 02:38:57.929625: Module register received
03/09/2009 02:38:57.937157: Registration response sent
03/09/2009 02:38:58.277671: Module Online Sequence
03/09/2009 02:38:59.639014: Module Online
03/09/2009 02:43:32.131569: Deleting route to FEX
03/09/2009 02:43:32.137928: Module disconnected
03/09/2009 02:43:32.138419: Offlining Module
03/09/2009 02:43:32.139368: Module Offline Sequence
03/09/2009 02:43:32.487749: Module Offline
03/09/2009 02:43:33.802096: Deleting route to FEX
03/09/2009 02:43:33.808777: Module disconnected
03/09/2009 02:43:33.809929: Offlining Module
03/09/2009 02:43:33.819480: Deleting route to FEX
03/09/2009 02:43:33.825866: Module disconnected
03/09/2009 02:43:33.827850: Offlining Module
03/09/2009 02:57:04.785223: Module register received
03/09/2009 02:57:04.786233: Registration response sent
03/09/2009 02:57:05.124862: Module Online Sequence
03/09/2009 02:57:06.854861: Module Online

show int status fex 102

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth102/1/1 HP-UX - BAY 1/2/9/ connected trunk full a-10G --
Eth102/1/2 HP-UX - BAY 1/2/9/ connected 1 full a-10G --
Eth102/1/3 -- disabled 1 auto auto --
Eth102/1/4 -- disabled 1 auto auto --
Eth102/1/5 -- disabled 1 auto auto --
Eth102/1/6 -- disabled 1 auto auto --
Eth102/1/7 HP-UX - BAY 7 - ME connected trunk full a-10G --
Eth102/1/8 HP-UX - BAY 8 - ME connected trunk full a-10G --
Eth102/1/9 HP-UX - BAY 1/2/9/ connected trunk full a-10G --
Eth102/1/10 HP-UX - BAY 1/2/9/ connected 1 full a-10G --
Eth102/1/11 -- disabled 1 auto auto --
Eth102/1/12 -- disabled 1 auto auto --
Eth102/1/13 -- disabled 1 auto auto --
Eth102/1/14 -- disabled 1 auto auto --
Eth102/1/15 -- disabled 1 auto auto --
Eth102/1/16 HP-UX - BAY 16 - M connected trunk full a-10G --

Jonathan Bekkers · ‎10-19-2014

Almost. That bug suggests the issue is fixed by shutting a port. This isn't the case for me. I just get a variation of the log entries that now say "port is admin down" constantly every minute.

ON NEXUS 1

(Where ports are left "down" in error state)

show fex detail

FEX: 101 Description: HP-UX B22HP #1 state: Online

FEX version: 5.2(1)N1(6) [Switch version: 5.2(1)N1(6)]

FEX Interim version: 5.2(1)N1(6)

Switch Interim version: 5.2(1)N1(6)

Extender Serial: ***********

Extender Model: N2K-B22HP-P, Part No: 73-13780-05

Bay: 1

Rack: AR31RACK7

Enclosure: melc701

Enclosure Serial: ***********

Rack Id: Default RUID

Card Id: 145, Mac Addr: 50:87:89:9c:2f:42, Num Macs: 65600

Module Sw Gen: 12594 [Switch Sw Gen: 21]

post level: complete

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/3

FCoE Admin: false

FCoE Oper: true

FCoE FEX AA Configured: false

Fabric interface state:

Po101 - Interface Up. State: Active

Eth1/3 - Interface Up. State: Active

Eth1/4 - Interface Up. State: Active

Eth1/5 - Interface Up. State: Active

Eth1/6 - Interface Up. State: Active

Fex Port State Fabric Port

Eth101/1/1 Up Po101

Eth101/1/2 Up Po101

Eth101/1/3 Down Po101

Eth101/1/4 Down Po101

Eth101/1/5 Down Po101

Eth101/1/6 Down Po101

Eth101/1/7 Up Po101

Eth101/1/8 Up Po101

Eth101/1/9 Up Po101

Eth101/1/10 Up Po101

Eth101/1/11 Down Po101

Eth101/1/12 Down Po101

Eth101/1/13 Down Po101

Eth101/1/14 Down Po101

Eth101/1/15 Down Po101

Eth101/1/16 Up Po101

Logs:

10/17/2014 07:05:47.225362: Module disconnected

10/17/2014 07:05:47.227077: Offlining Module

10/17/2014 07:05:55.403637: Module register received

10/17/2014 07:05:55.404791: Registration response sent

10/17/2014 07:05:55.902758: Module Online Sequence

10/17/2014 07:05:57.768018: Module Online

10/18/2014 02:09:57.459686: Deleting route to FEX

10/18/2014 02:09:57.466063: Module disconnected

10/18/2014 02:09:57.466546: Deleting route to FEX

10/18/2014 02:09:57.472627: Module disconnected

10/18/2014 02:09:57.473032: Offlining Module

10/18/2014 02:09:57.473491: Module Offline Sequence

10/18/2014 02:09:57.833046: Module Offline

10/18/2014 02:10:37.233778: Module register received

10/18/2014 02:10:37.234937: Registration response sent

10/18/2014 02:10:37.572000: Module Online Sequence

10/18/2014 02:10:39.28982: Module Online

10/18/2014 02:46:50.34357: Deleting route to FEX

10/18/2014 02:46:50.41777: Module disconnected

10/18/2014 02:46:50.42310: Deleting route to FEX

10/18/2014 02:46:50.48507: Module disconnected

10/18/2014 02:46:50.48910: Offlining Module

10/18/2014 02:46:50.49424: Module Offline Sequence

10/18/2014 02:46:50.426122: Module Offline

10/18/2014 02:46:50.567076: Deleting route to FEX

10/18/2014 02:46:50.573487: Module disconnected

10/18/2014 02:46:50.574429: Offlining Module

10/18/2014 02:49:06.119573: Module register received

10/18/2014 02:49:06.120740: Registration response sent

10/18/2014 02:49:06.455574: Deleting route to FEX

10/18/2014 02:49:06.463018: Module disconnected

10/18/2014 02:49:06.468630: Module Offline

10/18/2014 02:49:06.470796: Deleting route to FEX

10/18/2014 02:49:06.477068: Module disconnected

10/18/2014 02:49:06.479851: Offlining Module

10/18/2014 02:49:36.122581: Module timed out

10/18/2014 02:49:37.312507: Deleting route to FEX

10/18/2014 02:49:37.318953: Module disconnected

10/18/2014 02:49:37.319563: Deleting route to FEX

10/18/2014 02:49:37.326713: Module disconnected

10/18/2014 02:49:37.327648: Offlining Module

10/18/2014 02:49:37.328005: Offlining Module

10/18/2014 02:49:55.633427: Module register received

10/18/2014 02:49:55.634436: Registration response sent

10/18/2014 02:49:55.976219: Deleting route to FEX

10/18/2014 02:49:55.984003: Module disconnected

10/18/2014 02:49:55.984784: Module Offline

10/18/2014 02:49:55.988979: Deleting route to FEX

10/18/2014 02:49:55.996236: Module disconnected

10/18/2014 02:49:55.997205: Offlining Module

10/18/2014 02:50:25.640151: Module timed out

10/18/2014 02:50:30.433513: Module register received

10/18/2014 02:50:30.434516: Registration response sent

10/18/2014 02:50:30.776356: Deleting route to FEX

10/18/2014 02:50:30.788925: Module disconnected

10/18/2014 02:50:30.789619: Module Offline

10/18/2014 02:50:30.791322: Deleting route to FEX

10/18/2014 02:50:30.797533: Module disconnected

10/18/2014 02:50:30.812722: Offlining Module

10/18/2014 02:51:00.440217: Module timed out

10/18/2014 02:51:02.952487: Module register received

10/18/2014 02:51:02.953496: Registration response sent

10/18/2014 02:51:03.291286: Module Online Sequence

10/18/2014 02:51:05.116850: Module Online

show version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

BIOS: version 3.6.0

loader: version N/A

kickstart: version 5.2(1)N1(6)

system: version 5.2(1)N1(6)

power-seq: Module 1: version v2.0

Module 3: version v2.0

uC: version v1.2.0.1

SFP uC: Module 1: v1.1.0.0

BIOS compile time: 05/09/2012

kickstart image file is: bootflash:///n5000-uk9-kickstart.5.2.1.N1.6.bin

kickstart compile time: 10/10/2013 23:00:00 [10/11/2013 07:26:14]

system image file is: bootflash:///n5000-uk9.5.2.1.N1.6.bin

system compile time: 10/10/2013 23:00:00 [10/11/2013 08:53:50]

Hardware

cisco Nexus5548 Chassis ("O2 32X10GE/Modular Universal Platform Supervisor")

Intel(R) Xeon(R) CPU with 8263828 kB of memory.

Processor Board ID ***********

Device name: PFD-EXH-N5K-SW01

bootflash: 2007040 kB

Kernel uptime is 1 day(s), 5 hour(s), 50 minute(s), 29 second(s)

Last reset at 675509 usecs after Fri Oct 17 06:32:03 2014

Reason: Disruptive upgrade

System version: 5.1(3)N2(1)

Service:

plugin

Core Plugin, Ethernet Plugin

show interface status fex 101

--------------------------------------------------------------------------------

Port Name Status Vlan Duplex Speed Type

--------------------------------------------------------------------------------

Eth101/1/1 HP-UX - BAY 1/2/9/ connected trunk full a-10G --

Eth101/1/2 HP-UX - BAY 1/2/9/ connected 1 full a-10G --

Eth101/1/3 -- eKeyingEr 1 auto auto --

Eth101/1/4 -- eKeyingEr 1 auto auto --

Eth101/1/5 -- eKeyingEr 1 auto auto --

Eth101/1/6 -- eKeyingEr 1 auto auto --

Eth101/1/7 HP-UX - BAY 7 - ME connected trunk full a-10G --

Eth101/1/8 HP-UX - BAY 8 - ME connected trunk full a-10G --

Eth101/1/9 HP-UX - BAY 1/2/9/ connected trunk full a-10G --

Eth101/1/10 HP-UX - BAY 1/2/9/ connected 1 full a-10G --

Eth101/1/11 -- eKeyingEr 1 auto auto --

Eth101/1/12 -- eKeyingEr 1 auto auto --

Eth101/1/13 -- eKeyingEr 1 auto auto --

Eth101/1/14 -- eKeyingEr 1 auto auto --

Eth101/1/15 -- eKeyingEr 1 auto auto --

Eth101/1/16 HP-UX - BAY 16 - M connected trunk full a-10G --