Nexux 5596-UP "show hardware internal" command problem

[Iker Santamaria Molla]

Hi all,

I have a problem with one of our Nexus 5596. When I execute the following command on module 1 for any fc port, the switch displays an error:

show hardware internal fc-mac 1 port <fc_port_ID> port-event

**************************************************************************************************************************

switch-n5k-2# sh hardware internal fc-mac 1 port 43 port-event

gat_fcp_cli_port_port_event: dev_debug() returned error, errno = 22, rc = -1

switch-n5k-2# sh hardware internal fc-mac 1 port 44 link-event

gat_fcp_cli_port_link_event: dev_debug() returned error, errno = 22, rc = -1

******************************************************************************************************

The NX-OS version is: 5.1(3)N2(1b) and ports conecct to MDS-9000 and Servers.

Has someone else this problem?

Thanks in advance!

[chald]

Hello Iker,

I have only the same problem if I issue the same command on a non existing port.

Remember how the 5596 is build you have 32 ports in slot 1 and then slot 2, 3 and 4. with each 16 ports

So it is only a cosmetic bug that you can issue it and if you list your interface with a show interface brief you can see it.

running code 5.2(1)N1(4)

N5KSW1# sh hardware internal fc-mac 1 port 32 port-event

MMDDYY HHMMSS usecs Event Current State

-------------------- ----- -------------

- - IPS_DOWN

TL-N85R5-N5K# sh hardware internal fc-mac 2 port 1 port-event

MMDDYY HHMMSS usecs          Event                           Current State

--------------------         -----                           -------------

-                            -                               IPS_LINK_UP      

082813 111939 841857 (0000)  E_IPS_LINK_INIT_SUC             IPS_LINK_INIT    

082813 111939 841843 (0001)  E_IPS_IDLE                      IPS_LINK_INIT    

082813 111939 838663 (0000)  E_IPS_LR                        IPS_LINK_INIT    

082813 111939 832533 (0001)  E_IPS_TMO                       IPS_LINK_INIT    

082813 111939 822555 (0000)  E_IPS_LINK_INIT                 IPS_SPEED

N5KSW1# sh hardware internal fc-mac 1 port 32 port-event
MMDDYY HHMMSS usecs Event Current State
-------------------- ----- -------------
- - IPS_DOWN

N5KSW1# sh hardware internal fc-mac 1 port 33 port-event

gat_fcp_cli_port_port_event: dev_debug() returned error, errno = 22, rc = -1

N5KSW1# sh hardware internal fc-mac 1 port ?

<1-48> 1 based port number

N5KSW1# sh int brie

-------------------------------------------------------------------------------
Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port
                  Mode   Trunk                          Mode  Speed  Channel
                         Mode                                 (Gbps)
-------------------------------------------------------------------------------
fc2/1      790    auto   off     up               swl    F       4    --
fc2/2      790    auto   off     notConnected     swl    --           --
fc2/3      790    auto   on      up               swl    F       8    --
fc2/4      790    auto   on      up               swl    F       8    --
fc2/5      1      auto   on      sfpAbsent        --     --           --
fc2/6      1      auto   on      sfpAbsent        --     --           --
fc2/7      1      auto   on      sfpAbsent        --     --           --
fc2/8      1      auto   on      sfpAbsent        --     --           --
fc2/9      1      auto   on      sfpAbsent        --     --           --
fc2/10     1      auto   on      sfpAbsent        --     --           --
fc2/11     1      auto   on      sfpAbsent        --     --           --
fc2/12     1      auto   on      sfpAbsent        --     --           --
fc2/13     1      auto   on      sfpAbsent        --     --           --
fc2/14     1      auto   on      sfpAbsent        --     --           --
fc2/15     1      auto   on      sfpAbsent        --     --           --
fc2/16     1      auto   on      sfpAbsent        --     --           --

--------------------------------------------------------------------------------
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Port
Interface                                                                    Ch #
--------------------------------------------------------------------------------
Eth1/1        701     eth  trunk  up      none                        10G(D) 1
Eth1/2        701     eth  trunk  up      none                        10G(D) 3
Eth1/3        701     eth  trunk  up      none                       1000(D) 2
Eth1/4        701     eth  trunk  up      none                       1000(D) 2
Eth1/5        1       eth  access down    SFP not inserted            10G(D) --
Eth1/6        1       eth  access down    SFP not inserted            10G(D) --
Eth1/7        1       eth  trunk  up      none                        10G(D) --
Eth1/8        1       eth  access down    SFP not inserted            10G(D) --
Eth1/9        1       eth  trunk  up      none                        10G(D) --
Eth1/10       1       eth  trunk  up      none                        10G(D) --
Eth1/11       1       eth  access down    SFP not inserted            10G(D) --
Eth1/12       1       eth  access down    SFP not inserted            10G(D) --
Eth1/13       1       eth  access down    SFP not inserted            10G(D) --
Eth1/14       1       eth  access down    SFP not inserted            10G(D) --
Eth1/15       701     eth  trunk  up      none                        10G(D) 1
Eth1/16       1       eth  access down    Link not connected          10G(D) --
Eth1/17       1       eth  access down    SFP not inserted            10G(D) --
Eth1/18       1       eth  access down    SFP not inserted            10G(D) --
Eth1/19       1       eth  access down    SFP not inserted            10G(D) --
Eth1/20       1       eth  access down    SFP not inserted            10G(D) --
Eth1/21       1       eth  access down    SFP not inserted            10G(D) --
Eth1/22       1       eth  access down    SFP not inserted            10G(D) --
Eth1/23       1       eth  access down    SFP not inserted            10G(D) --
Eth1/24       1       eth  access down    SFP not inserted            10G(D) --
Eth1/25       1       eth  access down    SFP not inserted            10G(D) --
Eth1/26       1       eth  access down    SFP not inserted            10G(D) --
Eth1/27       1       eth  access down    SFP not inserted            10G(D) --
Eth1/28       1       eth  access down    SFP not inserted            10G(D) --
Eth1/29       1       eth  access down    SFP not inserted            10G(D) --
Eth1/30       1       eth  access down    SFP not inserted            10G(D) --
Eth1/31       1       eth  trunk  down    Link not connected          10G(D) --
Eth1/32       1       eth  trunk  up      none                        10G(D) --

--------------------------------------------------------------------------------
Port-channel VLAN    Type Mode   Status  Reason                    Speed   Protocol
Interface                                                                 
--------------------------------------------------------------------------------
Po1          701     eth  trunk  up      none                       a-10G(D)  lacp
Po2          701     eth  trunk  up      none                      a-1000(D)  lacp
Po3          701     eth  trunk  up      none                       a-10G(D)  lacp

--------------------------------------------------------------------------------
Port   VRF          Status IP Address                              Speed    MTU
--------------------------------------------------------------------------------
mgmt0  --           up     192.168.124.140                         1000     1500

-------------------------------------------------------------------------------
Interface  Vsan   Admin  Admin   Status      Bind                 Oper    Oper
                  Mode   Trunk               Info                 Mode    Speed
                         Mode                                            (Gbps)
-------------------------------------------------------------------------------
vfc1       1      F     on     trunking    port-channel1            TF   auto
vfc9       790    F     on     trunking    Ethernet1/9              TF   auto
vfc10      790    F     on     trunking    Ethernet1/10             TF   auto
vfc131     790    F     on     errDisabled Ethernet1/31             --        
vfc132     790    F     on     trunking    Ethernet1/32             TF   auto
vfc790     790    F     on     trunking    Ethernet1/7              TF   auto

N5KSW1# sh hardware internal fc-mac 2 port 1 port-event
MMDDYY HHMMSS usecs          Event                           Current State
--------------------         -----                           -------------
-                            -                               IPS_LINK_UP      
082813 111939 841857 (0000)  E_IPS_LINK_INIT_SUC             IPS_LINK_INIT    
082813 111939 841843 (0001)  E_IPS_IDLE                      IPS_LINK_INIT    
082813 111939 838663 (0000)  E_IPS_LR                        IPS_LINK_INIT    
082813 111939 832533 (0001)  E_IPS_TMO                       IPS_LINK_INIT    
082813 111939 822555 (0000)  E_IPS_LINK_INIT                 IPS_SPEED 

[Iker Santamaria Molla]

Hello Chald,

first of all, thank you for your interest and your response, but you are wrong. The Nexus 5596-UP have 48 ports in module 1, and 3 expansion slots for modules 2, 3 and 4.

The problem we have is that fc1/4x ports exist, and the error occurs with all the ports in module 1.

Moreover, we have 7 Nexus 5596 more in our enviroment, wit same configuration, and don't have these events.

Could it be a bug? Maybe, but I'm worried because "carmel" process has a consumption about 50% of the CPU.

Any idea is appreciated.

Thank you very much!

Regards.

*******************************************************************************************

sw-cpd-mm-n5k-2# show process cpu sort

PID    Runtime(ms)  Invoked   uSecs  1Sec    Process

-----  -----------  --------  -----  ------  -----------

3030    817808748  412111137   1984   46.3%  carmelusd

switch-n5k-2# show hardware internal fc-mac 1  port 36 statistics

gat_fcp_cli_port_stats: dev_debug() returned error, errno = 22, rc = -1

switch-mm-n5k-2# show hardware internal fc-mac 1  port 37 statistics

gat_fcp_cli_port_stats: dev_debug() returned error, errno = 22, rc = -1

switch-n5k-2# sh hardware internal fc-mac 1 port 43 port-event

gat_fcp_cli_port_port_event: dev_debug() returned error, errno = 22, rc = -1

switch-n5k-2# sh hardware internal fc-mac 1 port 44 link-event

gat_fcp_cli_port_link_event: dev_debug() returned error, errno = 22, rc = -1

[chald · Accepted solution]

Hello Iker,

Yes you are right the 96 have 48 fixed ports.

But in my 5548 it is 32 and i have the same error if i make the show command on a non exsiting port.

as shown up in my first reply.

If you have 7 identical N5596UP and you do not have the problem on others i would investergaate that.

Do you run the same NX-OS on them and are tey configured right?

If that is the case i would open a case with TAC I would also check that the NX-OS you are running has not been revoked.

My setup is a testlab so not en real production by my carmelusd task takes less than 1 % and the highes take about 6% CPU

sh processes cpu sort | include carme

3147     86117625  116934966    736    0.9%  carmelusd

Best regards,

Christian Hald

[Iker Santamaria Molla]

Hello Chald,

thank you very much for all.

Finally, we'll have to open  a TAC case.

Best Regards!

[Iker Santamaria Molla]

After opening a case with the Cisco TAC, they said us that we were affected by the following bug : CSCu27720.

https://tools.cisco.com/bugsearch/bug/CSCuj27720

The workaround to solve the problem was to reload the Nexus 5596UP switch.

[fmancebo · Accepted solution]

We are affected by this bug CSCu27720 (https://tools.cisco.com/bugsearch/bug/CSCuj27720) using NX-OS version 5.1(3)N2(1c).

Thanks for the info.

Best Regards.