So this is a a good question

eefranzen · ‎08-04-2015

I'm seeing a lot of these errors in the UCS Manager Event Logs. I tried to search for E4196386, but can't seem to find a reference to it.

Anyone know if these are normal events or should I be worried? The Netapp guys are seeing a lot of flogi's.

Code E4196386 Adapter 11/8/1 restarted. The Chassis and Blade number changes in all the entries, so not just one combination having issues, seems like it goes through and restarts all of them.

Firmware 2.2(3c)

Ernie

Keny Perez · ‎08-11-2015

So this is a a good question with very generic information, so let me ask you some other questions so we can start to answer to you.

-What blade(s) model?

-What is the adapter(s) model?

-What OS and what (storage) driver version is it running?

-Is this FC, FCoE, a direct attached NetApp?

This will help us get started.

-Kenny

jselover16 · ‎08-24-2015

I'm seeing the same thing as well...

Code: 2.2 (3g)

Blade Model

Cisco UCS B22 M3
UCSB-B22-M3

Adapter Model

Cisco UCS VIC 1240
UCSB-MLOM-40G-01

OS and Storage driver version

VMware ESXi 5.1
scsi-fnic 1.6.0.12b-10EM.500.0.0.472560
scsi-qla2xxx 911.k1.1-260EM.500.0.0.472560

FC with an EMC VNX 5300

Thanks,
Jeff

Keny Perez · ‎08-25-2015

*Can you get me the enic driver?

*Can you go to the inventory tab> CPU and tell me if the CPU shows as version 2 (v2)?

*Please paste the output of the commad "vmware -vl"

*Get the following commands from the server:

#show service-profile circuit server x/y <<<<x=chassis y=server

(nxos)# show log log | last 40

(nxos)# show log nvram | last 40

-Kenny

jselover16 · ‎08-25-2015

Here is the requested information

enic driver

net-enic 2.1.2.62-10EM.500.0.0.472560

CPU shows as version 2?

No, picture attached.

Output of "vmware -vl"

VMware ESXi 5.1.0 build-2583090
VMware ESXi 5.1.0 Update 3

Output of show service-profile command

DEV-801-UCS-A# show service-profile circuit server 2/2
Service Profile: DEV_vCloud/desx0836
Server: 2/2
    Fabric ID: A
        Path ID: 1
        VIF        vNIC            Link State  Oper State Prot State    Prot Role   Admin Pin  Oper Pin   Transport
        ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------
               889 vmnic0-mgt-a    Up          Active     No Protection Unprotected 0/0        0/6        Ether
               890 vmnic1-vmo-a    Up          Active     No Protection Unprotected 0/0        0/6        Ether
               891 vmnic2-ft-a     Up          Active     No Protection Unprotected 0/0        0/6        Ether
               892 vmnic3-vm-a     Up          Active     No Protection Unprotected 0/0        0/6        Ether
               897 vmhba1-a        Up          Active     No Protection Unprotected 0/0        0/20       Fc
              9089                 Up          Active     No Protection Unprotected 0/0        0/0        Ether
    Fabric ID: B
        Path ID: 1
        VIF        vNIC            Link State  Oper State Prot State    Prot Role   Admin Pin  Oper Pin   Transport
        ---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------
               893 vmnic4-mgt-b    Up          Active     No Protection Unprotected 0/0        0/7        Ether
               894 vmnic5-vmo-b    Up          Active     No Protection Unprotected 0/0        0/7        Ether
               895 vmnic6-ft-b     Up          Active     No Protection Unprotected 0/0        0/7        Ether
               896 vmnic7-vm-b     Up          Active     No Protection Unprotected 0/0        0/7        Ether
               898 vmhba2-b        Up          Active     No Protection Unprotected 0/0        0/40       Fc
              9090                 Up          Active     No Protection Unprotected 0/0        0/0        Ether

Output of log and nvram logs

Attached as well

eefranzen · ‎08-25-2015

I had more today at 12:57. I ran the show logs, but neither the log nor nvram shows anything going on at that time.

eNIC - 2.1.2.59

CPU E5-2697-v2

Blades are running RHEL

Ernie

Keny Perez · ‎08-26-2015

jselover16...

Server 2/2 is running the proper drivers. And I see a lot of "VIF DOWN" for server 2/2 on Aug 22nd...

Can you please paste the out of the following commands:

#connect local a

#con iom 2

#show platform software woodside oper

#exit

#connect local b

#con iom 2

#show platform software woodside oper

Thanks in advanced,

-Kenny

jselover16 · ‎08-26-2015

attached is the output of the requested commands

Keny Perez · ‎08-26-2015

thanks, one more output...

#connect local a

#con iom 2

#show platform software woodside sts

-Kenny

jselover16 · ‎08-26-2015

The output is attached

Keny Perez · ‎08-26-2015

So you have 2 blades and 2 cables between IOM and FI... why do we have port 3 connected instead of port 2 (I am talking about the IOM)? With 2 links between FI&IOM your server 2 should be pinned to port2 but is not connected so I do not know how this may behave in that condition (not saying that is the issue, just saying how it is documented)

I see HIFs (backplane ports in the IOM) 25 & 27 have flapped 8 times and the last one was on Aug 22nd and that is the reason why we see the VIFs going down on the 22nd.

Please open a TAC case base on this post and add the logs you provided here or just use this link as reference and have TAC get deep into what is going.. If you dont mind, please give feedback here when you get to the root cause. The information you have now is a very good start to get into what is going on... may be a known bug I am not aware of yet :)

Please rate all helpful answers

-Kenny

jselover16 · ‎08-26-2015

I actually already opened up a TAC case, i just commented on this thread as it was the same issue and i was waiting on my account to be added to the support contract. I have already referenced this post in the ticket. I will update it with the results when I have them.

eefranzen · ‎08-26-2015

I also opened a case and was told to upgrade the firmware and then see if we are still seeing the issue.

Ernie

Keny Perez · ‎08-26-2015

eefranzen,

If you run the same commands I have requested in this thread, do you also see the backplane ports of the IOM flapping for the specific server where the adapter is showing the "Code E4196386 Adapter ##/#/# restarted" message?

-Kenny