Problems after upgrade firmeware 2.0(4a) from 2.0(2r)

[Christopher Federico]

One of my Fiber interconnects become inoperable once a day . The FI's are 6248up . When this happens it looks as though all the ports shutdown then come back online . I checked the firmware and everything looks to be on the same version . Anyone have this issue . Should I roll back to 2.0 (2r)  ? I think I will have to call support on Monday .

Thanks

Chris

[abbharga]

Hi Chris,

How long after the upgrade did you see this issue?

Were you able to verify if the FI rebooted or if there was a process crash or anything which coould have caused these ports to flap?

./Abhinav

[Christopher Federico]

Hi Abhinav ,

I did the upgrade last Thursday night at 11:00 . The first crash was on Friday at 2:40 pm . The second crash was Saturday at 4:43 pm . After the first and second time the primary changes to the secondary like its supposed to do after a crash . The one that is crashing is the primary but after the last vrash I left it the secondary now .

This is the error code :

fltNetworkElementInoperable

Fault Code:F0291

Message

Fabric Interconnect [id] operability: [operability]

Explanation

This fault typically occurs when the fabric interconnect cluster  controller reports that the membership state of the fabric interconnect  is down, indicating that the fabric interconnect is inoperable.

Recommended Action

If you see this fault, take the following actions:

---

Step 1 Verify that both fabric interconnects in the cluster are running the same Kernel and System software versions.

Step 2 Verify that the fabric interconnects software version and the Cisco UCS Manager software versions are the same.

Step 3 If the above actions did not resolve the issue, create a show tech-support file and contact Cisco TAC.

When the fiber interconnect shuts off all the ports shutdown . I wonder when it crashes does it do a gARP so that the second FI takes over ?

I'm not sure what I can do ?

Thanks

Chris

[marcelvery]

Abbhinav,

There is a problem with 2.0(4a) where the flash area can get out of space and gives such problems

More info on:

http://stretch-cloud.info/2012/09/major-bug-hit-cisco-ucs-chassis-beware-before-updating-firmware-2-0-4a/

and

http://stretch-cloud.info/2012/09/cisco-ucs-chassis-iom-bug-engineering-update/

I updated one system from 2.0(3a) tot 2.0(4a) without problem, but am waiting to do a another system

[Eric Rose]

Just a note for clarificaiton, that issue is across 2.0 releases. Here is the information from the releases notes.

http://www.cisco.com/en/US/docs/unified_computing/ucs/release/notes/OL_25363.html#wp332021

The following bugs are common across Release 2.0:

Symptom    Under some conditions, the IOM upgrade fails and gets into a continuous reboot after the IOM is activated by FI. Rebooting the FI on a failed IOM update does not fix this issue.

Workaround   Contact Cisco technical support. (CSCuc15009)

Also note that this will not happen to all upgrades, just a small number.

[abbharga]

Hi Marc,

Greetings!

Yes we are fully aware of this issue, its not entirely a bug but a behaviour which we are tracking currently.

And the reason i asked Marc those questions was to narrow down to the cause of the issue being a FI crash / reboot or a disconnect of the IOM (which could have been due to this issue).

Thanks once again for pointing this out!

./abhinav

[Christopher Federico]

Something interesting ... On Saturday the primary FI crashed and I left it alone as the subordinate . It has not crashed since . I am going to call TAC this morning though but that is interesting .

[abbharga]

Hi Chris,

Apologies for not replying bak to you earlier.

while opening the case, please do the following:

1) collect a UCSM show tech and upload it to the SR,

2) Check for cores -> Admin tab -> faults,Events and Audit logs -> Core Files

In the right work pannel if you see any cores (with the timestamp after the upgrade on either FI) download it and attach it to the case as well.

You can download the core file by selecting it, right click and choosing the option to download it.

./abhinav

[KRIS PATE]

Is this bug fixed in the 2.0(4b) version that is supposed to be released today?

Kris

[Christopher Federico]

Update : Our 2 FI's only had one PSU plugged in and according to the Tech-support file  FI-A is pointing to a power lose. I since plugged in both ports each FI making sure that they are on separate circuits . When the FI seemed to crash it didn't produce a core file .

We will know in a couple of days if its the issue . TAC has been really helpful