UCS Leap Second 2015

[Keny Perez]

I put this in the blog section already,but I want to be sure you can see it in case you are not familiar with the Community directory/layout:

Hello,

I just want make everyone aware of this bug about the leap second:

https://tools.cisco.com/bugsearch/bug/CSCus83447/?reffering_site=dumpcr

Symptom:
UCS Fabric interconnect reload or switchover may occur due to Leap second update.

Conditions:
UCS Version 2.2(x).
This problem does not occur on 2.1 or before.

Workaround:
Disable NTP at least one day (24 hrs) prior to the event. NTP servers typically send the information concerning the upcoming leap second up to a full day in advance.
After the occurrence of the leap second you can safely re-enable NTP.

Further Problem Description:
There is a known Linux Kernel caveat (discussed in public forum at
http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-
linux-server-crashes-today?answertab=active#tab-top). UCS 2.2.x version runs the affected version of the Kernel.

-Kenny

[Walter Dey · Accepted solution]

Thanks Kenny

Just fyi: A leap second will be added on June 30, 2015 23:59:60 UTC.

http://www.timeanddate.com/time/leapseconds.html

Walter.

[Keny Perez]

I was waiting for at least a reply to mark it as answered.   :)

-Kenny

[Walter Dey · Accepted solution]

Table 7 Resolved Caveats in Release 2.2(3e)

Defect ID	Description	First Bundle Affected	Resolved in Release
CSCus69458	The heap-based buffer overflow vulnerability in the GNU C library, documented in Common Vulnerability and Exposures (CVE) CVE-2015-0235 is addressed.	1.0(1e)	2.2(3e)A
CSCus83447	Cisco UCS Fabric Interconnect reload or switchover due to a leap second update no longer occurs.	2.2(1b)A	2.2(3e)A

[Keny Perez]

yeap :) So it is disable NTP with enough time or upgrade before June 30th, 2015.

-Kenny
 

[chald]

Yes the CSCus83447 has been updates 3 times the last 14 days now back again to resolved status with bugfix status 2.2(3e)...

So my 50 cents goes to that diasble or change ntp to an unknown IP and then reenable it again after the leap second.

Is there a estimate of how big a chance of a siwtch over or kernel panic a linux whould have because configs are replicated and running same software. So if a kernel panic is comming then would this not happen again on both Fabric Interconnects. What is that risk procentage ?

-Christian

[Keny Perez]

I did not get the last part of your post and in regards to the percentage, I am sorry, I dont have those numbers.

-Kenny

[chald]

Hello Keny,

Thanks for quick responce.

The thing i tried to aske, in the last part was a theoretic question of what if both Interconnects fail then a failover is not enough.

if you have a 90% chance of one InterConnect halt or reboot. then there would be 80% chance or risk of having both down at the same time.

if both are in SYNC NTP then there is nothing to failover to.

The very last part i asked if there would be a more easy way to cut off NTP trafic (port 123) in a FW and disable NTP master in local LAN/DC.

And then reenable it again the 1 of July.

This is less of a config job and i think the result would be the same.

-CH

[Keny Perez]

Trying to bring this back to life as we get closer to the date... Please remember as a workaround, this needs to be disable 25hrs prior to the already specified time (UTC time!!!!!)

-Kenny

[Paul Jose]

thanks for the prompt response Kenny. I did check that link prior to my inquiry on the forum. Our code isnt't listed on the affected NOR the fixed list :-) The highest code on the affected list was 2.2(3c). The fixed list starts at 2.2(3e)... we are right in between. I didnt want to assume we are safe...

[Keny Perez]

yeah, the fix is 2.2(3e) so running 3d means it will not have the fix

-Kenny

[Paul Jose]

We have 2.2(3d) will that be affected? 

[Keny Perez]

Please check:

https://tools.cisco.com/bugsearch/bug/CSCus83447/?reffering_site=dumpcr <<< Bug on UCS to discuss this issue... See fixed versions at the bottom.

For more info, see:

http://www.cisco.com/web/about/doing_business/leap-second.html#~Overview,  <<< Cisco's official word

https://access.redhat.com/articles/15145  << Red Hat explanation

-Kenny