FYI: Software Upgrade Service (SUS) Outage

[Paula Talamo]

All,

I have received notification that the Software Upgrade Service (SUS) is down and troubleshooting is in progress. 

Problem Description: Software Upgrade Services (SUS) : getting No Service Contract" status for all SN's passed

Business Impact: Software Upgrade Service (SUS) is not providing correct Service Contract Information to the Cisco TMS product with software version and release keys for TMS managed endpoints.

ETA: 6pm PDT

If I hear that it has been resolved, I will post but be advised this is best effort due to my own availability so check after 6pm PDT just in case.  Apologies for the outage but I wanted to let you know.

Paula

[Martin Koch]

Hello Paula,

thank you for the info and keep us updated!

Martin

[Yanick Sauro]

All,

Here is the latest status:

·         Software Upgrade Services (SUS) is returning incorrect Service Contract Information.

·         The SUS Support Team is able to replicate the issue on STAGE.

·         The team has identified that there are format changes in the responses received from upstream systems

·         The SUS IT Support and EF teams are currently troubleshooting the changes that may be required to restore services on stage

·        The IT teams along with the Business unit have a checkpoint at 12:00 PM PT 6/11 to make for a business decision on how to move forward and discuss the next steps.

If you have any questions please feel free to reach out to me directly.

Yanick Sauro

Situation Manager

TelePresence Solutions Group

yanisaur@cisco.com

Phone: +1 703 707 1331

[Justin Ferello]

Paula,

Thank you for the original notification.

Yanick,

Thank you for keeping us updated.

Thank you,

Justin Ferello
Technical Support Specialist
KBZ, a Cisco Authorized Distributor
http://www.kbz.com
e/v: justin.ferello@kbz.com

[Yanick Sauro]

All,

Here is the latest status:

Current Status :   In Progress

·         Software Upgrade Services (SUS) is returning incorrect Service Contract Information

·         SUS support identified that there are changes in the response XMLs received by SUS from upstream(EF) which is  missing the service contract information

·         Team has identified this is related to a changes made at Entitlement Framework(EF) as part of the June monthly release

*** NEW ***  EF & DISP teams are currently developing the required code changes to pass the SN information as part of the response to SUS services to restore services for Tandberg. The IT teams and Business have an internal checkpoint at 2:00 PM PT 6/11 to review the status.

Next ETA is 03:30 PM PT

[Jens Didriksen]

This has actually gone on for a few weeks now

I opened a TAC case on May 29th as I've then had incorrect status reported for approx 2 -3 weeks.

It was resolved June 7th with all my systems showing correct status, only to re-surface appprox 36 hours ago.

Good to see notificiations re this finally being posted in this forum though.

/jens

[Yanick Sauro]

Hi Jens,

Totally agree with you, I saw the exact same pattern globally. Most of the issues were resolved mid-last week and than suddently everyone started having issues on Monday.

I just completed a call with Cisco IT and we are close to a resolution, they know what caused the issue and are testing a fix in a lab environment, if they get the green light the fix will be implemented tonight and tomorrow should be a better day for everyone!

For everyone benefit, we are currently focussing on the issue at hands but we are also working closely on multiple fronts with Cisco It to solidify the SUS service as we understand how big of an impact it can have when it goes down or doesn't report the right information.

I will be updating this post later on tonight,

Yanick

[Yanick Sauro]

Latest Update:

·         EF & DISP teams have made the required  code changes on stage(test environment) to pass the SN information as  part of the response to SUS services to restore services for Tandberg.   The SUS QA team is currently testing the changes. Next internal check  point is scheduled at 4:00 PM PT

Yanick

[Yanick Sauro]

Resolved - On Watch

The changes are deployed as part of an emergency bug fix at 6:30 PM PT 6/11/2013, the application teams have validated the changes and confirmed the correct responses XML is being passes from Entitlement framework(EF)

·         The SUS support team continues to monitor the requests for stability

To our Customers / Partners: Please validate that your units are now reporting the correct information in TMS, you might need to perform a force refresh since by default TMS only updates himself every 48 hours.

If you are still experiencing the issue please escalate to your Cisco CSE referring your existing cases or by opening a new service request.

Thank you,

Yanick

[david]

We were hit by this issue but this is now resolved this morning.

[Yanick Sauro]

Thank you David for the feedback it's appreciated.

Yanick