UCCE 7.5.8 - HDS Distributor problem

[Nguyen Quang Son]

Hi guys,

Can someone please tell me to which problem these messages correspond?

We're using ICM 7.5.8 with HDS+AW installed in the same machine

"Error","49422","29/03/2011 14:46:55","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","Process uaw on ICM\instance\Distributor went down for unknown reason. Exit code 0x1. It will be automatically restarted."
"Warning","49423","29/03/2011 14:46:55","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","Process uaw on ICM\instance\Distributor is down after running for 305 seconds. It will restart after delaying 1 seconds for related operations to complete."
"Warning","49418","29/03/2011 14:46:56","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","ICM\instance\Distributor node restarting process uaw after having delayed restart for 1 seconds."
"Warning","49424","29/03/2011 14:46:56","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","ICM\instance\Distributor node process uaw successfully reinitialized after restart."

Same thing in the HDS2

"Error","49422","29/03/2011 14:46:55","Cisco Systems, Inc. ICM","SRVPARUCCEHDS2","Node Manager","N/A","Process uaw on ICM\socge\Distributor went down for unknown reason. Exit code 0x1. It will be automatically restarted."
"Warning","49423","29/03/2011 14:46:55","Cisco Systems, Inc. ICM","SRVPARUCCEHDS2","Node Manager","N/A","Process uaw on ICM\socge\Distributor is down after running for 305 seconds. It will restart after delaying 1 seconds for related operations to complete."
"Warning","49418","29/03/2011 14:46:56","Cisco Systems, Inc. ICM","SRVPARUCCEHDS2","Node Manager","N/A","ICM\socge\Distributor node restarting process uaw after having delayed restart for 1 seconds."
"Warning","49424","29/03/2011 14:46:56","Cisco Systems, Inc. ICM","SRVPARUCCEHDS2","Node Manager","N/A","ICM\socge\Distributor node process uaw successfully reinitialized after restart."

Thes kind of message can be seen mostly in the busy hours.

I also check the Jaguar log for the number of simultenous open sessions thinking that might be the root cause but found nothing. There are only 2 or 3 sessions opened.

Thanks you very much, any suggestion is welcome.

Best Regards,

[geoff]

updateaw is responsible for keeping the local AW DB up to date as configuration is changed through to the Router and Logger DB. Then change notifications are distributed to all registered AWs, who then update the local AW DB.

Does this occur after a change has been made to the config?

Regards,

Geoff

[jessepbeatty · Accepted solution]

You may need to apply ES15 for 7.5(8). Does the updateaw process close and re-open when you make config changes?

See this field notice:

http://www.cisco.com/en/US/ts/fn/633/fn63371.html

[Nguyen Quang Son]

Hi Jessep, Geoff

Thank you for your 5 cents.

The Distributor updateaw process does restart after making a change but the thing is it seems restart in other case as well and I have to say, quite offten in the peak hours.

Btw, config are updated correctly (so I guess I didn't hit the Field Note), until now, thoses errors have no visible impact to the production.

Do the update process have to process anything if there is no change to the config at all?

Best Regards,

[geoff]

Do the update process have to process anything if there is no change to the config at all?

I would say "no".

it seems restart in other case as well and I have to say, quite offten in the peak hours.

Pointing to network issues. Somehow the real-time feed from the Central Controller to the Distributor is being interrupted.

Regards,

Geoff

[spaulin]

Apply the ES mentioned in the field notice, to all your AWs.

Other than the defect mentioned in the field notice, UpdateAW is VERY stable.

In over fifteen years supporting and training on this product, this defect (a side-effect of a Microsoft change, btw) is the only one I'm aware of for updateaw.  It is highly unlikely you're hitting some other, as-yet-unknown problem.

As a side note, Node Manager logs (or excerpts from Windows Eventvwr) are not particularly helpful.  You'd want to share the actual updateaw (uaw) process log, showing the events leading up to it restarting.

[Nguyen Quang Son]

Hi all, thank you for your response.

I really dont think this could be a network related problem since

- The Rogger and HDS-AW reside in the same site, connected to a same switch.

- The problem can be seen from both sites. There is unlikely the same error for two separated switches at the same time.

As recommended by Spaulin, I went ahead and do a check at uaw process log. Now I'm thinking of a database problem, I'm not an expert of SQL and what the h*** is "Central log was truncated. Must initialize local database" ? (T___T)

Here attached the aw log file and Windows eventvwr correspondent.

"Warning","49423","31/03/2011 11:10:29","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","Process uaw on ICM\socge\Distributor is down after running for 57 seconds. It will restart after delaying 1 seconds for related operations to complete."
"Error","49422","31/03/2011 11:10:29","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","Process uaw on ICM\socge\Distributor went down for unknown reason. Exit code 0x1. It will be automatically restarted."
"Warning","49424","31/03/2011 11:09:32","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","ICM\socge\Distributor node process uaw successfully reinitialized after restart."
"Warning","49418","31/03/2011 11:09:32","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","ICM\socge\Distributor node restarting process uaw after having delayed restart for 1 seconds."
"Warning","49423","31/03/2011 11:09:31","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","Process uaw on ICM\socge\Distributor is down after running for 74 seconds. It will restart after delaying 1 seconds for related operations to complete."
"Error","49422","31/03/2011 11:09:31","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","Process uaw on ICM\socge\Distributor went down for unknown reason. Exit code 0x1. It will be automatically restarted."
"Warning","49424","31/03/2011 11:08:17","Cisco Systems, Inc. ICM","HDS1","Node Manager","N/A","ICM\socge\Distributor node process uaw successfully reinitialized after restart."

Thanks.

Best Regards,

[geoff]

That trace is exactly what we saw when the problem mentioned in the Field Notice started occurring. You need that Engineering Special so that the Recovery Key calculations can be done correctly. Otherwise, it keeps thinking the AW DB is out of whack and keeps rebuilding it.

Regards,

Geoff

[spaulin]

You ask  what is "Central log was truncated. Must initialize local database"?

that's the signature of the defect!

Apply the ES, and your problems will go away.

[Nguyen Quang Son]

Cool, all I need is that confirmation.

Btw, I only need to install the ICM 7.5(8)ES15 on the two HDS-AW, correct ?

Is reboot required ?

Thank you all for your help

Best Regards,

[jessepbeatty · Accepted solution]

Correct. It should be installed on any machine that has the updateaw process.

No reboot, but you do need to stop the services to run the installer. There's a release notes document on the ES15 download page, but here are the steps. It is a very quick install.

• Stop the ICM Distributor node using ICM Service Control.
• Launch the Installer provided for ES15 and follow the instructions on the screen.
• Start the Distributor process from ICM Service Control.

[Nguyen Quang Son]

Cool, thank you very much Jessep