Solved: Configuration Manager not able to start it because of failing in Distributor

bilalghayad · ‎02-15-2011

Hi All;

I am doing a refresh technology, the old version is ucce 7.2 and the new version is 7.5 (but the new hardware in a new site, while new site has WAN connectoin with production site, so the final target is to have cluster over WAN for the IPCC).

The domain controller is replicated with the production.

RGR is installed and side_A database retored and migrated using the EDMT.

HDS database installed and migrated using the EDMT.

When I did the installation of the AW, I selected real-time and HDS during the installation, IP addresses were given.

Now, when starting the services at the AW machine (which contains also the HDS now), the distributor is not starting successfully but I see the HDS connected with the RGR. So the distributor is not able to do a feed from the RGR.

Distributor updateaw displays:

Real-time feed not active - sleeping for 5 seconds before retry..

And Distributor consol keep doing shutdown (it is like doing restarting ...).

Now, when I try to start the configuration manager, it displays the following:

(-9934) Unable to initialize the real-time feed for instance stc. Unable to proceed until the real-time can be established. The feed may be inactive or you may not have permission to connect to feed.

What could be the reason for this?

Really I was afraid that it is wrong to have a Domain Controller replicated with the production, because I am doing installation for new hardware to be side A for the UCCE, while at the same time there is side A and side B in the production. Does this effect?

While doing trouble shooting, I faced something strange in the registry, that the IP address of the central controller is the IP address of side B of the production, and the SQL database is version 6 (while in the new hardware, I used SQL 2005). What is the reason for this?

I am attaching some snap shot that could help.

Appreciate really the help and I openned TAC in it with Cisco, but the TAC engineer said let me think in it

Regards

Bilal

geoff · ‎02-18-2011

One more thing that is important -- you are building a duplexed Router, but you said you were only working on side A.

If you don't bring up side B, the Router on side A will only go into service (simplexed) if it can see 50% or more of the configured PGs running. So if you start side A Logger and Router, and don't start some PGs, the Router will be waiting for its peer.

Is the Router really in service? Can you show me a screen shot of the rtr process on side A that shows that the config was loaded and the message "All configuration operations complete" is on the screen?

Regards,

Geoff

View solution in original post

geoff · ‎02-18-2011

Well, I have a doubt in the Service Account Manager:

You should not have any doubts - it works correctly.

In my opinion, you seem to take many opportunities to play with and adjust settings that you should just leave set to the defaults, or for the system to create.

If the installer has the Domain Manager connection to the AD, it will create all the accounts it needs automatically, stored under the Cisco_ICM Organizational Unit. Why do you care what they are?

The only thing you should have done correctly is the installation of SQL.

Here, you should have created a user on the domain for SQL and SQL Agent to run under, and you need to go into Local Users and tweak certain settings, as described in the staging guide. Then when installing SQL, give it this Domain user to run both SQL and SQL Agent under - as described in the staging guide. It will work otherwise - as the Domain Admin, but the above is best practices.

I always create SQL with mixed authentication and set the SQ password. ICM does not use this - it uses named pipes under Windows Authentication, but that's not an issue.

Please just follow the basic install, take the defaults at every turn. If you need to, you can go back and change things.

Regards,

Geoff

View solution in original post

geoff · ‎02-19-2011

I can't see the attachments - it says "QUEUED".

When you start Logger A and then Router A in a duplex environment, the Router will not start until it can talk to Router B; if you don't start Router B, it will wait until it can talk to 50% or more of the configured PGs. Are you doing that?

MSSQL, I am using local system login (because really using the this account login via domain, it was not start automatically so I have to start the SQL service manually, but using local system login, it start well), but this should not effect.

No, in a way it does not matter what login account the SQL Service runs under. But a Domain account with specific Local User tweaks, as specified in the Cisco Staging Guide, is the way to do it.

Regards,

Geoff

View solution in original post

geoff · ‎02-19-2011

But if we are talking about 50% or more of the configured PGs, then I have maybe to re run the setup and select PG to 1 and then I have to switch on the CM PG (which I installed), so the Router A will be able to talk with the CM PG, and here I can satisfy the 50%, which means the Router A will work, am correct?

Yes. When doing a big site upgrade (say 10-30 PGs), we usually change the config in the registry when the Router comes up. Just start with 1, then upgrade number 2, and tell the Router about it. It's dynamic. Don't do it with setup - but you do need to know the two registry keys to change for each PG.

Regards,

Geoff

View solution in original post

geoff · ‎02-19-2011

Again, it is queued so I am inseting them here:

Exactly - the Router is the one to watch. It says "synchronization holdoff disabled" and you need to bring up that PG and you will see it load the config from the Logger into memory. Then wait until it says "all config operations complete". Now it is up and you can start the Distributor and get the real time feed.

Regards,

Geoff

View solution in original post

geoff · ‎02-21-2011

By the way, appreciate if u can advise me that from where you got to know this information that 50% of PGs should communicate with the Router to be able to start work?

To be specific - for a duplexed Call Router to start without its peer:

(a) for the side A Call Router, it must be able to see >= 50% of the number of PGs configured in the registry.

(b) for the side B Call Router, it must be able to see > 50% of the number of PGs configured in the registry.

Note the difference.

Regards,

Geoff

View solution in original post

geoff · ‎02-15-2011

Remember that the HDS and the AW are two different things.

The HDS connects to the Logger (in your case, LoggerA which is running on RGR A) through the replication processes. This is the way historical data is replicated to the HDS from the Logger. You say that this is working.

The real time feed is from the Distributor to the Router - in your case, RouterA which is running on RGR A.

Are you in the middle of a migration where you have side A RGR (Logger and Router) up, but side B is down?

Normally, with both side A and side B Routers running, the Distributor is able to establish the real-time feed to either of these. You may have configured both the side A and side B Router and the side A and side B Logger in the "Real-time Distributor Properties" window, but if side B is not up, try changing this to just have the side A machines and set the Central Controller preferred side to side A. When you have side B RGR OK, you can come back and add the side Bs.

Is the AW you are trying to get going colocated with RGR A?

Regards,

Geoff

bilalghayad · ‎02-16-2011

Remember that the HDS and the AW are two different things.

Yes, I know this.

The HDS connects to the Logger (in your case, LoggerA which is running on RGR A) through the replication processes. This is the way historical data is replicated to the HDS from the Logger. You say that this is working.

As u see in the attached pic, it is working. Because this is shown in the LoggerA replication process.

And yes, I have logger and router on one machine, so I have RGR.

And it is side A (RGR A).

The real time feed is from the Distributor to the Router - in your case, RouterA which is running on RGR A.

That is correct, but which process in the RGR that can show me if the Distributor succeeded or not?

Are you in the middle of a migration where you have side A RGR (Logger and Router) up, but side B is down?

Yes, I am now in the middle. But as I am using Refresh Technology, so the production (Side A and Side B)

is running, while I am now working on new separated hardware, and I am now working on side A of that new

hardware.

From the other side, we are using a domain controller with is replicated with the production.

I feel my problem could be in the Distributor permission to feed from the RGR? Could be? But now able

how to check this.

One important information, that the registry is containing values for side B of the production, that

make me afraid if the Domain Controller provided these values? I agree with u that in the beginning I gave

the IP addresses of side B of the production, but I repeat the setup and put a fake IP address for side B

to avoid it from communicating with side B production.

Yes I selected that central controller A is the preferred side.

Normally, with both side A and side B Routers running, the Distributor is able to establish the real-time feed to either of these.

Could it be related to the permissions that Distibutor needs it to be able to feed from the RGR?

You may have configured both the side A and side B Router and the side A and side B Logger in the "Real-time Distributor Properties" window, but if side B is not up, try changing this to just have the side A machines and set the Central Controller preferred side to side A. When you have side B RGR OK, you can come back and add the side Bs.

I agree with u about this, but to be sure we are talking about same thing: Do u mean in the Real-time

Distributor Properties window, is that in the setup? Or something else?

If u mean by this window in the setup, yes this is what I did (although in the beginning I placed the IP

addresses of side B of the production, but then I come back and I placed a fake IP addresses for side B.

Although I did this, but still I see in the Registry the IP addresses of side B production !! Why?

From where it came these IPs?

Is it also wrong to place a fake IP addresses for side B, well, what is the recommended in that case?

Is the AW you are trying to get going colocated with RGR A?

No, AW colocated with HDS on the same hardware, while RGR A in another hardware.

Really I am worry about the Distributor permissions, and I am worry why in the registry I see the IP

addresses of side B production? I am attaching the registry that I am talking about it.

Any advise?

Regards,

Geoff

geoff · ‎02-16-2011

bilalghayad,

You should use the quote icon so that we can easily see what I wrote and what you wrote.

Is the AW you are trying to get going colocated with RGR A?

I meant at the same premise, not the same machine.

Regards,

Geoff

bilalghayad · ‎02-16-2011

Yes they are in the same premise.

Well, I found a link that talk about the blocking of the port because of the partition:

http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a008010f845.shtml

But really I did not understand how to resolve my problem.

Well, in that link, they mention that the used port is 40047, but when I ran netstat -a at the RGR A, I did not see this port at all.

About my database: I place the data at drive "D" and the log file at drive "C", the tempdb is already at drive "C", any other thing is needed to have the partition correct?

At the distributor (the AW machine), I selected the partition, and I placed the HDS database at drive "D" and its log file at drive "C". Any thing else I have to do?

About the permissions: The same user that I use to login for the machine and to the Domain Controller, I added as a member for the Cisco root, instance, ... etc (via the Domain Manager), am correct or I am missing something in the security?

Actually on my distributor, it is using other port than 40047 to connect with the logger (because I am not on the server now, I think it was 40001), actually both of those ports are not possible to do a telnet for them at the logger or from the AW to the logger?

What do u think?

Highly appreciated ur help.

Regards

Bilal

geoff · ‎02-16-2011

How about a reinstall? Reinstalling an AW doesn't take long.

Back up the HDS, use ICMDBA to remove the HDS database. Remove alll the ICM software and clean up the file system and registry.

Install ICM again and JUST install a Distributor AW (don't initially make it an HDS). Check that you get the real time feed.

Now run setup again and convert the AW to an AW/HDS. Restore the DB from the backup and check that replication is working.

Regards,

Geoff

bilalghayad · ‎02-16-2011

Actually I did what u said (and yes, I started by Dributor without HDS), but did not remove the HDS database. Maybe I need to retry this again but with removing HDS and reinstall as u said.

But, what do u think about the link I sent for u and the partitioning?

If to talk about partitioning: how to enable it at the logger and AW? I put the data files in the drive "D" and the log file in the drive "C", what else need to be done?

I am thinking in another way: If I am not able to telnet ip_address port of the logger from the Distributor machine, then I beleive there is a problem at the logger, is it possible? What is the port that the Distributor is using it to communicate with the logger to feed the Data?

About the permission (using Domain Manager): what I did is correct?

Thanks a best regards for ur kindly help and reply.

Regards

Bilal

geoff · ‎02-16-2011

Actually I did what u said (and yes, I started by Dributor without HDS), but did not remove the HDS database. Maybe I need to retry this again but with removing HDS and reinstall as u said.

Doesn't really matter if the HDS DB stays there.

If to talk about partitioning: how to enable it at the logger and AW? I put the data files in the drive "D" and the log file in the drive "C", what else need to be done?

None of this matters.

If I am not able to telnet ip_address port of the logger from the Distributor machine, then I beleive there is a problem at the logger, is it possible? What is the port that the Distributor is using it to communicate with the logger to feed the Data?

You should be able to do that - check the port utilization guide to find the correct port number.

You haven't got the INSTANCE NUMBER wrong, have you? Should be zero but it could have changed to 1.

I'll check in my lab system .

About the permission (using Domain Manager): what I did is correct?

Nothing to worry about.

Regards,

Geoff

geoff · ‎02-16-2011

I can telnet to router A from distributor on 40007. router B on 41007

Regards,

Geoff

bilalghayad · ‎02-16-2011

OK, I have to check the INSTANCE NUMBER, but basically as I remember it is 0.

Now, what if I was not able to telnet (even from the logger it self) to port 40007? That means, there is a block on that port (the process it self is not started well). Here is the question: What could be the possibilities? That is why I thinked in the partitions.

Really, it is stange and not able to know why feeding is not working.

By the way: where I can find a guide for the ports (this snap shot u sent)?

Again, fully thanks for ur help.

Regards

Bilal

geoff · ‎02-16-2011

Download the Port Utilization Guide from Cisco.com

Regards,

Geoff

bilalghayad · ‎02-17-2011

Dear;

It could be related to the following reason:

I migrated the database as following:

I backup it from the production and restore it in the new environment and then I run the EDMT as a common ground. I did this because I do not need to stop the service at the production, so I preferred to do that way.

Maybe I was have to use Refresh Technology and bring the database directly from the production?

Appreciate ur advise.

Regards

Bilal

bilalghayad · ‎02-17-2011

Dear;

I uninstall every thing and removed the DB and now I am installing again the servers.

Regarding to RGR:

After complete the installation and using the icmdba I created the DB sideA (I selected partition=2 and really I do not know if it should be 1 or 3 and what is the difference), then I started the servcies (logger service first and then router service), while doing the installation, I just gave IP addresses for side A and did not give IP addresses for side B.

But after installing the services, I did not see the rtfeet service, and when I run netstat -a, I can not see the port 40007 (the instance number is 0), even after installing the Distibutor and creating the awdb in the admin machine, I am not able to see the port 40007 at the RGR (actually, I do not see the service rtfeed, or it has another name)?

By the way, while doing installation for the RGR, there is a windows screen asking about the Network Interfaces Paramters, really I accept the default (as I beleive this is used for pre routing, correct?) I attached this screen if u can advise plz.

Regarding to partition, currently I see in the loggerA configlogger displaying that "partitioning is enabled".

What could be the reason that the rtfeed is not started at the RGR? I am talking about fresh installation.

Thanks for your kindly advise.

Regards

Bilal

bilalghayad · ‎02-17-2011

Maybe the following could help us:

1) Maybe I have to start logger while I have AWDB, otherwise the rtfeed process will not start because it might be blocked as was not able to connect to AWDB? In other words, if I start the RGR (logger and then RGR) while the AWDB have not been installed, then might cause not to have rtfeed at the router?

Because really when i did the fresh installation, I did not try to restart the logger after installed the AW (Distributor and AWDB)?

Note: I am writing this and I am outside the site ..

2) Any special database settings need to be done on the Registry (username, password for the database)?

3) Referring to the link below:

http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a008010f845.shtml

Do u c in ur Logger Process Log that it is trying to connect to the AWDB?

Do u c in ur Logger Process Log the below:

11:03:31 Trace: Using pipe ICR\dar\Distributor\InputToLogger3516663121 for Input To Logger.
11:03:31 Trace: Using pipe ICR\dar\Distributor\OutputFromLogger3516663121 for Output From Logger.
11:03:31 Trace: Processing Config Messages on behalf of user dbo

All of this to know why I do not have rtfeed at the RGR which cause the Distributor at the AW to fail.

Thanks for ur kindly help.

Regards

Bilal

geoff · ‎02-17-2011

After complete the installation and using the icmdba I created the DB sideA (I selected partition=2 and really I do not know if it should be 1 or 3 and what is the difference)

Why are you enabling Partitioning? Do you know what that's for? I'll give you a hint ... almost no one uses that now, and many big companies that used to use it have gone through the difficult exercise of getting rid of it!

That's your problem.

Regards,

Geoff