Re: ASR9K Series devices inventory is not working.

NikolaiAntonov · ‎04-08-2012

Hi all. Inventory in CiscoWorks with new devices ASR9K Series is not working. CW version: LMS3.2.1. Device: ASR-9006 AC Chassis. Credentials correct. Can any help me?

Screenshot1: inventory request fail.

Screenshot2: RME knows Cisco ASR9006 Router.

Marvin Rhoads · ‎04-08-2012

Your screenshot implies that ssh is failing.

Are you able to ssh to the ASR 9k from your CiscoWorks server using a 3rd party tool like PuTTY?

NikolaiAntonov · ‎04-08-2012

Dear Marvin, thanks for help.

Yes: ssh from server with PuTTY success.

Vinod Arya · ‎04-09-2012

LMS would only need snmp to do the Invenotry. If Inventory is failing please chekc if LMS is able to do snmpwalk to the device or not. You can try to test snmpwalk from LMS server to device via device centre or using snmpwalk.exe from $NMSROOT/Objects/jt/bin/ directory.

CLI eg:

c:\progra~1\CSCOpx\objects\jt\bin\>snmpwalk -v2c -c public 10.104.149.180 sysObjectID

SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.9.1.283

If device is accessible via snmp, invenotry shouls succed. try to increase the snmp timeout as well in LMS from :

RME > Admin > System Preferences RME Device Attributes

Even if it fails you may want to share failing error and IC_Server.log from server.

-Thanks

-Thanks Vinod **Rating Encourages contributors, and its really free. **

NikolaiAntonov · ‎04-10-2012

Dear Vinod, thx for answer.

Snmpwalk works well. But there is an issue with device credentials. CW cannot connect to ASR at all. Sync archive work fails and CDA work fails for ssh and telnet, but reachability tests from CW seems good. I've tried to sniff packets with WS. And there is the issue: at first CW tries telnet and fails three times (It sent right login and password, but there is some kind of failure and it types password in the wrong field)

follow tcp stream in WS gives such output:

"username: username

password:

username: password

password:

"

But there aren't any failed attempts on TACACS+ server log.

Afterwards it tries ssh and fails again(WS screen attached). Logs from device tells such a thing (only for ssh):

SSHD_[65867]: %SECURITY-SSHD-6-INFO_GENERAL : Client ---.---.---.--- closes socket connection

SSHD_[65867]: %SECURITY-SSHD-3-ERR_GENERAL : Failed in version exchange

SECURITY-SSHD-6-INFO_GENERAL : Incoming SSH session rate limit exceeded

Credentials seems to be rigth. Putty connections from CW server via ssh and telnet under CW credentials are successfull. I have changed snmp/telnet/ssh timeouts in different manner but it didn't help.

Vinod Arya · ‎04-11-2012

Usually it is essential to configure the $NMSROOT\objects\cmf\data\TacacsPrompts.ini file. As with tacacs+ Auth you can also define custom login username and password prompt, hence this file is important to be configured, only in case of Telnet not SSH.

So just check the login prompt you get in your device when you try to do telnet and mention the same in you

TacacsPrompts.ini file.

Example:

>Following is the content of TacacsPrompts.ini :

[TELNET]

USERNAME_PROMPT=

PASSWORD_PROMPT=

> Following is the username and pasword propmt for my device :

> As per this please modify the file as :

[TELNET]

USERNAME_PROMPT=Username:

PASSWORD_PROMPT=Password:

**NOTE : File is case Sensitive. be specific as per what you get while logging in.

Also, try to increas the timeout for Telent via RME > Admin > Device attribute. Just save and try to run the job again for failing device.

-Thanks

Vinod

-Thanks Vinod **Rating Encourages contributors, and its really free. **

NikolaiAntonov · ‎04-12-2012

Dear Vinod,

Thanks a lot. Your solution is very helpfull. But i disinformed You. Sorry for that. I'm actually intrested in config archive rather than inventory collection. So config archive works well via telnet, but with ssh it fails. And CDA work for ssh fails.

Is there the same solution with ssh?

With ssh there is some other promts:

login as:

yourlogin@deviceIP's password:

Should i change TacacsPrompts.ini file in the same manner as for telnet?

Or there is some ssh daemon bug on the device?

-Thanks

Nikolay

Vinod Arya · ‎04-12-2012

No SSH does not requires that file. Though you can add these prompts to the file with comma separation and try.

There are some known issues wih LMS with IOX-XR using SSHv2. You may be hitting CSCte95623. There is a patch for LMS 3.2.1/RME4.3.2, available with TAC.

-Thanks

Vinod

-Thanks Vinod **Rating Encourages contributors, and its really free. **

NikolaiAntonov · ‎05-31-2012

Hello again and thx for advice,

I've tried the solution from Cisco for this bug (CSCte95623 ), by manipulating delays values in cmdsvc.properties file and restarting cfgmngmt process. I've changed delay values in very different manner (delay after connect, tunesleepmills, login, e.t.c). Unfortunately this solution didn't help. A CDA work for SSH fails all the time. Also i've manipulated

ssh rate-limit and ssh session-limit values on device. It's a pity that opportunity to set on only sshv1 on device doesn't exist, so CW tries to connect only with sshv2 and there is no chance to check how it work with sshv1.

I'm becoming a bit desperate about that issue. Any ideas?!

There is some output from ssh debugs on device:

debug ssh server

RP/0/RSP1/CPU0:May 31 12:02:14.068 : SSHD_[1114]: Spawned new child process 5869901

RP/0/RSP1/CPU0:May 31 12:02:14.149 : SSHD_[65869]: Client sockfd 3

RP/0/RSP1/CPU0:May 31 12:02:14.151 : SSHD_[65869]: Setting IP_TOS value:192

RP/0/RSP1/CPU0:May 31 12:02:14.152 : SSHD_[65869]: After setting socket options, sndbuf33792, rcvbuf - 33792

RP/0/RSP1/CPU0:May 31 12:02:14.153 : SSHD_[65869]: Connection from ------------ port ---------

RP/0/RSP1/CPU0:May 31 12:02:14.158 : SSHD_[65869]: (addrem_ssh_info_tuple) user:()

RP/0/RSP1/CPU0:May 31 12:02:14.162 : SSHD_[65869]: Session id 0

RP/0/RSP1/CPU0:May 31 12:02:14.162 : SSHD_[65869]: Exchanging versions

RP/0/RSP1/CPU0:May 31 12:02:14.164 : SSHD_[65869]: %SECURITY-SSHD-6-INFO_GENERAL : Client ------ closes socket connection

RP/0/RSP1/CPU0:May 31 12:02:14.164 : SSHD_[65869]: %SECURITY-SSHD-3-ERR_GENERAL : Failed in version exchange

RP/0/RSP1/CPU0:May 31 12:02:14.164 : SSHD_[65869]: In cleanup code, pid:5869901, sig rcvd:0, state:1

RP/0/RSP1/CPU0:May 31 12:02:14.166 : SSHD_[65869]: Cleanup sshd process 5869901, session id 0

RP/0/RSP1/CPU0:May 31 12:02:14.171 : SSHD_[65869]: Closing connection to --------

RP/0/RSP1/CPU0:May 31 12:02:14.171 : SSHD_[65869]: Sending Disconnect msg

RP/0/RSP1/CPU0:May 31 12:02:14.172 : SSHD_[65869]: sshd_shm_acquire_lock: SHM Lock is NULL

RP/0/RSP1/CPU0:May 31 12:02:14.172 : SSHD_[65869]: sshd_shm_unlock: SHM Lock is NULL

RP/0/RSP1/CPU0:May 31 12:02:14.184 : SSHD_[1114]: Signal 18 received in handler: pid 5869901

RP/0/RSP1/CPU0:May 31 12:02:14.207 : SSHD_[1114]: ratelimit_msecs:1000.000000, ratelimit_count:1

RP/0/RSP1/CPU0:May 31 12:02:14.207 : SSHD_[1114]: elapsed:145.976000, ratelimit_msecs:1000.000000, count:1

RP/0/RSP1/CPU0:May 31 12:02:14.207 : SSHD_[1114]: %SECURITY-SSHD-6-INFO_GENERAL : Incoming SSH session rate limit exceeded

And CDA ssh work log from CW:

Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1571,Iam inside ssh ....

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1573,Initial time_out : 0

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1583,Computed time_out : 30

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1599,After computing time_out : 30

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getSshCmdSvc,1637,inside getSshCmdSvc with timeout : 30000

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getSshProtocols,1743,Inside getsshprotocols with time out : 30000

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getSshCmdSvc,1651,SSH2 is running

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,136,Got CmdSvc for SSH

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,141,Before Resetting the counters i.e before invoking counters for CredType :: SSH

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,151,After Resetting the counters i.e before invoking counters for CredType :: SSH

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,156,Getting Primary credentails to reset again to Primary only..

[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,201,trying to connect for SSH

[ Thu May 31 12:10:18 MSD 2012 ],ERROR,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,272,Got CmdSvcException com.cisco.nm.lib.cmdsvc.CmdSvcException: java.net.SocketException: Connection reset

at com.cisco.nm.lib.cmdsvc.OpConnect.invoke(OpConnect.java:57)

at com.cisco.nm.lib.cmdsvc.SessionContext.invoke(SessionContext.java:299)

at com.cisco.nm.lib.cmdsvc.Engine.process(Engine.java:57)

at com.cisco.nm.lib.cmdsvc.LocalProxy.process(LocalProxy.java:22)

at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:190)

at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:166)

at com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler.verify(CmdSvc_CDACredTypeHandler.java:202)

at com.cisco.nm.xms.xdi.pkgs.LibCda.GenericCdaHandler.checkSanity(GenericCdaHandler.java:37)

at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.checkSanity(CdaJobEngine.java:1565)

at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.run(CdaJobEngine.java:1429)

at com.cisco.nm.rmeng.inventory.cda.job.CdaJobMonitor$ExecutorThread.run(CdaJobMonitor.java:244)

[ Thu May 31 12:10:18 MSD 2012 ],ERROR,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,308,exception occured at the time of closing cmdsvccom.cisco.nm.lib.cmdsvc.CmdSvcException: java.net.SocketException: Connection reset

at com.cisco.nm.lib.cmdsvc.OpConnect.invoke(OpConnect.java:57)

at com.cisco.nm.lib.cmdsvc.SessionContext.invoke(SessionContext.java:299)

at com.cisco.nm.lib.cmdsvc.Engine.process(Engine.java:57)

at com.cisco.nm.lib.cmdsvc.LocalProxy.process(LocalProxy.java:22)

at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:190)

at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:166)

at com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler.verify(CmdSvc_CDACredTypeHandler.java:202)

at com.cisco.nm.xms.xdi.pkgs.LibCda.GenericCdaHandler.checkSanity(GenericCdaHandler.java:37)

at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.checkSanity(CdaJobEngine.java:1565)

at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.run(CdaJobEngine.java:1429)

at com.cisco.nm.rmeng.inventory.cda.job.CdaJobMonitor$ExecutorThread.run(CdaJobMonitor.java:244)

[ Thu May 31 12:10:18 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,310,Some exception not handled....

[ Thu May 31 12:10:18 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,312,Not for enable test