04-08-2012 06:18 PM
Hi all. Inventory in CiscoWorks with new devices ASR9K Series is not working. CW version: LMS3.2.1. Device: ASR-9006 AC Chassis. Credentials correct. Can any help me?
Screenshot1: inventory request fail.
Screenshot2: RME knows Cisco ASR9006 Router.
04-08-2012 06:56 PM
Your screenshot implies that ssh is failing.
Are you able to ssh to the ASR 9k from your CiscoWorks server using a 3rd party tool like PuTTY?
04-08-2012 07:02 PM
Dear Marvin, thanks for help.
Yes: ssh from server with PuTTY success.
04-09-2012 12:33 AM
LMS would only need snmp to do the Invenotry. If Inventory is failing please chekc if LMS is able to do snmpwalk to the device or not. You can try to test snmpwalk from LMS server to device via device centre or using snmpwalk.exe from $NMSROOT/Objects/jt/bin/ directory.
CLI eg:
c:\progra~1\CSCOpx\objects\jt\bin\>snmpwalk -v2c -c public 10.104.149.180 sysObjectID
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.9.1.283
If device is accessible via snmp, invenotry shouls succed. try to increase the snmp timeout as well in LMS from :
RME > Admin > System Preferences RME Device Attributes
Even if it fails you may want to share failing error and IC_Server.log from server.
-Thanks
04-10-2012 05:03 AM
Dear Vinod, thx for answer.
Snmpwalk works well. But there is an issue with device credentials. CW cannot connect to ASR at all. Sync archive work fails and CDA work fails for ssh and telnet, but reachability tests from CW seems good. I've tried to sniff packets with WS. And there is the issue: at first CW tries telnet and fails three times (It sent right login and password, but there is some kind of failure and it types password in the wrong field)
follow tcp stream in WS gives such output:
"username: username
password:
username: password
password:
"
But there aren't any failed attempts on TACACS+ server log.
Afterwards it tries ssh and fails again(WS screen attached). Logs from device tells such a thing (only for ssh):
SSHD_[65867]: %SECURITY-SSHD-6-INFO_GENERAL : Client ---.---.---.--- closes socket connection
SSHD_[65867]: %SECURITY-SSHD-3-ERR_GENERAL : Failed in version exchange
SECURITY-SSHD-6-INFO_GENERAL : Incoming SSH session rate limit exceeded
Credentials seems to be rigth. Putty connections from CW server via ssh and telnet under CW credentials are successfull. I have changed snmp/telnet/ssh timeouts in different manner but it didn't help.
04-11-2012 12:33 AM
Usually it is essential to configure the $NMSROOT\objects\cmf\data\TacacsPrompts.ini file. As with tacacs+ Auth you can also define custom login username and password prompt, hence this file is important to be configured, only in case of Telnet not SSH.
So just check the login prompt you get in your device when you try to do telnet and mention the same in you
TacacsPrompts.ini file.
Example:
>Following is the content of TacacsPrompts.ini :
[TELNET]
USERNAME_PROMPT=
PASSWORD_PROMPT=
> Following is the username and pasword propmt for my device :
> As per this please modify the file as :
[TELNET]
USERNAME_PROMPT=Username:
PASSWORD_PROMPT=Password:
**NOTE : File is case Sensitive. be specific as per what you get while logging in.
Also, try to increas the timeout for Telent via RME > Admin > Device attribute. Just save and try to run the job again for failing device.
-Thanks
Vinod
04-12-2012 04:46 AM
Dear Vinod,
Thanks a lot. Your solution is very helpfull. But i disinformed You. Sorry for that. I'm actually intrested in config archive rather than inventory collection. So config archive works well via telnet, but with ssh it fails. And CDA work for ssh fails.
Is there the same solution with ssh?
With ssh there is some other promts:
login as:
yourlogin@deviceIP's password:
Should i change TacacsPrompts.ini file in the same manner as for telnet?
Or there is some ssh daemon bug on the device?
-Thanks
Nikolay
04-12-2012 07:28 AM
No SSH does not requires that file. Though you can add these prompts to the file with comma separation and try.
There are some known issues wih LMS with IOX-XR using SSHv2. You may be hitting CSCte95623. There is a patch for LMS 3.2.1/RME4.3.2, available with TAC.
-Thanks
Vinod
05-31-2012 01:52 AM
Hello again and thx for advice,
I've tried the solution from Cisco for this bug (CSCte95623 ), by manipulating delays values in cmdsvc.properties file and restarting cfgmngmt process. I've changed delay values in very different manner (delay after connect, tunesleepmills, login, e.t.c). Unfortunately this solution didn't help. A CDA work for SSH fails all the time. Also i've manipulated
ssh rate-limit and ssh session-limit values on device. It's a pity that opportunity to set on only sshv1 on device doesn't exist, so CW tries to connect only with sshv2 and there is no chance to check how it work with sshv1.
I'm becoming a bit desperate about that issue. Any ideas?!
There is some output from ssh debugs on device:
debug ssh server
RP/0/RSP1/CPU0:May 31 12:02:14.068 : SSHD_[1114]: Spawned new child process 5869901
RP/0/RSP1/CPU0:May 31 12:02:14.149 : SSHD_[65869]: Client sockfd 3
RP/0/RSP1/CPU0:May 31 12:02:14.151 : SSHD_[65869]: Setting IP_TOS value:192
RP/0/RSP1/CPU0:May 31 12:02:14.152 : SSHD_[65869]: After setting socket options, sndbuf33792, rcvbuf - 33792
RP/0/RSP1/CPU0:May 31 12:02:14.153 : SSHD_[65869]: Connection from ------------ port ---------
RP/0/RSP1/CPU0:May 31 12:02:14.158 : SSHD_[65869]: (addrem_ssh_info_tuple) user:()
RP/0/RSP1/CPU0:May 31 12:02:14.162 : SSHD_[65869]: Session id 0
RP/0/RSP1/CPU0:May 31 12:02:14.162 : SSHD_[65869]: Exchanging versions
RP/0/RSP1/CPU0:May 31 12:02:14.164 : SSHD_[65869]: %SECURITY-SSHD-6-INFO_GENERAL : Client ------ closes socket connection
RP/0/RSP1/CPU0:May 31 12:02:14.164 : SSHD_[65869]: %SECURITY-SSHD-3-ERR_GENERAL : Failed in version exchange
RP/0/RSP1/CPU0:May 31 12:02:14.164 : SSHD_[65869]: In cleanup code, pid:5869901, sig rcvd:0, state:1
RP/0/RSP1/CPU0:May 31 12:02:14.166 : SSHD_[65869]: Cleanup sshd process 5869901, session id 0
RP/0/RSP1/CPU0:May 31 12:02:14.171 : SSHD_[65869]: Closing connection to --------
RP/0/RSP1/CPU0:May 31 12:02:14.171 : SSHD_[65869]: Sending Disconnect msg
RP/0/RSP1/CPU0:May 31 12:02:14.172 : SSHD_[65869]: sshd_shm_acquire_lock: SHM Lock is NULL
RP/0/RSP1/CPU0:May 31 12:02:14.172 : SSHD_[65869]: sshd_shm_unlock: SHM Lock is NULL
RP/0/RSP1/CPU0:May 31 12:02:14.184 : SSHD_[1114]: Signal 18 received in handler: pid 5869901
RP/0/RSP1/CPU0:May 31 12:02:14.207 : SSHD_[1114]: ratelimit_msecs:1000.000000, ratelimit_count:1
RP/0/RSP1/CPU0:May 31 12:02:14.207 : SSHD_[1114]: elapsed:145.976000, ratelimit_msecs:1000.000000, count:1
RP/0/RSP1/CPU0:May 31 12:02:14.207 : SSHD_[1114]: %SECURITY-SSHD-6-INFO_GENERAL : Incoming SSH session rate limit exceeded
And CDA ssh work log from CW:
Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1571,Iam inside ssh ....
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1573,Initial time_out : 0
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1583,Computed time_out : 30
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getCmdSvc,1599,After computing time_out : 30
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getSshCmdSvc,1637,inside getSshCmdSvc with timeout : 30000
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getSshProtocols,1743,Inside getsshprotocols with time out : 30000
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext,getSshCmdSvc,1651,SSH2 is running
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,136,Got CmdSvc for SSH
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,141,Before Resetting the counters i.e before invoking counters for CredType :: SSH
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,151,After Resetting the counters i.e before invoking counters for CredType :: SSH
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,156,Getting Primary credentails to reset again to Primary only..
[ Thu May 31 12:10:17 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,201,trying to connect for SSH
[ Thu May 31 12:10:18 MSD 2012 ],ERROR,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,272,Got CmdSvcException com.cisco.nm.lib.cmdsvc.CmdSvcException: java.net.SocketException: Connection reset
at com.cisco.nm.lib.cmdsvc.OpConnect.invoke(OpConnect.java:57)
at com.cisco.nm.lib.cmdsvc.SessionContext.invoke(SessionContext.java:299)
at com.cisco.nm.lib.cmdsvc.Engine.process(Engine.java:57)
at com.cisco.nm.lib.cmdsvc.LocalProxy.process(LocalProxy.java:22)
at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:190)
at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:166)
at com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler.verify(CmdSvc_CDACredTypeHandler.java:202)
at com.cisco.nm.xms.xdi.pkgs.LibCda.GenericCdaHandler.checkSanity(GenericCdaHandler.java:37)
at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.checkSanity(CdaJobEngine.java:1565)
at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.run(CdaJobEngine.java:1429)
at com.cisco.nm.rmeng.inventory.cda.job.CdaJobMonitor$ExecutorThread.run(CdaJobMonitor.java:244)
[ Thu May 31 12:10:18 MSD 2012 ],ERROR,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,308,exception occured at the time of closing cmdsvccom.cisco.nm.lib.cmdsvc.CmdSvcException: java.net.SocketException: Connection reset
at com.cisco.nm.lib.cmdsvc.OpConnect.invoke(OpConnect.java:57)
at com.cisco.nm.lib.cmdsvc.SessionContext.invoke(SessionContext.java:299)
at com.cisco.nm.lib.cmdsvc.Engine.process(Engine.java:57)
at com.cisco.nm.lib.cmdsvc.LocalProxy.process(LocalProxy.java:22)
at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:190)
at com.cisco.nm.lib.cmdsvc.CmdSvc.connect(CmdSvc.java:166)
at com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler.verify(CmdSvc_CDACredTypeHandler.java:202)
at com.cisco.nm.xms.xdi.pkgs.LibCda.GenericCdaHandler.checkSanity(GenericCdaHandler.java:37)
at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.checkSanity(CdaJobEngine.java:1565)
at com.cisco.nm.rmeng.inventory.cda.job.DoCDAonDevice.run(CdaJobEngine.java:1429)
at com.cisco.nm.rmeng.inventory.cda.job.CdaJobMonitor$ExecutorThread.run(CdaJobMonitor.java:244)
[ Thu May 31 12:10:18 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,310,Some exception not handled....
[ Thu May 31 12:10:18 MSD 2012 ],INFO ,[Thread-1],com.cisco.nm.xms.xdi.pkgs.LibCda.CmdSvc_CDACredTypeHandler,verify,312,Not for enable test
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide