LMS 4.0.1 Inventory collection - Partially successful devices

[douglas.mckee]

Good Morning,

We have been using LMS 4.0.1 to backup our device configs for over 5 years now. As of last week 90% of our devices are coming back partially successful. We have never had this issue before and have done a restore using a verified backup with the same results. Has anyone had this issue before and if so how was it resolved?

Thank You,

Doug

[Allen A]

Hi Doug,

What is the error message you get? Usually when it says 'Partially Successful' means that it collected something but it didn't finish the job completely (Most of the time it archives the running config and fails for the vlan.dat depending on the protocols you are using). Have you tried removing and re-adding one device for testing? 

Allen

[douglas.mckee]

Allen,

Below is the error. It looks like the config is being backed up but not the vlan.dat files.

*** Device Details for ##.#.#.## *** 
Protocol ==> SSH 
Selected Protocols with order ==> SSH,TFTP,RCP,SCP 
Execution Result:
 
STARTUP 
Primary Login Succeeded 
/ Primary Enable Succeeded 
CM0060 PRIMARY STARTUP Config fetch SUCCESS for ##.#.##.###, version number 1 archived. 
 
RUNNING 
Primary Login Succeeded 
/ Primary Enable Succeeded 
CM0061 PRIMARY RUNNING Config fetch SUCCESS for ##.#.#.##, no change in configuration. 
 
VLAN 
CM0151 VLAN RUNNING Config fetch failed for ##.#.#.## Cause: 
Transfer Protocol ==> Unknown / Not Applicable 
Selected Transfer Protocols with order ==> 
 
tftp failed, Check whether tftpboot directory has proper permissions. 
VLAN Config fetch is not supported using TFTP. 
 
RCP is not supported as a connection protocol for VLAN Fetch, Only TELNET and SSH are supported as connection protocols for VLAN Fetch 
 
SCP is not supported as a connection protocol for VLAN Fetch, Only TELNET and SSH are supported as connection protocols for VLAN Fetch 
Action: Check if protocol is supported by device and required device package is installed. Check device credentials. Increase timeout value, if required. 
 

[douglas.mckee]

We have not removed and readded any of the devices yet. A few weeks ago we had an issue where we couldn't receive job emails and it turned out to be an issue with the firewall on the server that was blocking smtp. Our server person will be back tomorrow and he will check if tftp may be being blocked as well.

[Allen A]

Doug,

That could a reason why it isn't working, however TFTP is not a supported protocol for VLAN fetch, so it would be expected to fail,  perhaps it would be a good idea to enable debugs for this job and see exactly why it is failing.  If you are interested in checking if TFTP is being blocked, you can  try using TFTP as the only protocol and see if that archives the devices config.

 Have you also tried increasing the SSH timeout values?  

Allen

[douglas.mckee]

Allen,

I will go ahead and enable debug for our next job that runs for the ICServer application. Our SSH timeout values are set at 120 seconds and most devices are just a few hops away. I guess we could increase this for 500 seconds but not sure if this would change anything considering how high our current settings are. I can test a single device that is failing with the TFTP only protocol and see if that's still an issue. I will get back with you once I have these results.

Thank You,

Doug

[douglas.mckee]

Allen,

I increased the timeout value and enabled debug configuration for Archive Management. Below is the output prior to us having issues and the IC_Server log information as possible errors.

I did notice some devices that are still successful are authenticating with SNMPV3 while others are not. Some of the other successful devices in LMS 4.0.1 had both SNMPV1/2 along with SNMPV3 being enabled. When logging onto the device it shows only SNMPV3 as being enabled. 

As a test I removed the SNMPV1/2 additional information from a successful device in LMS 4.0.1 and left the SNMPV3 enabled and the device failed at the next archive backup.

VLAN  (Prior to having issues)
 
Transfer Protocol ==> tftp 
Selected Transfer Protocols with order ==> 
CM0060 VLAN RUNNING Config fetch SUCCESS for ##.#.###.##, version number 778 archived. 

-------------------------------------------------------------------------------------------------------

Caused by:
com.cisco.nm.lib.snmp.futureapi.SnmpReqTimeoutException: SnmpRequestTimeout on ##.#.###.### while performing SnmpGet at index = -1
 at com.cisco.nm.lib.snmp.futureapi.SnmpFuture.value(SnmpFuture.java:193)
 at com.cisco.nm.xms.xdi.DeviceContext.retrieveSysObjectID(DeviceContext.java:255)
 at com.cisco.nm.xms.xdi.DeviceContext.getSysObjectID(DeviceContext.java:218)
 at com.cisco.nm.rmeng.util.rmedaa.RMEDeviceContext.getSysObjectID(RMEDeviceContext.java:1067)
 at com.cisco.nm.xms.xdi.SdiEngine.calcDevTypesAndConstrAGIs(SdiEngine.java:315)
 at com.cisco.nm.xms.xdi.SdiEngine.request(SdiEngine.java:308)
 at com.cisco.nm.xms.xdi.SdiEngine.getDevRepr(SdiEngine.java:302)
 at com.cisco.nm.rmeng.inventory.ics.core.CollectionController.run(CollectionController.java:687)

cisco.nm.rmeng.util.logger.RMELogger,755,com.cisco.nm.rmeng.util.rmedaa.RMERepository,getAttr,181, Error getting value for: ##.#.###.##|SNMP_V3_MODE from system. Returning null
[ Wed Nov 04  12:11:29 AKST 2015 ],DEBUG,[Thread-18],com.cisco.nm.rmeng.util.logger.RMELogger,755,com.cisco.nm.rmeng.util.rmedaa.RMERepositoryUtil,fallBacktoSecondary,205,fromCmdSvc false fallBack false
[ Wed Nov 04  12:11:29 AKST 2015 ],DEBUG,[Thread-18],com.cisco.nm.rmeng.util.logger.RMELogger,755,com.cisco.nm.rmeng.util.rmedaa.RMERepositoryUtil,incrementLoginEnableCounters,69,fromCmdSvc false fallBack false