Inventory Collection Failure Following Upgrade to 3.2

[BRANDON PORTER]

We have 90 Cisco VPN 3002 Hardware clients deployed that we manage using Ciscoworks. Before our recent upgrade from 3.1 to 3.2 all VPN 3002 devices previously inventoried and fetched configs successfully.

Following upgrade this week we are seeing Inventory Collection failures for some (not all) of the VPN 3002 devices.

Not sure if upgrade from 3.1 to 3.2 installs a new device package for this device, but checked following upgrade and there are no Software, Device or Package updates on CCO that we need to install.

I have attached output from IC_Server.log for one of the failed VPN 3002 devices and it looks like the Inventory Collection failure is SNMP related (see attached).

Any idea if we need to upgrade code on the VPN 3002 devices? I tried deleting device from Ciscoworks and readding without success and Config Fetch is working on readded device.

[Joe Clarke]

There appears to be a problem getting the ifTable from these devices. It would be helpful to see a sniffer trace filtering on SNMP traffic to one failing device, then on one successful 3002 when performing inventory collection to see if there are any obvious problems. The device is replying with a noSuchName error, but it's not obvious exactly which object is causing the problem.

The other error here is related to the following Altiga objects:

alSepModuleStatsType

alSepModuleStatsState

alSepModuleStatsDspCodeVersion

Again, a noSuchName error is coming from the device.

There were no code changes in the 3.2 device package for VPN3000 devices, so something must be going on with these devices.

[BRANDON PORTER]

I will work on getting a sniffer on the link. From the Ciscoworks Device Center I am able to run an SNMP Walk against the VPN 3002 using v2c. I have attached the SNMP Walk output.

It seems like if the Ciscoworks server can walk the device and fetch its config, it should be able to collect inventory.

Any other things we can check in the meantime?

[BRANDON PORTER]

I am not sure if I did this capture correctly, but I installed Winpcap on our Ciscoworks server, created a packet capture from Ciscoworks to the 172.18.51.65 device and I have attached the capture output.

Let me know if this is of any help or if I need to re-run or modify the packet capture.

[Joe Clarke]

Post a capture of a successful inventory collection of a VPN3000 device.

[Joe Clarke]

It looks like the v2c GETBULK requests are all failing, but I'm curious as to why you have some that are working. That's why I'd like to see a sniffer trace of one of those devices. What is different between the working and non-working devices?

[BRANDON PORTER]

Upon further review all VPN 3002 devices are failing Inventory Collection following our upgrade. We have 90 devices deployed and 22 show inventory collection failed. All of the 3002 devices that showed successful Inventory Collection in Ciscoworks all completed their last schedulded Inventory Collection on 7/21/09. We ran our upgrade from 3.1 to 3.2 on 7/22/09. I ran new collections on the devices that previously collected successfully and now all 3002 devices show up as failing.

I'm going to go ahead and open a TAC case on the issue. If there is anything else we can troubleshoot in the meantime let me know and thanks for taking a look at this for us.

[Joe Clarke · Accepted solution]

That explains it, then. The switch to supporting SNMPv2c has broken things for the VPN3000 series which support v2c but not GETBULK. This is a bug.

[Velin Georgiev]

I faced the same problem. Is there any solution? I couldn't find any bug description for this problem.

[Joe Clarke]

The bug is CSCtb00637.  A patch is available from TAC which fixes the problem.

[jmorrison_phn]

Ran into this problem with 3.2 and VPN3020. But your  bugid CSCtb00637 says:

CSCtb00637            Bug Details

tls's traffic failed after wan interface "shut" and "no shut"

None Symptoms: When user Shutdown Enthernet NSI interface and no shutdown, The TLS L2VPN traffic fail.

What's the right bugid/patch?

[Joe Clarke]

Sorry, the bug is CSCtb06637.

[bszemethy]

Hi!

I need that patch. What can I do?

Balazs

[Joe Clarke]

You can open a service request with TAC to get the patch, or wait a week or so as LMS 3.2 SP1 will be posted, which contains the fix.