Upon investigation, I found the error was from the built-in SNMP probe that PTRG uses to discover the VPN statistics for ASAs. It is described as follows in the PRTG manual:
SNMP Cisco ASA VPN Connections Sensor
The SNMP Cisco ASA VPN Connections sensor monitors the VPN connections on a Cisco Adaptive Security Appliance using the Simple Network Management Protocol (SNMP).
The sensor can show the following:
- Active email sessions
- Active Internet Protocol Security (IPsec) sessions
- Active LAN-to-LAN (L2L) sessions
- Active LB sessions
- Active sessions in total
- Active switched virtual circuit (SVC) sessions
- Active users
- Groups with active users
The probe had been working perfectly well across prior releases going back at least 5 years.
PRTG does not document the SNMP Object ID (OID) variables the sensor uses they use since they consider those part of their value to package all that for the customer and keep the details to themselves. However, I was able to ascertain exactly what they do by pausing all sensors it was polling and then re-enabling just that one.
A quick Wireshark capture with the capture filter for only SNMP queries (udp port 161) revealed the following:
(Once I got that, I re-enabled all the sensors.)
From the data of the captured frame we can see 9 variables are queried with an SNMP get-request. They are all taken from the Cisco remote access system (cras) monitor MIB tree. We can find a reference here explaining the variables:
http://oidref.com/1.3.6.1.4.1.9.9.392.1.3
The problem is that, when the ASA responds, one of those variables is not recognized:
Note the “noSuchObject” response for variable 1.3.6.1.4.9.9.392.1.3.23.0.
A check on the ASA itself confirms that variable is no longer supported:
asa# show snmp-server oidlist | i 1.3.6.1.4.1.9.9.392.1.3 [681] 1.3.6.1.4.1.9.9.392.1.3.1. crasNumSessions [682] 1.3.6.1.4.1.9.9.392.1.3.2. crasNumPrevSessions [683] 1.3.6.1.4.1.9.9.392.1.3.3. crasNumUsers [684] 1.3.6.1.4.1.9.9.392.1.3.4. crasNumGroups [685] 1.3.6.1.4.1.9.9.392.1.3.5. crasGlobalInPkts [686] 1.3.6.1.4.1.9.9.392.1.3.6. crasGlobalOutPkts [687] 1.3.6.1.4.1.9.9.392.1.3.7. crasGlobalInOctets [688] 1.3.6.1.4.1.9.9.392.1.3.8. crasGlobalInDecompOctets [689] 1.3.6.1.4.1.9.9.392.1.3.9. crasGlobalOutOctets [690] 1.3.6.1.4.1.9.9.392.1.3.10. crasGlobalOutUncompOctets [691] 1.3.6.1.4.1.9.9.392.1.3.11. crasGlobalInDropPkts [692] 1.3.6.1.4.1.9.9.392.1.3.12. crasGlobalOutDropPkts [693] 1.3.6.1.4.1.9.9.392.1.3.21.1.2. crasGroup <some lines omitted for brevity> [734] 1.3.6.1.4.1.9.9.392.1.3.22.1.8. crasActGrpOutOctets [735] 1.3.6.1.4.1.9.9.392.1.3.26. crasIPSecNumSessions [736] 1.3.6.1.4.1.9.9.392.1.3.27. crasIPSecCumulateSessions [737] 1.3.6.1.4.1.9.9.392.1.3.28. crasIPSecPeakConcurrentSessions [738] 1.3.6.1.4.1.9.9.392.1.3.29. crasL2LNumSessions [739] 1.3.6.1.4.1.9.9.392.1.3.30. crasL2LCumulateSessions [740] 1.3.6.1.4.1.9.9.392.1.3.31. crasL2LPeakConcurrentSessions [741] 1.3.6.1.4.1.9.9.392.1.3.32. crasLBNumSessions [742] 1.3.6.1.4.1.9.9.392.1.3.33. crasLBCumulateSessions [743] 1.3.6.1.4.1.9.9.392.1.3.34. crasLBPeakConcurrentSessions [744] 1.3.6.1.4.1.9.9.392.1.3.35. crasSVCNumSessions [745] 1.3.6.1.4.1.9.9.392.1.3.36. crasSVCCumulateSessions [746] 1.3.6.1.4.1.9.9.392.1.3.37. crasSVCPeakConcurrentSessions [747] 1.3.6.1.4.1.9.9.392.1.3.38. crasWebvpnNumSessions [748] 1.3.6.1.4.1.9.9.392.1.3.39. crasWebvpnCumulateSessions [749] 1.3.6.1.4.1.9.9.392.1.3.40. crasWebvpnPeakConcurrentSessions asa#
The earlier oidref.com link tells us that the no-longer-supported 1.3.6.1.4.1.9.9.392.1.3.23 variable is used for “The number of currently active Email proxy sessions” – something we do not really care about. (…and something that I don’t know why we ever would have cared about on an ASA – does anybody know?)
What we are interested in (for my use case of monitor remote access SSL VPN users) is 1.3.6.1.4.1.9.9.392.1.3.35.0 or crasSVCNumSessions which is “The number of currently active SVC sessions”.
(NOTE: “SVC” is a legacy term for SSL VPN Client. It is in contrast with “WebVPN” which is used when referring to clientless SSL VPN. It can be a bit confusing since both are enabled in the “webvpn” section of an ASA configuration.)
Solution
Since the built-in poller is failing us, the alternative approach I used was to create a custom SNMP poller in PRTG. I did so as shown below, giving it a name and the OID we had found earlier:
I assigned that sensor to the ASAs (in lieu of the PTRG built-in one for VPN status) and – voila – we have a green sensor and a useful graph:
Note: At the time I first encountered the problem we were using PRTG version 20.1.56.1547+. While writing this up, I noticed there is a slightly newer release available - 20.2.58.1629. I since upgraded to that release and re-tested the built-in sensor. It continues to exhibit the problem. So, we will stick with the custom sensor for now.
Conclusion
During any system upgrade, analysis of behavior before and after is important to understand possible impacts to operations. When discrepancies are noted, a careful observation of the exact problem area combined with rigorous analysis can often highlight a potential solution. As we endeavor to continue to secure our systems using software which addresses recently disclosed vulnerabilities we must balance that with the need to operate and maintain the same systems.
References
- The recent set of vulnerabilities announced by Cisco that prompted us to upgrade.
- ASA 9.12 main train release notes.
- ASA 9.12 interim release notes
- PRTG manual section describing the Cisco ASA VPN Connections Sensor
