DAP Debug trace not reporting all hostscan data

[jsblendorio]

My users are using Anyconnect 3.1.02040 with hostscan 3.1.02040 and csd 3.6.6228.

When running debugs from the console, or searching through logs, for some users there is no debug data being generated for the hostscan extensions.

Normally the debug will generate this:

DAP: User xxx, Addr x.x.x.x: Session Attribute endpoint.device.protection_extension="3.6.4900.2"

This is followed by the endpoint.fw, av, as, and av results. About half of the time I do not see these results in the debug.  There are no errors reported.  The DAP policy seems to function properly regardless of this output but this has made it difficult to troubleshoot login issues.  Has anyone else experienced this?

Thanks.

[Jim Thomas]

Same issue and opening a case..... unfortunately it is sporadic. I can connect with one connection profile and see all the beautiful enpoint info, then disconnect and connect to a different CP and it chokes part-way through the debug and never shows the endpoint info and therefore fails the DAPs.

[jsblendorio]

The cscan log file stored on the connecting PC still contain the endpoint info, what happens is does not transfer all of this info to the firewall.  This doesn't seem to have any bearing on the actual outcome of the DAP checks.

[blumley]

Very similar if not the same issue here as well.   We were running 8.2.5 and seeing a lot of DAP policy syslogs, endpoint,av, .fw, and .as,   messages missing.

Upgraded to 8.4.6(5)  and "thought" issues were resolved.   But now seeing same missing syslog records.  logging to ASDM shows ALL "endpoint." messages without issue.   Syslog queue does not seem to be overflowing.  Same symptoms with UDP or TCP syslogs.  Cisco TAC could only suggest upgrading to 9.x IOS, But I can't justify another downtime and upgrade unless I have a concrete reason for the issue.

Turning on " DEBUG DAP TRACE"  appear to alliviate the issue a bit.  But not certain if it completely eliminates the symptoms.  I can only guess that the debug on those messages may elevate them in the cpu process table.

Anyone else have any good ideas?  I am still waiting for TAC to come back with something that makes sense.

-BL

[Jim Thomas]

Yes we found the answer. The issue is that there was, prior to 9.x code a limit of approximately 1k-1000 lines of output allowed in the debug buffer. So this means depending on what hostscan finds (AV,AS,multiple AV,certs, etc), each one of these items consumes these debug lines. It turns out that if you have a lot of certs, every line of the cert is also presented on a separate line to the ASA via hostscan which consumes a lot of this buffer space. The short term solutions is to find a testing host without all these certs ...etc. But in reality its not a good solution. We had the BU increase the buffer space to 1400 lines and we still were filling this....Sooooo, the final solution was an engineers build that increased the buffer to 2MB and all worked fine. I think this will make it into a 9.1(3) release . I'll try and find a tracking ## for you.

By the way, converting debug output to syslogs results in the same scenario. The underlying issue is the debug buffer limit.

Jim Thomas
Cisco Security Course Director
Global Knowledge
CCIE Security #16674

[blumley]

Thank you for that info!

I have no need to collect ALL the endpoint.certificate, endpoint.device, and endpoint.os.hotfix hostscan results.

Is there a way to supress specific categories of hostscan results messages without disabling all of them entirely?

In particular I am only interested in:

aaa.cisco.username

aaa.radius

aaa.cisco.grouppolicy

aaa.cisco.class

aaa.cisco.username

aaa.cisco.tunnelgroup

endpoint.(av|fw|as)

endpoint.os.version

endpoint.os.architecture

endpoint.policy.location

endpoint.device.hostname

A couple other ideas I have been tossing around:

• Are there SNMP tables for the hostscan results perhaps?
• Has anyone looked at writing the hostscan results to a folder localy on the ASA's and pulling those files periodically via SCP?
• I was also looking into sending these messages as SNMP traps, but that looks to have the same issue with the buffer.

-BL

[Jim Thomas]

So unfortunately, you cannot restrict what is sent. Its an interesting thing though, many people think that the ASA is requesting the info from the device, which is the case if you are looking for specific registry entries or files on the drive, etc. But everything else (AV,AS, Certs, etc) automatically are forwarded to the ASA and then the ASA parses that ouput for the desired parameters. So, unfortunately, no you cannot limit this info. However, you can look at the logs on the local client or you have onesies and twosies that you need to troubleshoot. Trying to go long-term with that type of troubleshooting and with a lot of clients is just not doable.

So, as for the other questions, there is a kicker to all this. The hostscan information is not retained. Yup you got it, not retained at all. So once that debugging info is parsed, there is no way to go back and run reports on it or to see who has met posture or not. I filed another feature bug with this since the times have really changed and we MUST have this information moving forward. We'll see what they do with that request though as it will take more resources to store and report on that data.

The conversion from debug to anything is really where you're hosed again like you mentioned. The limiting factor is the debug buffer.

Heres the thing I've found during a recent large deployment. Cisco hasnt been utilizing the hostscan to the effect where clients are BEGINNING to use this. THere are a ton of little gotchas with their implementation. Things I found are getting better (or at least as I'm told) but I had to basically abandon a lot of the ASDM settings for endpoint control checks (EPCs) and use the advanced LUA. Actually if you use LUA, you can come up with some really cool scripts. I'm going to blog about this soon but as an example ...... Cisco doesnt have a button to click on in ASDM for an "All AV" check. Which as we all know is useful for contractors entering the network. It is however, supported using LUA . Unfortunately TAC doesnt support LUA scripting although the option is available.........wheres the "whaaaaa--mbulance"

enjoy!

Jim Thomas
Cisco Security Course Director
Global Knowledge
CCIE Security #16674

[blumley]

Did you find the case# for this issue on the ASA's?

I am experiencing a rash of new instances of missing DAP results on ASA 9.1.5  and AnyConnect 3.1.05152 as well as clientless WebVPN.