Unable to see ACI contract hit counts

[Wayne Knight]

I am running ACI 5.2(5c). I am not seeing any hits for the contracts in use. Using the contract_parser tool, all contracts are showing [hit=?]. We are not using policy compression as I know this will disable hit counts. I have also checked the monitoring policies and these seem to be the default. I have another customer aci and can see hit counts, both monitoring policies look the same. I am not seeing any stats using "show system internal policy-mgr stats". No ingress/egress or pkts count.

Any ideas or what to check. I dont recall ever seeing these stats since install.

[Sergiu.Daniluk]

Hi @Wayne Knight 

Do you have the "log" enabled for the contract filters:

Take care,

Sergiu

[Wayne Knight]

Hi @Sergiu.Daniluk 

I dont have the log option set. Here is some more outputs that show log is not set on either. I have removed naming and replaced with <> placeholders. For the working one there are no entries with hit=? they will either have 0 or some higher value, while the non-working one all entries have hit=?

Just to add, this is not impacting traffic, just our ability to see stats. Is this expected behaviour or am I missing something.

working example
contract parser output
[9:4372] [vrf:<vrf name>] permit any <epg1>(49153) <epg2>(49154) [contract:<contract name] [hit=941447994]

show zoning-rule | grep 4372
| 4372 | 49153 | 49154 | default | uni-dir | enabled | 3112960 | | permit | src_dst_any(9) |

Non working example
All entries have hit=?
[9:46394] [vrf:<vrf name>] permit any <epg1>(41) <epg2>(38) [contract:<contract name>] [hit=?]

show zoning-rule | grep 46394
| 46394 | 41 | 38 | default | uni-dir-ignore | enabled | 2621440 | | permit | src_dst_any(9) |

Wayne

[Sergiu.Daniluk]

Ah got it. Sorry, initially I thought contract logging is not working.

So basically, what you say is that the contract parser is showing "hit=?" for some contracts, right?

Based on the code:

1712                    # initial stats (set to ? if not found)
1713                    hits = "[hit=?]"

  seems like the initial value of the hit count is "?" and if the contracts rule ids are not found in the actrlRuleHit5min.

Try this command and see if you get any counts: 

moquery -d sys/actrl/scope-2621440/rule-2621440-s-41-d-38-f-9/CDactrlRuleHit5min

I do not have any results with "?" so it's hard for me to say what is causing exactly this issue.

Cheers,

Sergiu

[Wayne Knight · Accepted solution]

So I did a bit more digging and also looked into the python code to see why it was having the ? instead of the values, and I too came across actrlRuleHit5min. It turns out I cannot even query this as below or its parent class

apic-01# moquery -c actrlRuleHit5min
No Mos found

apic-01# moquery -c actrlRule
Error: URL: http://127.0.0.1:7777/api//class/actrlRule.xml?
Code: 400
Output: <?xml version="1.0" encoding="UTF-8"?><imdata totalCount="1"><error code="400" text="Unable to process the query, result dataset is too big"/></imdata>
Data Posted:
None

This led me to a bug hunt and came across the following. (our current rule count is over 40000).

CSCwc08792  - Contract stats not updated for more than 21809 rules

Symptom:
Contract stats not updated for more than 21809 rules thought current scale limit for actrlRuleHit5min contract stats is 25k.

Conditions:
When there is a big scale of  zoning-rules,  Contract stats not updated for more than 21809 rules

[Sergiu.Daniluk]

"moquery -c actrlRuleHit5min" - should be done on the leaf.

[Wayne Knight]

I ran the moquery on the "working" apic and can see data for actrlRule and actrlRuleHit5min, so it is still a valid test. For completeness I have run this against two different leafs, still no stats, but can see parent data for actrlRule.

leaf-01# moquery -c actrlRuleHit5min
No Mos found

leaf-01# moquery -c actrlRule | more
Total Objects shown: 50452
<output omitted>

leaf-03# moquery -c actrlRuleHit5min
No Mos found

leaf-03# moquery -c actrlRule | more
Total Objects shown: 49210
<output omitted>