FWSM in Cisco 6500 - High CPU usage

[dneggers1]

Hi Guys,

We have a Cisco Catalyst 6500 with a FWSM running V 4.0(6)

We have noticed that when we apply new rules into the ACL (through ASDM or CLI) that after the ACL is applied,

the CPU sits very high (90-100%) for up to 20 min.

During this time, the new rules you have entered into the ACL, do not work, until the CPU drops back to normal baseline usage which is about 10%, after about 20 min or so...

Looking at CPUHOG during this time, doesnt give us an indication about what is happening..

Any ideas?

Dion

[Panos Kampanakis]

Keep in mind that if you ACL is big and if you have ACL optimization enabled the CPU could spike up to 10-15 minutes.

Also note bug "CSCta62033: Adding remark lines to an optimized ACL can trigger prolonged high CPU" that is fixed in 4.0.7.

I hope it helps.

PK

[dneggers1]

thanks for the reply,

However we are not using ACL optimisation..

Could this occur without Optimisation turned on?

Cheers

Dion

[Panos Kampanakis]

Hmm, 20 mins is a little high. Up to 10 depending on traffic could be normal.

It could also be the bug I mentioned.

PK

[Sven Hruza]

Hello,

we use a FWSM cluster in 6k5 with Sup720, too.

Software version is 4.0(12).

We see the same, changing one ACL results in having CPU of 90% over nearly 10 minutes.

For that time the new ACL is not active.

Is there some new information about that?

Thanks a lot!

Sven

[Panos Kampanakis]

Sven,

If your ACLs size is big. more than 50K rules for example, this is probably normal.

Especially if you also have ACL optimization on.

In general is it normal to see your CPU go to 80-90% and the time depend on ACL size and optimization.

The 90% should yield to other processes so it should not interrupt traffic. And also, while the new ACL is compiled the old ACL is backed up in a special ACL partition and it is the ACL still used before the new ACL is compiled and put into action.

I hope it helps.

PK

[Sven Hruza]

Thanks a lot for the reply!

Yes, I think we have something about 65k of rules.

I can see that in the output of sh np 3 acl count 0, right?

-------------- CLS Rule Current Counts --------------
CLS Filter Rule Count       :             0
CLS Fixup Rule Count        :          5621
CLS Est Ctl Rule Count      :             0
CLS AAA Rule Count          :             0
CLS Est Data Rule Count     :             0
CLS Console Rule Count      :            58
CLS Policy NAT Rule Count   :             0
CLS ACL Rule Count          :         65493
CLS ACL Uncommitted Add     :             0
CLS ACL Uncommitted Del     :             0

---------------- CLS Rule MAX Counts ----------------
CLS Filter MAX              :          3747
CLS Fixup MAX               :          5621
CLS Est Ctl Rule MAX        :           624
CLS Est Data Rule MAX       :           624
CLS AAA Rule MAX            :          8744
CLS Console Rule MAX        :          2498
CLS Policy NAT Rule MAX     :          2498
CLS ACL Rule MAX            :        100567

And is it right that the max number of ACL is 100567 for the system? What will happen if we get more than those ACLs?

Thanks a lot!

Sven

[Panos Kampanakis]

Yep, you seem to have many ACL rules, so the compilation will take a few minutes.

If you reach the 100K limit then the FWSM will not let you add more rules and it will give you an error when you add a rule saying "ACL rule limit reached".

I hope it makes sense.

PK

[Kureli Sankar]

Here is a nice document that I wrote on FWSM acl limit: https://supportforums.cisco.com/docs/DOC-8786

-KS

[Sven Hruza]

Thanks a lot for the repsonses and the link to your documentation.

I will try to reduce the number of ACLs and objects on my FWSM.

Sven

[Panos Kampanakis]

Glad we could help.

Take care,

PK

[sudheesh.pb]

You can enable ACL optimization , which will reduce the number of ACLs, and the FWSM will keep the optimized configuration separate than the normal running configuration. Then copy the optimized running configuration in to the running configuration using the command "copy optimized-running-config running-config" will replace the existing running configuration with the optimized one. using this you will be able to reduce the number of ACLs and there by increasing the FWSM performance. Please let me know once you done.

[Sven Hruza]

Hello,

thanks for that hint, sudheesh.ph.

I thought that ACL optimization is only for finding double ACLs in the configuration.

Or is there something else the optimization will do?

It is not easy to activate the optimization because the FWSM is very important for our production.

So I can't try that in the next time.

How can I activate that but get my "normal" config running?

I want to check out the differences between normal and optimized config.

I thought by activating the optimization in ASDM and apply it, it will get active in the running-config and is productive.

Thanks a lot!

Sven

[sudheesh.pb]

Hi,

ACL optimization will find the double entries and it will try to combine the rules if possible. you will be able to see minimum of a 60% reduction in the ACLs after enabling optimization. There is no issues on doing this exercise on a production blade, as normally it will not impact the usual traffic and sessions.

The fwsm keeps the original ACL in the configuration for user convenience. However, the version that is compiled into the hardware is the ACL displayed through the “show access-list optimization” command. Therefore, after entering the “access-list optimization enable” command, you will see two ACLs present in the configuration. Modifications are always made to the original ACL, and the optimization process runs its course using the new changes. Users cannot directly modify the optimized version of the ACL. So it is possible for you to compare both the configurations.  rank me if this information helps you..

Thanks,

Sudheesh