FWSM strange behavior

[Calin C.]

Hello all,

I have a strange problem with FWSM and maybe you can help with an advice.

From time to time the firewall stops to process traffic, this happens occasionally without any reason – no new connections are established, sometimes established session are terminated. During this time the FWSM responds very slow or not responding at all, OSPF neighborship to MSFC is flapping, login to FWSM not possible, L3 Interfaces on FWSM not responding to ICMP packet.

The issue dissapear as it appeared, without any action from our side..

During the problem persists, the load on the FWSM is not increased

- CPU, MEM & Connection in normal parameters

- Health indicators show everything is OK

- Interfaces are OK

The only thing we found which is indicating an issue are the NP thresholds as they are unusual high and increasing sometimes very fast

sh np bl

                 MAX   FREE   THRESH_0   THRESH_1   THRESH_2

NP1 (ingress)  32768  32768      15423     555210    9251522

    (egress)  521206 521206          0          0          0

NP2 (ingress)  32768  32768      36333    1981344   42078832

    (egress)  521206 521206          0          0          0

NP3 (ingress)  32768  32768      65082    3004200   11550832

    (egress)  521206 520636          0          0          0

after 49 minutes

sh np bl

                 MAX   FREE   THRESH_0   THRESH_1   THRESH_2

NP1 (ingress)  32768  32768      15423     555210    9251599

    (egress)  521206 521206          0          0          0

NP2 (ingress)  32768  32768      36333    1981344   42078832

    (egress)  521206 521206          0          0          0

NP3 (ingress)  32768  32768      79248    3048863   11840478

    (egress)  521206 520063          0          0          0

uptime: 2 years 99 days

CLS ACL Rule Count: 10390

Then we reloaded the FWSM module and the thresholds are now ok but I’m not sure if the issues is gone now or if it will appear again

                 MAX   FREE   THRESH_0   THRESH_1   THRESH_2

NP1 (ingress)  32768  32768        206      26689     591922

    (egress)  521206 521206          0          0          0

NP2 (ingress)  32768  32768        162      38850     918569

    (egress)  521206 521206          0          0          0

NP3 (ingress)  32768  32768          0          0          0

    (egress)  521206 520633          0          0          0

what could be the issues? I don’t think it’s HW because we have the FWSM in Active/Standby mode and after switching to Standby unit the behavior is the same.

Could this be caused eg. by a DoS attack?

Thanks a lot,

Calin

[Marvin Rhoads]

Given >2 years uptime, I would suspect your code level is a bit behind. What FWSM software release are you running? There are several bugs regarding NPs documented as having been fix in various FWSM 4.x releases. Reference the 4.0 Release Notes and search for "NP".

You didn't mention whether you are in single or multiple context mode. If the later, you are pushing the acl rule count close to the default maximum for a context. Have you taken a look at the limits and ways to reallocate them in the configuration guide (here)?

You may have seen this external link, but in case you haven't, there is an interesting explanation of the network processors and their role here. There is also some additional information here on CSC at this link (and further reading linked from there).

Hope this helps.