09-20-2010 01:22 PM - edited 03-08-2019 06:36 PM
There are many times where indications of oversubscription or excessive load on a firewall or a network device are not enough to prove if oversubscription is really happening. Thus, it is often confusing how to identify and solve such issues. This document will present the basic troubleshooting steps that someone needs to take in order to pinpoint an oversubscription problem on a Cisco FWSM firewall and will propose potential solutions to overcome it. The corresponding document for the ASA is located here.
The most important aspect of solving an oversubscription issue is its identification. Network engineers will often incorrectly attribute network problems to excessive traffic which leads devices like the firewalls to be wrongly considered as the bottleneck. Other times they will focus on other parts of the network in cases were the firewall processing power is not enough to handle the traffic. There can be multiple indications of load problems on firewall devices and putting them together will help us understand if traffic is indeed the reason of the problem or if we should focus elsewhere. That is what this section will try to describe.
Oversubscription almost never occurs by itself. It will most of the times be presented as another network problem that results from it. Such often include packet loss, slow response or drops. In general, an oversubscribed device that can't handle the load will inevitably drop some packets. Packet drops will affect sensitive applications or will cause TCP re-transmissions and affect the user experience by making transactions look as if they are taking more time to complete. If we wanted to summarize the problems that occur due to excessive load we would describe them as network degradation. Of course, someone must be careful and NOT attribute all problems that fall under the "degradation umbrella" as load issues. The indications we will present below will help more on identifying if such issues should be attributed to excessive load.
A "busy" firewall device will almost always show it on its CPU. We can check the CPU use with the command "show cpu".
FWSM# show cpu
CPU utilization for 5 seconds = 14%; 1 minute: 10%; 5 minutes: 10%
A CPU ranging above 80%-90% could indicate high traffic load.
Though, someone would need to pay attention as the FWSM's architecture is different than that of an ASA firewall and thus the CPU is not handling regular packet processing as on an ASA. It is worth noting that due to its architecture an FWSM could have low CPU while being oversubscribed. The reason is that on an FWSM the traffic is handled by 3 network processors (NP1, NP2, NP3) that are not included in the "sh cpu" output. Only for the FWSM, we can also check the aferomentioned network processor block thresholds.
FWSM# sh np block
MAX FREE THRESH_0 THRESH_1 THRESH_2
NP1 (ingress) 32768 32768 1234 242333 34343434
(egress) 521206 521206 0 0 0
NP2 (ingress) 32768 32768 0 0 0
(egress) 521206 521206 0 0 0
NP3 (ingress) 32768 32768 2333 44443 324434354
(egress) 521206 521206 0 0 0
Seeing counters increasing for these thresholds shows us that the processors are getting close to their limits The NP blocks and what they mean are explained here.
The FWSM interfaces statistics cannot be used to check if traffic an be handled by the device. These interfaces do not physically exist on the device. They are "virtual" between the switch and the FWSM backplane (6Gig interfaces forming an Etherchannel), so they will not show errors, overruns or underruns even if the FWSM is oversubscribed due to high traffic. Though, we could potentially check the "show nic" command output.
FWSMr# show nic
interface gb-ethernet0 is up, line protocol is up
Hardware is i82543 rev02 gigabit ethernet, address is 0012.0023.9200
PCI details are - Bus:0, Dev:0, Func:0
MTU 16000 bytes, BW 1 Gbit full duplex
15233374 packets input, 5822183269034098688 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
10486585 packets output, 4727775737442992128 bytes, 0 underruns
input queue (curr/max blocks): hardware (0/25) software (0/0)
output queue (curr/max blocks): hardware (0/13) software (0/0)
interface gb-ethernet1 is up, line protocol is up
Hardware is i82543 rev02 gigabit ethernet, address is 0012.0023.9200
PCI details are - Bus:0, Dev:0, Func:0
MTU 16000 bytes, BW 1 Gbit full duplex
8849745 packets input, 4807174976778010624 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
4 packets output, 1477468749824 bytes, 0 underruns
input queue (curr/max blocks): hardware (0/16) software (0/0)
output queue (curr/max blocks): hardware (0/1) software (0/0)
This command shows the interfaces between the CPU and NP3 (mentioned above). Seeing "no buffer" or "error" counters going up could mean excessive control plane traffic between the CPU and NP3, which could point to oversubscription.
Next it is worth checking the traffic that the device is seeing. We need to clear the traffic ("clear traffic" command) statistics before checking them ("show traffic" command). We are doing that because we want to see the traffic while the problem is occurring and thus be able to tell if load is related to the problem investigated. Looking the aggregate traffic output from "show traffic" carries information since the last reload or the last time the counters were cleared, so it will not help us identify how much traffic the box is seeing for the time we are troubleshooting. After the "clear traffic" we let the box collect statistical information for 2-5minutes and we do "show traffic" to get the traffic the interfaces saw.
FWSM# clear traffic
...
...5 minutes go by...
...
FWSM# show traffic
..
int2:
received (in 2090512.330 secs):
2327338 packets 319964508 bytes
1 pkts/sec 1 bytes/sec
transmitted (in 2090512.330 secs):
2327498 packets 338246 bytes
23323 pkts/sec 24324456 bytes/sec
int3:
received (in 2090512.330 secs):
1858298 packets 139580776 bytes
32422 pkts/sec, 291777777 bytes/sec
transmitted (in 2090512.330 secs):
139235118 packets 103732100 bytes
234242343 pkts/sec, 2232423237 bytes/sec
There are long discussions that people could start trying to tell if a firewall or any other device is hitting its traffic processing limits or not. Experience has shown that there is controversy on what the numbers show and what engineers consider as being close to the numbers or not. It is worth clarifying a few points. Let's use the FWSM as an example. Its name throughput is 5.5Gbps. So the question is, "if my FWSM sees about 5Gbps is it at its limits or not?". A quick answer would be "No". Though, we must not forget that there are many factors involved in this question. In the network industry name speeds of devices come out under certain tests. These tests are repeated and an average is presented as the maximum speed. Though, not always is "real-world" traffic the same traffic as the one used in the tests. We could use the aforementioned FWSM for example. Usually, the name speed tests involve stateless protocols with big packets. For a TCP web browsing application though, the packets are much smaller and TCP uses ACKs and is a "synchronized" protocol by nature. That would add more load to the firewall itself, which would make its maximum throughput value drop. On top of that, if the ASA has http inspection configured (which will do deep packet inspection for http) then we understand that its maximum processing throughput would be less than 5Gbps. It is obvious that even though 5.5Gbps is indeed the throughput the device can achieve, its real-world throughput, based on applications, traffic nature and configuration could practically be less. That is why in our performance documents we also try to provide other metrics. These include the "packets per seconds" (pps) and what is often seen as "real-world HTTP".
Another consideration on top of traffic load for the firewall devices is connections and connection rates. That is another field that could trigger various disagreements. The command we would use to see the connections on our firewall are "show conn count" and "show resource usage".
FWSM# show conn count
2 in use, 86 most used
FWSM# show resource usage
Resource Current Peak Limit Denied Context
Telnet 1 1 5 0 System
Syslogs [rate] 1 293 N/A 0 System
Conns 2 86 10000 0 System
Xlates 5 116 N/A 0 System
Hosts 6 49 N/A 0 System
FWSM-multi-context# show resource usage
Resource Current Peak Limit Denied Context
SSH 1 1 15 0 admin
Syslogs [rate] 118 348 unlimited 0 context1
Conns 89 893 unlimited 0 context1
Xlates 150 1115 unlimited 0 context1
Hosts 15 18 unlimited 0 context1
Conns [rate] 603 14694 unlimited 0 context1
Now, let's ask one more questions for the output from our FWSM above: "In the peak connection rate I see about 15K connections and in the specifications I read that the maximum supported rate is 100K conns/second. 15K is much less than 100K, so why do I see NP3 threshold counters increasing, showing me that I am overloading Network Processor 3 that handles connection establishments?". For someone to be able to answer that question he would need to keep in mind that the rate that is mentioned in the specifications is the average rate over one full second. To explain it better, here are a few examples:
So, it is obvious that bursts of traffic or connections could affect the performance of a firewall even if the averages over time does not seem to exceed the limits.
Additionally, having few connections through the box does not necessarily mean that traffic is not high. Theoretically speaking, someone could have 10 connections passing 1Gbps each and thus oversubscribing an FWSM with very few conns.
Now, it is equally important to mention options for overcoming an oversubscription issue. We would suggest to the reader to keep in mind that if a device is oversubscribed it is usually best to add more processing power by using more or more powerful devices. Though, there might be cases where we could get away with it by implementing some workarounds after identifying the root cause and the traffic profiles. Determining causes of oversubscription/excessive load should rely on external tools and traffic analysis.
When the CPU is high, we can try to see where it is spent and then we might be able to alleviate it from the process that takea most CPU cycles. We can collect the output of the "show process" command, wait for 1 minute and collect it once more.
ASA# show process
PC SP STATE Runtime SBASE Stack Process
Lwe 0805510c d52a0cf4 09fbeed8 0 d529edf0 7544/8192 block_diag
Mrd 081beaa4 d52d087c 09fbe438 873 d52b0a38 123848/131072 Dispatch Unit
Msi 08f6348f d5784f8c 09fbde4c 13 d5783088 7792/8192 y88acs06 OneSec Thread
Mwe 08068bc6 d578938c 09fbde4c 0 d57874e8 7576/8192 Reload Control Thread
Mwe 08070976 d5794314 09fc07f8 0 d5790760 12496/16384 aaa
Mwe 08d094ed d60111ec 09fbde4c 4 d57948e8 6872/8192 UserFromCert Thread
Mwe 08c331eb d57987f4 d57d47d0 0 d5796a70 6920/8192 Boot Message Proxy Process
Mwe 080a49f6 d579d37c 09fc0854 107 d5799488 8968/16384 CMGR Server Process
Mwe 080a4f05 d579f4a4 09fbde4c 20 d579d610 7696/8192 CMGR Timer Process
Lwe 081bdecc d57a8b9c 09fceba8 0 d57a6c98 7216/8192 dbgtrace
Mwe 08498525 d57b11c4 09fbde4c 172 d57af440 4712/8192 eswilp_svi_init
Msi 0861af45 d57c4734 09fbde4c 28 d57c2850 6952/8192 MUS Timeout Check Thread
Mwe 08d094ed d5a3845c 09fbde4c 0 d57cb0e0 7016/8192 netfs_thread_init
Mwe 09378625 d57d952c 09fbde4c 0 d57d76d8 7612/8192 Chunk Manager
Mwe 08932ea4 d57eadfc 09ebdb4c 0 d57e8ef8 7904/8192 IP Address Assign
Mwe 089c501f d597faa4 09ebebd0 0 d597dba0 7904/8192 Client Update Task
Lwe 093c1dba d5984404 09fbde4c 685 d5980570 15888/16384 Checkheaps
Mwe 08b9e1f2 d5994bf4 09fbde4c 1 d598cd80 31888/32768 Session Manager
Mwe 08cb45b5 d599aae4 d7cbd3b0 4 d5997090 14312/16384 uauth
Mwe 08c52475 d599d11c 09f0f884 0 d599b218 7376/8192 Uauth_Proxy
Msp 08c893ce d59a35b4 09fbde4c 2 d59a16b0 7792/8192 SSL
Mwe 08cb1f46 d59a5754 09f15434 0 d59a3870 7272/8192 SMTP
Mwe 08caac96 d59a98dc 09f15398 30 d59a59f8 15096/16384 Logger
Mwe 08cab4c5 d59ab9f4 09fbde4c 0 d59a9b80 7728/8192 Syslog Retry Thread
Mwe 08ca511e d59adb9c 09fbde4c 0 d59abd08 7192/8192 Thread Logger
Mwe 08e9c492 d59d83a4 09f492e8 0 d59d64c0 7040/8192 vpnlb_thread
...
Then he can do the diff of the "Runtime" column for all the processes (keep in mind that a process might show up twice or more). By sorting the diffs from maximum to minimum we can see the processes that take most of the CPU. There are cases where for example we might see an inspection process or the logging process taking most of the CPU. In such cases we can disable the inspections if they are not needed or turn down the logging level and save some CPU for the device. Please note that processes like "Dispatch_Unit" and "interface polling" relate to regular packet processing and there is not much that can be done to alleviate the CPU from them.
If the traffic hitting the firewall is excessive, we can also try to send only necessary traffic through it. Although, this solution is not practical in most setups, there might be cases where someone has alternate routes for his traffic and he might not need to "firewall" all packets. In such scenarios he can use policy based routing (PBR) to divert to the firewall only traffic that needs to be "firewalled".
Especially for the FWSM, someone MIGHT be able to change the load balancing on the backplane of the switch for the 6-Gig port-channel between the FWSM and the switch, in order to hash the packets equally to both Network processors. Though, this is a viable solution only if one of the NP1,2 is oversusbscribed and not the other. We would suggest opening a TAC case to investigate such a solution if only one NP (1 or 2) is increasing its threshold counters..
In case of using two firewalls in failover in Active/Standby mode, if the Active Unit cannot handle the traffic you might be able to temporarily use an Active/Active setup to share it between both units. You would need to have the firewalls in multi-context mode and have one or more contexts active on the primary unit and one or more contexts active on the secondary. That way both firewalls will be passing traffic (for the context/s that they are active) and might not be oversubscribed. Though, you need to remember that in case one of a units failure, all contexts (thus all traffic) will be running on one unit and then you will be back to an oversubscribed scenario. Active/Active failover for oversubscription cases should only be used (if used at all) as a temporary solution with precaution, until a permanent solution is put in place.
Finally, the ultimate solution would be for someone to add more hardware to his network or use more firewalls. That way he could divert traffic that the device/s can handle and there would be no oversubscription.
Hi,
This document describes how to get the FWSM statistics from the command line interface with commands like
sh np block
or
show traffic
Is there an equivalent method to collect these statistics over time on a console, with SNMP and a mib for instance.
Thanks
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: