C6500/Sup720 - Understand link between CPU interrupt level and CEF (or dCEF)

[parisdooz12]

Hi,

I try to understand what packets do use the CPU at interrupt level.

I understand well what packets need to be processed level (in other words: software swiched), for example, packets destinated to the Sup720 itself.

Is this hardware switched packets which trigerred an interrupt on the CPU? Even if the packet is handled by ASIC, CPU still need to make a treatment?

What happens in case of card equipped with DFC: is the packet s 100% treated by a CPU on the card itself?

Thaks by advance by your help.

[Joseph W. Doherty]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

See if the section Cisco Catalyst 6500: Day in the Life of a Packet, at the bottom of: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html, helps any.

The prior doesn't mention interrupts, but generally interrupts are generated by hardware that need the attention of the CPU.  Often this attention is not per packet.

In Cisco routers, interrupt packet forwarding is a synonym for fast past forwarding in contrast to process forwarding.

[amabdelh]

It will depends on the ASIC, if it cant handle the packet it will be punted to the cpu, for example packets with IP options. Some other type of packets also need to be punted to the CPU like if ttl=1 or packets targeted to the CPU like routing protocols, ICMP.

Sent from Cisco Technical Support iPhone App

[InayathUlla Sharieff]

Hi,

1- Processes level vs Interrupt level

Processes level is the software switched.:

Proces switching means ARP/routing process/CDP process mainly IOS processes

Interupt level: CPU utlization due to interupt switching is caused due to packets which are not performed by the HW present on the data cards. Normal data pkts should not be processes towards the cpu.

2-

Is this hardware switched packets which trigerred an interrupt on the CPU? Even if the packet is handled by ASIC, CPU still need to make a treatment?

Once the Hardware table has been synchronized between the Sup SP card and DCEF cards on submodule then all the hardware forwarding will be direclty done by the module with dcef itself. The packets only will be punted to the cpu when there is no entry present on the cam/mac table entry.

3-

What happens in case of card equipped with DFC: is the packet s 100% treated by a CPU on the card itself?

I am not sure what you are looking for here. The L2 data will be pushed from the sup module to the DFC card hence all the layer 2 forwarding will be done direclty from the module itself. The packets will be punted to CPU only when it doesnt find any entry on the dfc card.

HTH

Regards

Inayath

[parisdooz12]

Hi,

thanks for the answers.

Here is the current CPU usage on the C6500

CPU utilization for five seconds: 24%/9%

According to these values:

- 15% (24-9) are due to packets processed (ex: ARP, TTL 1, IP Option, etc ...)

- 9% of CPU is due to interrupts

I understand that these 9% are due to entries not in DFC or CFC (some of our cards have no DFC), and so are punted to CPU. However, I am a bit surprised that so many entries (according to the CPU usage) are not in the DFC.

[InayathUlla Sharieff · Accepted solution]

Hi Pari,

Possible Causes of High CPU Utilization Due to Interrupts

High CPU utilization on an interrupt level is primarily caused by packets handled on interrupt level. Interrupts are generated any time a character is output from the console or auxiliary ports of a router.

Universal Asynchronous Receiver/Transmitters (UARTs) are slow compared to the processing speed of the router, so it is unlikely, though possible, that console or auxiliary interrupts can cause a high CPU utilization on the router (unless the router has a large number of tty lines in use).

There are several reasons for high CPU utilization due to interrupts:

• An inappropriate switching path is configured on the router
• The CPU is performing alignment corrections
• The router is overloaded with traffic
• There's a bug in the Cisco IOS® software running on the router
• Voice ports are configured on the router
• There are active Asynchronous Transfer Mode (ATM) interfaces on the router
• Too many packets are being punted from PXF to the Route Processor (RP)

Inappropriate Switching Path

To troubleshoot this potential issue, verify the following:

• Check whether or not the router is running Cisco Express Forwarding:
  • Verify the configuration for the ip cef global configuration command.
  • Verify that Cisco Express Forwarding is enabled and working by issuing the show ip cef summary command.
  • Verify that Cisco Express Forwarding is enabled as the switching path on all interfaces. You can see this in show cef interface and show ip interface output. If Cisco Express Forwarding is configured, but not enabled on an interface, this means that the interface encapsulation is not supported in Cisco Express Forwarding. Verify that Cisco Express Forwarding is operational, that is, check whether packets are really switched through the router using Cisco Express Forwarding by looking at the show cef not-cef-switched .
  • Using the show cef drop command and the show interfaces switching command (this is a hidden command you can use to look for cache misses), verify that Cisco Express Forwarding isn't dropping packets. If this is the case, see the CEF troubleshooting page.
• Verify if any of the interfaces have long access lists configured.
  • As a general rule of thumb, any access list with over ten lines is considered long.
  • Repeatedly going over long access lists is very CPU-intensive. With NetFlow switching, if the flow is already in the cache, you no longer need to check the access list. So in this case, NetFlow switching would be useful. You can enable NetFlow switching by issuing the ip route-cache flow command.
  • Note that if Cisco Express Forwarding and NetFlow are both configured on an interface, Cisco Express Forwarding will be used to make a switching decision.
• Verify that NetFlow switching is configured on the router:
  • Check the statistics by issuing the show ip cache flow command. Look at the number of new flows per second.
  • If Cisco Express Forwarding is not enabled, enable Cisco Express Forwarding to speed up the switching decision.
  • If there are no long access lists, try disabling NetFlow switching.

HTH

Regards

Inayath

*Plz rate all usefull posts.

[parisdooz12]

Thanks, exactly what I looked for!

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

CPU utilization for five seconds: 24%/9%
According to these values:
- 15% (24-9) are due to packets processed (ex: ARP, TTL 1, IP Option, etc ...)
- 9% of CPU is due to interrupts
I understand that these 9% are due to entries not in DFC or CFC (some of our cards have no DFC), and so are punted to CPU. However, I am a bit surprised that so many entries (according to the CPU usage) are not in the DFC.

I don't believe your interpretation is correct.

The 9% should represent CPU involved with normal packet/frame forwarding using the main sup.  Unlike a software based router, this CPU deals with the overhead of managing the hardware that performs the actual packet/frame forwarding.  Basically, this CPU usage should reflect the Mpps being used for packet/frame forwarding.

Your 9% should indicate you're using that percentage of the 15 or 30 Mpps (NB: 15 vs. 30 depends on bus mode) forwarding capacity of your sup720.

As you mention you have some cards with DFCs, they too have CPU stats (have you looked at them?), and whatever they do locally would be off-loaded from the main sup.

The 15% is the sum of all other CPU processes, including, but not limited to, software based forwarding.  I.e. some or all of this might be for packets "punted".  The show CPU proc will show where this CPU is being used.

Now that I understand, your question about understanding interrupts seem to be more about investigation of your sup's CPU stats, you might also want to review: http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml

Again, as you have DFCs, the DFC should off load some this kind of CPU loading.

[parisdooz12]

Hi Joseph,

thanks for these informations.

I indeed noticed the CPU on DFC cards. Look at one of the DFCs, here is the output of process using the most of CPU:

253 19045216 139994087 136 3.51% 1.32% 1.35% 0 fw_lcp process

283 6198900 693729 8935 3.35% 0.67% 0.50% 0 L2 MAC oob sync

245 22710796 743782 30534 1.59% 1.61% 1.61% 0 Vlan Statistics

308 2538192 1572974 1613 0.87% 0.35% 0.22% 0 XDR LC Backgroun

355 5427248 2215600 2449 0.39% 0.40% 0.39% 0 CEF: IPv4 proces

It seems that some processes are linked to DFC itself (L2 MAC oob sync, perhaps fw_lcp too), for synchronization. Other processes like CEF: IPv4 process seems indeed permit to offload CPU on the suprvisor.