Cisco Interrupt Level Switching

[Umesh Shetty]

Hi Friends,

I needed some information on Interrupt level switching. We have recently had a case in our organization where the WAN bandwidth on a Cisco 2801 router was upgraded and which probably overwhelmed the bandwidth that the router could suport (46 Mbps). Post the upgraded the CPU process of the router would go high upto 90% during production hours. There was no process on the router which showed up using the high resources and it was apprently the interrupt switching that was driving the utilization high. We had CEF enabled on all interfaces on the router as well.

1) I want to know what exactly interrupt level switching is ?

2) Why is the packet interrupt swicthed when there is CEF enabled?

3) Cisco documents say that "Interrupt-level switching means that when a packet arrives, an interrupt is triggered which causes the CPU to postpone other     tasks in order to handle that packet.At times when there is heavy traffic load will this not hamper other processes ?

Regards

Umesh Shetty

[Bilal Nawaz · Accepted solution]

Hello, please review these docs for more info.

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a7306.shtml

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

Hope this helps

Sent from Cisco Technical Support iPhone App

[ravikantt]

Hi Umesh,

Is it possible to share Show Processes output of your cisco 2801 router?

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

#1 Fast path packet forwarding.

#2 CEF is a fast path technique.

#3 Not necessarily.  Exactly how Cisco forwards their packets would be proprietary, but generally real interrupt handlers service the interrupt as quickly as possible and often hardware can prioritize interrupts.

PS:

A 2801 doesn't really have the performance to guarantee it can process 46 Mbps.

[Umesh Shetty]

What I have Understood till now :

From these documents what i could learn is there are two main methods of switchng packets At the Process level orAt the Interrupt Level.

Process Level : creates an IP Input_Process and gets schedules as per the CPU scheduling algorithm and it directly does a routing table lookup to idetify the next hop and the longest match and then the arp cahe to identify the mac address rewrite information of the next hop.

Since it has to

1) Wait for the scheduler to run the IP_INPUT process and 

2) Has to lookup the entire routing table and the arp cahe for packet routing it is more CPU intensive and a slower way of switching packets.

Interrupt Level : When there is an packet at the input interface and interrupt is generated to interrup the currently running process and the packet is swicthed.

It is faster becasue

1) It dies not have to wait for the scheduler

2) Interrupt switching does not consult the routing and the arp cahe table directly but has some form of easily available       caching that it uses to route the packet out of the output interface.

3) What kind of the cahing it uses depend on what kind of interrupt switching is used.

   a) Fast Switching

   b) Optimum Switching 

   c) Cisco Express Forwarding

So if the CPU on my router goes high and the router reports high util due to interrupts I can assume that although it is due to packet switching the switching method on the router is either Fast, Optimum or CEF switching and the only reason why the CPU is high is because the traffic rate is reaching the routers limit (incase of a 2801 46 Mbps without any features and could be lower if other features are enabled.). The only way to resolve this is either ofload the router a little bit by stoppping some CPU intensive features such a encryption  , shaping , tunnel etc.

In my case tac finally confirmed that the amount of traffic passing through the router and the features enabled (QOS, Shaping , Queuing , GER with IPsec) are too much for a 2801 to handle and a hardware upgrade was suggested.

The one Query that I have is  :

Assuming If the CPU during this time when a new packet arrives is only 30 % utilized why can't the router use the the remaining cou cycles to switch the packet rather than interrupt a process which already running ?

Regards

Umesh Shetty

[Joseph W. Doherty · Accepted solution]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

The one Query that I have is  :
Assuming If the CPU during this time when a new packet arrives is only 30 % utilized why can't the router use the the remaining cou cycles to switch the packet rather than interrupt a process which already running ?

Whether the packet is "interrupt" switched or "process" switched, arrival of any packet probably generates some form of hardware interrupt.  As subsequent packets may arrive, and hardware generally has fixed resources to how many packets may be held by the harware, any other process that's currently active is often interrupted to clear the hardware resources.  This also generally happens regardless how busy the CPU is.

Once the hardware interrupt is serviced, packet processing also generally takes a fixed amount of time for every packet (the inverse of the packets-per-second specification) so again it doesn't matter what the total CPU loading is (well until you totally use all of it).

[Umesh Shetty]

Hi Joseph,

What I've understood is that on L3 Routers (not multilayer switches) which do not have dedicated hardware for switching packets like a PFC card on a 6500,  for every packet the CPU has to be interrupted. The CPU then will check if there is a database available like a fast switching cache or an IP CEF FIB. If it is available there will be no IP_INPUT process invoked and the packet will be forwarded using the information from the fast/CEF switching database.

If there is no scuh data available the IP_INPUT process will be invoked and the packet will be sceduled, now because it is a slow process which happens as per the CPU clock speed the overall throughput of the device reduces considerably and the CPU will be utilized even at comparatively lesser data rates.

If my understanding is right on platforms such as 6500, ASR , 7600 and 7200 there is no interrupt generated for normal transit packet(packets not needing special treatment) becasue there is dedicated hardware for Switching packets like the PFC incase of a 6500 and an ESP incase of a ASR router and hence the throughput of these devices is higher.

I'll be happy if you can please share your views and correct me if I am wrong !!!

Thanks in Advance !!!

Regards

Umesh Shetty

[Joseph W. Doherty · Accepted solution]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

Only Cisco could truly explain interrelationship between hardware and packet forwarding on any of their platforms, and they're unlikely to do so publicly.

This also means we really don't know when or what interrupts are generated for packet forwarding under different conditions on different platforms.

Even "seeing" an IP process, that shows non-fast-path packet forwarding can be deceptive, as we "see" what Cisco's IOS wants us to see.  I.e. "fast path" or interrupt forwarded packets might still be scheduled too, just shown to us "hidden" and as "interrupt" CPU.

The only thing we sort of know, on any platform, some packet forwarding can be done very quickly, and some not so quickly.  As there's often a huge differences between quickly and not so quickly, it's nice if the vendor's OS can give us some indication what kind of forwarding a platform is doing because sometimes we can address the causes of not so quickly (e.g. packet fragmentation).