cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1259
Views
5
Helpful
10
Replies

High Interrupt on NPE-G2

bennyng11
Level 1
Level 1

Hello

I'm using a pair of cisco 7200 with NPE-G2 router for my VoIP operation. I'm facing a high interrupt situation where running around total 100k Ingress packet per second, but running at around 55% Interrupt CPU, and when have 250k Ingress PPS, it is running 90% interrupt CPU. Since the performance is significantly lower than 2Mpps, it does not looks right on my router, and I would like to get some advice from the expert here.

Here is the features that i have

BGP4 IPv4 - 8 Peers, 2 peers sent full route

BGP4 IPv6 - 7 Peers, 2 peers sent full route

VRRP

Netflow

~15 sub-interfaces

No Service Policy

No ACL

I had done some research on the internet, and follow the instruction on the troubleshooting guide

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml, and don't spot anything wrong. All packet is running on CEF.

Attached the show tech and CPU profile.

Thanks in advance.

Benny

10 Replies 10

Hi Benny,

Just couple of questions:

1. What is the total bandwidth terminating / used on the box ?

2. Is IP CEF enabled ? if not, enable and check; it will surely decrease utilization comparatively.

3. Since, you have netflow enabled, definately it will cost couple of CPU cycles, suggestion will be that disable unwanted netflows.

Regards,

Smitesh

Dear Smitesh

1. The box is switching around 60Mb traffic @ peak.

2. IP CEF is enabled

3. The netflow consumed around 5% cpu @ peak, as i just turn on the netflow on 4 uplinks.

Thanks

Benny

kitlim888
Level 1
Level 1

Dear Benny,

I am facing the same problem... have you found a solution yet?

Rgs

Kit

Hello Kit

Kinda resolved.

We let go for all Port Adapter, only use the based board GE interface, and using 12.2SRE6 Service Provider version. + using complied ACL (although we don't have any ACL, but someone suggested)

Now the CPU down to 56% CPU when 180kpps. closer to what TAC had tested

https://puck.nether.net/pipermail/cisco-nsp/2007-April/040171.html

I think the key is not using any Port Adapter and change the IOS to 12.2SRE, i did some test where the CPU can go up to 70% with the Port Adapter (2FE) when doing 150kpps.

Best Regards

Benny

Hi Benny,

Thanks a million! We will try and let you know what's our result.

Best Regards

Kit

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Benny interesting results.  Both expected and unexpected.

If you search these forums, I believe you'll find other complaints that -G2 performance isn't always what's expected compared to earlier NPEs especially the -G1, i.e. 2x the Mpps.

The older performance reference sheet, http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf, notes performance is based on actual testing, but we don't known what interfaces were actually used for such testing.

We often assume all interfaces are alike, but as your information shows, there might be much more overhead attached to a PA Ethernet interface than an on-board Ethernet interface.  If there is, then it's not unexpected performance would slow.

The reference link you provided also shows how performance can be unexpected and isn't always nicely linear.  For example, the beginning of the testing shows a -G2 with a higher CPU usage than a -G1 for the same traffic PPS rate.  Or, on the -G2, 650 Kbpps until 750 Kpps all show 99% usage.

A month or so ago, I tried to explain on another posting, performance can be quite variable because there are so many variables involved.  Another poster was adamant that since he got a certain CPU usage on his device at a certain traffic rate, everyone else should too, but my point was unless it's exactly the same traffic, with exactly the same device config, you really cannot claim that.  Basically your "mileage may vary".

Hi Joseph,

In my case, we are using 7206VXR NPE-G2 with PA-GE (port adapter) on IOS 12.4 (24) T6 with only 1 upstream running at default route BGP (not full routes). At about 160k pps the cpu is already 95-100% with less than 1% processes using cpu (mostly interrupts), with packet lossess happening.

As we are using this for VOIP the packet size is small (around 70 bytes/ packet), it is close to the 64 byte packets size as mentioned in the reference sheet.

At 160k pps both ways it is just too far away from the 1M pps (both ways or 2Mpps one way) as published by Cisco.

We are scheduling next Monday to redo the fibre termination to move from the PA-GE to the Gig E port on the main board before i change the IOS version. Also I wonder whether there is any problem with the buffers on my router. Details as below.

Appreciate any guidance/ideas on this issue. Thx in advance!

# show buffer
Buffer elements:
     1118 in free list (1119 max allowed)
     3518637367 hits, 0 misses, 619 created

Public buffer pools:
Small buffers, 104 bytes (total 101, permanent 50, peak 181 @ 1w0d):
     100 in free list (20 min, 150 max allowed)
     13322296 hits, 773 misses, 645 trims, 696 created
     151 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 153 @ 4d02h):
     22 in free list (10 min, 150 max allowed)
     4460521 hits, 353 misses, 656 trims, 656 created
     23 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50):
     50 in free list (5 min, 150 max allowed)
     3945565 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 11, permanent 10, peak 12 @ 3w3d):
     10 in free list (0 min, 100 max allowed)
     0 hits, 1 misses, 1 trims, 2 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
     0 in free list (0 min, 10 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
     0 in free list (0 min, 4 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Interface buffer pools:
Syslog ED Pool buffers, 600 bytes (total 282, permanent 282):
     250 in free list (282 min, 282 max allowed)
     644 hits, 0 misses
IPC buffers, 4096 bytes (total 2, permanent 2):
     2 in free list (1 min, 8 max allowed)
     0 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 511, permanent 256, peak 511 @ 3w3d):
     255 in free list (256 min, 1024 max allowed)
     171 hits, 85 misses, 0 trims, 255 created
     0 failures (0 no memory)
     256 max cache size, 256 in cache
     2178139146 hits in cache, 0 misses in cache

Particle Clones:
     1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 128 bytes (total 512, permanent 512):
     0 in free list (0 min, 512 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     0 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 2048, permanent 2048):
     2048 in free list (1024 min, 4096 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Private particle pools:
HQF buffers, 0 bytes (total 2000, permanent 2000):
     2000 in free list (500 min, 2000 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):
     0 in free list (0 min, 1000 max allowed)
     1000 hits, 0 fallbacks
     1000 max cache size, 619 in cache
     1977892200 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions
GigabitEthernet0/2 buffers, 512 bytes (total 1000, permanent 1000):
     0 in free list (0 min, 1000 max allowed)
     1000 hits, 0 fallbacks
     1000 max cache size, 872 in cache
     3052455021 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions
GigabitEthernet0/3 buffers, 512 bytes (total 1000, permanent 1000):
     0 in free list (0 min, 1000 max allowed)
     1000 hits, 0 fallbacks
     1000 max cache size, 872 in cache
     128 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions
FastEthernet0/2 buffers, 768 bytes (total 530, permanent 400):
     128 in free list (128 min, 1200 max allowed)
     359 hits, 43 fallbacks, 304 trims, 434 created
     0 failures (0 no memory)
     400 max cache size, 274 in cache
     2129934577 hits in cache, 2 misses in cache
     14 buffer threshold, 1 threshold transitions
GigabitEthernet3/0 buffers, 768 bytes (total 2000, permanent 2000):
     0 in free list (0 min, 2000 max allowed)
     2000 hits, 0 misses
     2000 max cache size, 1871 in cache
     1486379008 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions

Cisco published performances using the on-board ports, not the PA.

As you notices yourself above, with NPE-Gx the PAs are meant only for connectivity, not performance.

Hello Kit

what is the license you are using for your 7200? we was using advance ip and advance enterprise before. the performace was exactly what you shown. than we had changed to 12.2SRE6 service provider license the cpu loweb down to my level. you might give a try.

also i suspect the complied acl contribute a bit as well.

btw we are running voip as well.

benny

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Your buffer stats look reasonable, but if your IOS supports it, you might turn on auto buffer tuning.

What Benny discovered, a different IOS working better/faster, supposes that different (or more likely) newer IOSs might have changed how they do what they do for some critical (bottleneck) processing.  As the -G2 is a software based router, a more efficient algorithm, or taking advantage of instructions unique to the -G2, not available in -G1 and earlier NPEs, could improve performance.

Review Cisco Networking for a $25 gift card