Has anyone seen issues on Cisco ISR / ISR G2 routers running 15.x code while doing NAT for large quantities of users (3000+) involving lock ups? This seems to be an ongoing issue for a customer. The symtom is present on 2811, 2911 and a 3945. The 3945 never spikes over 10% processor or memory utilization, yet still has these issues. The routers have been tested running 15.0(1)M5, 15.0(1)M7 and 15.1(4)M4 which all showed simular results. Other router side services are running such as NetFlow v9 to two destinations. Another interesting symtom, doing a "show ip nat translations" command will crash the router if executed at its peak usage period. NAT is to a single IP (sourced from outside interface), not a pool.
HOSTNAME#show mem alloc total
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 1231A16C 753819284 91719528 662099756 647785156 629474108
I/O 3C200000 65011712 14865036 50146676 50146676 50145724
HOSTNAME#show mem debug leak chunk
Adding blocks for GD...
I/O memory
Address Size Alloc_pc PID Alloc-Proc Name
Chunk Elements:
AllocPC Address Size Parent Name
Processor memory
Address Size Alloc_pc PID Alloc-Proc Name
Chunk Elements:
AllocPC Address Size Parent Name
NA 20A597C 8 13CF9CD4 (IPnat RAS appl )
NA 20A5984 8 13CF9CD4 (IPnat RAS appl )
NA 20A5994 8 13CF9CD4 (IPnat RAS appl )
NA 20A59A4 8 13CF9CD4 (IPnat RAS appl )
NA 20A59AC 8 13CF9CD4 (IPnat RAS appl )
5E193EC 148378C0 12 145E39C0 (MallocLite)
5E193EC 148378DC 12 145E39C0 (MallocLite)
5E193C8 1483794C 12 145E39C0 (MallocLite)
5E193EC 14837968 12 145E39C0 (MallocLite)
5E193C8 148379A0 12 145E39C0 (MallocLite)
5E193EC 148379BC 12 145E39C0 (MallocLite)
5E193C8 1483E738 12 145E39C0 (MallocLite)
5E193C8 1483E770 12 145E39C0 (MallocLite)
5E193EC 1483E7C4 12 145E39C0 (MallocLite)
5E193C8 1483E818 12 145E39C0 (MallocLite)
