Could not execute sh run command in my router

ahvn · ‎08-27-2003

Dear all,

Could any one help me out to find why I could not get the sh run command working in my router.I wa able to view my runnning conf with that command till, suddenly its not working.I know I can use sh conf, its working too.

But some error mssgs are also coming when I use sh run, I am pasting the error mssg here.Anyone who have similiar experience kindly help me out.

The error mssg is below,

4d17h: %SYS-2-MALLOCFAIL: Memory allocation of 129016 bytes failed from 0x602E38

E8, alignment 0

Pool: Processor Free: 2414612 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Exec", ipl= 0, pid= 49

-Traceback= 6036BBF0 6036DB10 602E38F0 6030A188 6030A258 6030045C 6030CE24 60360

C04 60360BF0

Thanks in advance.

Homin

darylanderson · ‎08-27-2003

I'm sure I had this problem last week when my network was in the grip of the nachi virus. Try using the "clear arp" command. The try "sh run". If it doesn't work then it's not the same problem I had.

deilert · ‎08-27-2003

Your processor memory is fragmented , you could have a memory leak on the router , you need to reload the box to remedy this , it may be functioning ok now but soon there will be no memory left on the box, arrange for down time and reload .

ruwhite · ‎08-27-2003

That's odd.... Normally, when the router runs out of memory to run a show command, it will show memory instead! :-) Could you post a copy of your show memory here? You're either fragmented, or just out of memory.

Russ.W

ahvn · ‎08-27-2003

Hi Russ,

Below is the ouput of my router's sh memmory,

Router#sh memory

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 61600FE0 10481696 9736896 744800 64072 93640

I/O 40000000 4194304 2515396 1678908 1632180 1677472

I have implemented PBR through the null 0 interface to contain the dreadfull nachi-worm.This problem was seen after that.Thousands of packets are now discarded to the null 0 interface.Could this be a problem because of that. The CPU is utilisation is around 50% and memory utilization is sometimes close to 95%.Hope this problem will be solved once I upgrade my router memory.My router is 4500 series and I have 32 MB main memory and 4 MB shared Memory.I plan to upgrade that with 64 MB main and 16 MB shared.

Kindly help me out.

Rgds,

Homin

ruwhite · ‎08-28-2003

You're fragmented, not out of memory.... It's odd that this would happen after you run a route to null0. Are you running CEF switching on this box? Hmmm... But you're running PBR, and I don't know if we will do the fast discards with PBR, in the fast (interrupt) path.

What we need to do is figure out why you're fragmenting, but that takes a good bit of work, actually. :-( You'll need to look at show proc mem, and see which processes have a lot of alloc's and free's, and then, from there, peruse through show mem alloc or show mem summ, and figure out if the processes with high numbers of allocs and frees are, in fact, holding a lot of small pieces of memory scattered all over the place. My guess is the problem is going to be in ip input, if you just started seeing this with a change in pbr, but, if cef is running, and cef is fast discarding this traffic, that really shouldn't be happening.

Is there anything else on the box which is fluctuating constantly? A routing protocol with constant flaps, full bgp tables with a peer that's flapping, or something else?

At this point, other than troubleshoot it, the only choice is to reload the box. There's no way to bring a router out of fragmentation other than reloading it, other than possibly waiting long enough for it to defragment on its own, which isn't likely to happen without some change in the network conditions.

I think I'd reload it, then monitor the box for a while, a couple of times a day, and watch for developing fragmentation. If you see the box starting to fragment again, then open a tac case, or try to peruse the memory show commands and figure out what the culprit is.

Russ.W

riw@cisco.com

sachinraja · ‎08-28-2003

homin, try blocking icmp on our router. This might actually be a nachi.. apply ip route-cache flow on the ethernet and show us the result of sh ip cache. there might be an attack on port 0800 (icmp type 8 packet) or http or tftp etc...

ahvn · ‎08-28-2003

Sachin,

Instead of blocking ICMP I have appliead PBR for nachi-worm and also put ACL for blaster prone ports.As russ said, as the packets are being discarded to the null interface its taking a lot of toll on the router memory.Attack on my router was huge around 500K traffic more than what my links use to have.Now the traffic has normalised but router is being affected.

Russ , I have enabled cef and also disabled ip route-cache from my interfaces.

I have not reloaded the router still and is scheduled for appropriate time.Now I am getting my sh run output after I cleared the arp-cache , but some times it does not come.Or you all suggest me to block ICMP rather than discarding those packets..??

Kindly help me out.

Rgds,

Homin

sachinraja · ‎08-28-2003

Homin,

If it is an attack on port 0800 I would close my eyes and do a "deny icmp any any" to be really safe. I had also tried PBR, but that reduced the CPU only upto 50 % . Have you identified the PC's generating the virus ?

ruwhite · ‎08-29-2003

I would suggest what Sachin is suggesting--block the packets, rather than PBR routing them to null0. CEF is going to serve you better in the long run, it will discard these packets more quickly with less impact on memory.

In the meantime, your only other thing is to wait 'til you can reload the router.

Russ.W