Network Progressively degrading

[jmichel1976]

Ok, here is a problem that has me absolutely stumped, hopefully I'll find an answer here among the geniuses.  about 2 weeks ago, got a call that the network has come down to a crawl, thought odd, people couldn't ping by name, or by ip sometimes and sometimes vice versa,  found a faulty dns server, i upgraded all of our dell 6248 switch stacks to newest IOS, hoping it was a problem with them.  we currently ahve about 8 vlans on our LAN for about 300 users protected by a 2821 ISR.  a couple days later, things slow down to a crawl, ping times are up in the 600ms range. reboot the router and everything is back to >1ms. It started doing it more frequently, so i was thinking maybe leaky memory on the router. so i took the flash with the 12.4(19b) IOS and put it in a backup 2821 that had 12.3.. Copied the config and viola, everything was merry again, this was yesterday. got a call taht everything was slow again tonight. what's odd is, if i ping on the same vlan, everything is >1ms, but as soon as i try to  ping to a different vlan i get the high latency.  trying to telnet from outside into the router and vpn is brutally slow.  I'm kinda at a loss where to look now.  Any ideas out there?

[m.kafka]

what's the show cpu...? anything unusual?

positively enabled cef on the layer 3 functionality? on all interfaces?

did you look for release notes on the IOS in use? maybe Limited Deloyment? do you need that specific IOS or could you use a general deployment?

pinging on the same vlan is layer 2, 28xx is handling layer 3?

maybe whip up a simple network-sketch so we have a better idea, how things are connected on your network

Rgds,

MiKa

[jmichel1976]

Not sure on cef, how would i check.  basically we have 3 floors all connected by fiber, in the main datacenter we have 7 dell 6248 connected in a stack, the other 2 floors have 1 or 2 dell 6248 connected by fiber to a 6224f in the stack.  Maybe attaching a config might help a bit, i'm attaching a config. see if that helps.

[dominic.caron]

A few thing to check next time you have a degradation:

Router CPU - If high, check broadcast and multicast level on the interfaces, try to figure out what trafic is impacting the router.

State of the ARP table...is there instability?

It's probably something generated by one host on your network.

[m.kafka]

Hi,

the config is unfortunately not enough, Like I mentioned: schow cpu, sh mem, sh ip cef and similar commands would give some insight about what is going on.

Which wccp dou you use? everything ok on that part?

How high is the user traffic? maybe too much for the router?

sh interface will tell you buffer underruns or overruns or dropped packets

there are many things to veryfy, maybe start with my suggestions

Rgds,

MiKa