All,
First a little background:
I had an issue last night that I was able to track down to at least 3 workstations. About 6:30PM, I started getting notifications from my monitoring server that all of my VPN sites were down, and then the big one came: my load balancer wasn't responding. I was able to connect to the vpn, which goes through the load balancer so that was strangeness #1. Long story short, I ran wireshark on a workstation that I remoted into because my core switch (stack of 3750s) processor usage was at 99%. The Mwheel (what process is this??) process was at 60%. In wireshark, I was seeing a server throwTONS of random multicast address pings. Someone I work with got on that server, and it had a notification of w32.spybot worm that our AV caught. He rebooted the server and all was well. The processor went down to 30% on the switch stack.
I then looked at my multicast routing table and filtered it by my local subnet "sh ip mroute | in 10.12" OMG....there were literally HUNDREDS of these random addresses from 3 computers (not including the one above). I remoted into another of these boxes and put wireshark on it. The AV found the virus, but before I rebooted I started a capture to see if it was doing the multicast ping and it was. After reboot, it stopped.
Now the question:
Since the switch was saturated, the monitoring server couldn't ping out consistently. The switch had a message in the log that stated something like, "A multicast storm was detected....." and "Recv queue starved...." (Not exact errors.) I could ping the last device that's in the chain before it hits the outside world, and I'd lose packets, BUT (here's where the weirdness comes in) I could ping PAST the device and not lose any traffic. So it looks like this:
host -> switch -> 192.168.1.5 -> Internet -> 4.2.2.1
Pinging from host would result:
Reply from 192.168.1.5
Reply
Request timed out
Request timed out
Reply
But pinging from the same host, at the same time, would result in replies:
Reply from 4.2.2.1
Reply from 4.2.2.1
Reply from 4.2.2.1
Reply from 4.2.2.1
Reply from 4.2.2.1
The only thing that I can figure is that the device was having to process the icmp packet when it would come through and it was too overwhelmed to generate a reply, whereas in the second scenario it was just needing to send it through. Am I on the right track? What is the mwheel process? I'm assuming that it's multicast related.
Thanks!
John
HTH,
John
*** Please rate all useful posts ***
