Cisco 6509 Issues

[tmelino01]

I have a quick question or not so quick. We have been having network issues for the past week or so. This all started around the time we had an attack from a hacker which overwhelmed our catylist ( Cisco 6509 ) with a suprevisor 720. We have one IT person on the staff and we subcontract through another IT company. Neither of them seem to be able to  figure it out. We have ~75 machines that can download data from the outside ( ~3 Mb/s ). Then we have  ~50 machines that start to download OK ( ~3 Mb/s )  but then drop to speeds of 10 kb/s then just hang at that speed forever. Both sets of machines also have the same external IP address. This problem seems to only occur when connecting with one or two servers on the outside (http://nomads.ncep.noaa.gov). It seems to be that they are being throttled in some way but the provider insists that they are not throttling us and have checked. I believe them since some internal machines work.  Does it make sense for the issue to be with the server that we are connecting to since some machines work and others don't?  I have also contacted the ISP (TW Telecom) who checked the paths and said everything seemed to be ok on their end. Do you have any ideas what could cause something like this? It really doesn't make much sense at this point. Is it possible some unseen switch got thrown on the Catalyst that we can't see when it is being overwhelmed to save the network? and how would we correct it if it did?

Thanks in Advance!!!

Tim

[james.doukas]

Hi Tim,

This is a fairly broad question without a lot of details on your setup. Some info based on below might help figure this out.

1. Are the users directly connected to the 6509 or are they on different switches connected to the 6509.

2. Does the 6509 handle all the layer3 routing?

3. Whats the commited rate for your Internet connection? Can it burst? Is it shared with other departments?

4. What type of firewall is sitting between your users and the Internet connection.

5. Are all the users on the same subnet or are they split up?

6. Are the one's having the issue within the same subnet as those that work or are they seperated?

7. Have you tried testing from the pc's with the issue to different download sites besides the one above?

8. You mentioned a hacker attack. Was this some sort of virus? Were the pc's affected?

There are a lot of unknowns here.

[tmelino01]

1. The machines are connected in several ways some of them go through additional switches while others are connected directly. Neither configuration seems to make a difference. Some machines work and others don't.

2. Not sure I'll have to check

3. We have a 100 mb/s connection with around 50% utilization at the high end we are usually around 30%.

4. I am checking on the firewall type.

5-6. There are around 6 or 7 different subnets with each having machines that work and some that don't.

7. We have tried many other places all seem to work. Even the ftp connection seems to be OK to the server I mentioned before. It only seems to be on port 80.

8. The hacker broke into one of our external machines. From there he launched attacks on servers outside of our network. To our knowledge he only got to the outside servers and not on any of the now slow machines. While the attack was going on it overwhelmed our network.

[james.doukas]

Only port 80....hmmm...

Would you happen to know if you have a proxy server in-path or via a pac file or wccp?

[tmelino01]

I just checked we have a Cisco 5510 ASA in place right now, and the 6509 does all the 3 layer routing. There should be no proxy setup on the network. I have done traceroutes on most of the internal machines and get consistent results with just 2 hops before the outside. One to the catalyst and another to the ASA.

[james.doukas]

Well, the fact that only port 80 is affected seems to indicate more of a service type issue then a network type issue. You also mentioned it only seems to occur when you are connecting to "one or two servers on the outside". Have you contacted the remote agency hosting those devices? Have you verified that the machines with the issue aren't infected with some sort of malware? If you go to another website to download a file from one of the machines affected, does it perform ok?

[tmelino01]

Yes we have checked the machines they all look ok. They are all Linux servers running Centos. We have been in touch with the data providers but they seem to insist that the problems are on our end and have pretty much stopped responding to questions. According to them they have searched through all of their logs and they see nothing out of the ordinary. I have also tested these downloads at my house and a couple other places with no issues.  The provider said they do not throttle anyone when using the http service, but limit to 120 connections when using FTP. What type of service type issue would make one machine work and another not both on the same subnet. We ordered 2 or three new servers in the last week and none of them work , but if I bring the machine to my house it works fine.

[james.doukas]

At this point you may want to try to capture a session from one of the problem systems going to the problem destination. It seems very odd that it only affects some of your internal systems going to only one or two external destinations.

Are your internal hosts automatically connecting to the remote system or are you only seeing this when manually connecting? Will one system be fine while the other is slow? Also, do you know if the external destination is using some sort of load balancing? 

One thing that can slow data transfers down is tcp zero window. However, this usually only happens when the system in question is very busy.

[tmelino01]

The machines that this is running on is not busy at all they are doing very little. We have had to switch all of our operational data ingestion to one of our working systems. Actually my desktop system has issues even loading the web page on the server http://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2013091306/ . My machine will sit there and eventfully just time out with an unknown network error. This happens on one of the windows systems as well. The problem happens if I connect manually or automatically to the server. I do know that they are load balancing on their end with two separate servers. I can start 2 separate instances of my data grab code on a working machine and a broken machine at the same time, and they always behave the same way. The slow machines are slow and the fast machines are fast. Is it possible some switch got thrown on the catalyst when the network was bogged down to slow traffic to certain places to maintain the integrity of the network?

[james.doukas]

For the tcp zero window, I was thinking the remote host might be sending you the "lower your window size" since it may be the busy one.

If some switch got put onto the network, it wouldn't affect just http to a single site. Most network wide problems would affect all traffic, since switches only deal with layer2 and layer3 (if they are layer3 capable). If you pull some data from one of the normally slow machines from a different site, how does it perform? Also, where would someone put this switch if they were trying to resolve the issue? If you have physical access to your network devices, you could check your physical connections. Have you tried moving one of the problem servers to a switch where another server is working? Once again, I don't know your full setup.

[tmelino01]

Most of the servers are plugged into a patch panel then directly into the catalyst with very few switches in the way except for the Desktop machines which are on 2 separate switches. The catalyst is then plugged into the ASA. There is only one server room at this location where we are having a problem. We have an offsite collocation facility where there are no issues whatsoever. If we pull data from any other server we see normal speeds even on the slow machines! We have tried moving slow machines from a known slow port in the catalyst to another where a machine was fast but it made no difference. The only place to put another switch would be to bring down the eniter network and put a switch before the ASA and bypass everything. We are trying to avoid network disruptions as much as possivle. In your opinion would it be a good idea to reboot the catalyst? Is there any tools or software you can use to diagnose where the slowdown is occuring?

[james.doukas]

From everything you've metioned, I don't think the issue is with your Catalyst switch. Does each server pull the same data? Perhaps the issue could be with what you're actually downloading. Do you have automated scripts? Could a server that is working ok be switched to downloading the content one of the problem servers is having?

[tmelino01]

Yes each server pulls the same data but it is in a primary/backup configuration. So if one machine is down the other automatically takes over. We started pulling the problem datasets on one of the working machines with no problems. I also want to let you know that these broken machines used to work in the past but then suddenly stopped. The problem is neither of our ingesion machines work so we had to place this download burden on a machine that wasn't meant for this. The scripts are all automated and use wget to pull the files. My concern is that it is internal to us. I would expect that if it was an issue on the providers end that I would have problems at my house and other places that use this data would be having issues too.

[james.doukas]

So you're saying you can take one of these problem systems to a different location and it works ok or you have simply tried the same process but havne't actually tested it with one of the problem systems?

[tmelino01]

I physical took one of the machines to an outside network ( to my house)  and it worked fine.As another test we then switched the ports of a bad machine and a good machine. We concluded that it didn't matter the port in the catalyst since the machines still acted the same way.