SG300 switches have poor performance in Layer 3??

[stownsend]

We have several of the SG300 Serices switches. We use them to route VLAN traffic to Remote Offices, Internet Connections, and WiFi Access Points.

In one remote office we have a SG300-10 setup to route the HQ Network and the remote Office Subnet. The SG300 is Connected to HQ via Fiber and has multiple Tagged VLANs on it. If I do speed tests over the Fiber Link on the Incoming Tagged Netwotk I get Decent performance, 80Mbs. If I switch to a networtk that is not priginating from HQ, and have the SG300-10 route packet, I get dismal performance. 15-20Mbs.

I Fireded up a New SG300-28P FW v1.2.7.76. Added a the HQ VLAN 101 and new VLAN 1025 . Mapped some Tagged and untagged ports for each.  Switch was connected to HQ Network as untagged VLAN 101.  I put a laptop on an Untagged VLAN 101 port. Ran some tests, cam back with 750-850Mbs. Great.  Put the same laptop on a Tagged 101 Port, Configured the NIC for Tagged VLAN 101, Same test, same Speeds, 750-850Mbs.

I then  Configured laptop for Tagged VLAN 1025. Connected to tagged VLAN 1025 port. Ran speed tests, resuts were 15-20Mbs!

I then  Configured laptop for Untagged VLAN 1025. Connected to unagged VLAN 1025 port. Ran speed tests, resuts were 15-20Mbs!

It was only the Laptop and the Connection to the HQ net on the SG300-28P. Why is the performance of this unit soooooo poor when it needs to route?

Other Switches have FW v1.0.0.27 or FW v1.1.2.0. They have Similar speed issues. All Configured for Layer 3.

Thanks!

Scott<-

[Tom Watts]

Hi Scott, unfortunately there is a lot of information missing. What are you trying to access from vlan 1025? What vlan does the resource you're accessing reside? How are you measuring the speed? What are the distances for these links? Are you experiencing local route issues, meaning communication between vlans with connections originating on the same switch?

I've been working with these switches before they were on the market, I can't say I've ever run in to a throughput issue even with fiber to ethernet conversions.

-Tom
Please mark answered for helpful posts

[stownsend]

On the Test SG300-28P here in my Office I connect an Untagged VLAN 101 Port to our 10.1.x.x network.   I connect a Laptop up to another Untagged VLAN 101 10.1.x.x port.

I used an App called LAN Speed Test by Totusoft. It has a Client and server app that is used to measure network Speed.   I have the Server App installed on a server in our Datacenter.

When I run the Client on the laptop Connected to the untagged VLAN 101 port, I get Speeds in the 700-800Mbs range. That's close enought to 1Gbs for me for this test.

I then take the same laptop plug it into a untagged VLAN 1025 10.25.x.x network port.  Run the same speed test from the same latop and I get 15-20Mbs results.

the 10.25.x.x Subnet is only visable on the SG300-28P.

I Moved the Test App Server to another Machine that is connected to an UnTagged VLAN 101 port.   This is Interesting.

Running the test from the Laptop on the VLAN 1025 network. the First Test is 50-100Mbs and Subsequest Tests are all in the 700-800Mbs range. I wait a few minutes, then I get the 50-100Mbs test, then back up to the 700-800 range.

Both the new server and the old server are using the same Default Router IP as the SG300-28P.

Now it would seem like there is something in between my Test Server in the Datacenter and the Test Switch, that when a Packet is 'routed' there is a lag.  When testing to the Local to the Switch test Sever, the First attempt is getting the Lag as it needs to find the route, though subsequest tests already know where to go and are bypassing the equipment that has the lag?

Now I'm not really sure what the issue is...  )-:

Scott<-

[Tom Watts]

Can you run multiple sets of trace routes from both directions to see if the route hops match up correctly?

-Tom
Please mark answered for helpful posts

[stownsend]

I believe the Issue is that my Default Gateway IP is on a Cisco 2620 with a FE (10/100) port.

Makes a Little bit of sence, if the Packets need to be routed through the gateway, and the gateway is 10/100 vs Gigabit, Though I'd expect to see packets more in the 70-80 range, not the high teens.

We origonally had the Firewall as the Gateway IP though we had some issue and Cisco mention that it should not be the Gateway, we only had a 50Meg Internet Connection, so a 100base NIC should be fine.   We have since Upped the Branch Office Switches to Gigabit and Upped the Link from 3Meg VPN to 100Meg Fiber.

I've altered the Gateway Address on a few servers and test Clients and several of the SG300 Switches to one of the SG300 Switches at HQ. Then Duplicated all of the Routes in the old Gateway 10.1.0.1 to the SG300. 

The Speed Tests are Much Improved.

Remote Office Link

01-02-2013    16:47:57      10.2.0.7        10.9380480 Mbps    25.6109280 Mbps

01-03-2013    06:28:58      10.2.0.7        54.5029120 Mbps    65.4284640 Mbps

Test SG300-28P

01-02-2013    14:48:50     10.25.133.134    24.5234480 Mbps    9.6867360 Mbps

01-02-2013    14:49:03     10.25.133.134    17.4678720 Mbps    7.8186720 Mbps

01-02-2013    14:58:03     10.25.133.134    17.2096960 Mbps    8.9100800 Mbps

01-03-2013    06:20:29     10.25.133.134    751.2908080 Mbps   832.4246000 Mbps

01-03-2013    06:20:30     10.25.133.134    780.0182720 Mbps   172.0453040 Mbps

01-03-2013    06:20:32     10.25.133.134    776.5618080 Mbps   864.2902320 Mbps

Though now another Concern is the inconsistant Speed. If I do the Testing though several layers of Switchgear in the same subnet, its always in the 700-800 Range. Through the SG300's now and not using the Cisco 2620, I'm getting Occational Lags, Nothing consistant like using the 2620, and always above 100.  The Average Values below are misleading though. The Min is low 100s and the Highs are 800s, throw in a few 100's in a 10 payload series and is drops the Average pretty quick.   So in the 10 Payload series has extreams in it.

Two Test Servers in Datacenter behind a few switches, Client behind SG300-28p, Servers have GW Address of Production SG300-28P

01-03-2013    06:16:50      10.1.1.30        12.3712640 Mbps    42.2743360 Mbps

01-03-2013    06:17:48      10.1.1.30        248.7404720 Mbps   780.3152960 Mbps

01-03-2013    06:17:51      10.1.1.30        802.2313280 Mbps   625.9214720 Mbps

01-03-2013    06:17:54      10.1.1.30        822.3077280 Mbps   267.9503440 Mbps

01-03-2013    06:18:02      10.1.1.30        236.6436720 Mbps   388.5522800 Mbps

01-03-2013    06:18:06      10.1.1.30        215.9918640 Mbps   259.3957840 Mbps

01-03-2013    06:36:15      10.1.1.30        781.5636960 Mbps   416.5594480 Mbps

01-03-2013    06:36:19      10.1.1.30        841.7524320 Mbps   271.0857280 Mbps

01-03-2013    06:36:24      10.1.1.30        306.7223600 Mbps   789.3750160 Mbps

01-03-2013    06:36:29      10.1.1.30        843.7545840 Mbps   801.8500320 Mbps

01-03-2013    06:36:33      10.1.1.30        830.2770480 Mbps   344.9267200 Mbps

01-03-2013    06:36:38      10.1.1.30        221.9329120 Mbps   351.7301680 Mbps

01-03-2013    06:36:44      10.1.1.30        792.1214000 Mbps   461.3861840 Mbps

01-03-2013    06:36:48      10.1.1.30        313.2628960 Mbps   373.1908640 Mbps

01-03-2013    06:36:53      10.1.1.30        783.6339600 Mbps   805.2455280 Mbps

01-03-2013    06:36:57      10.1.1.30        807.4151360 Mbps   803.8262160 Mbps

01-03-2013    06:37:00      10.1.1.30        137.4077040 Mbps   799.7618320 Mbps

01-03-2013    06:37:06      10.1.1.30        165.5438320 Mbps   802.4233200 Mbps

01-03-2013    06:52:43      10.1.0.133       715.3523520 Mbps   770.3262640 Mbps

01-03-2013    06:52:44      10.1.0.133       810.1815520 Mbps   192.7391760 Mbps

01-03-2013    06:52:46      10.1.0.133       171.2564080 Mbps   729.6127920 Mbps

01-03-2013    06:52:48      10.1.0.133       782.3471200 Mbps   754.5255040 Mbps

01-03-2013    06:52:49      10.1.0.133       714.4515680 Mbps   713.9179200 Mbps

01-03-2013    06:52:50      10.1.0.133       810.3522480 Mbps   489.1850400 Mbps

01-03-2013    06:52:52      10.1.0.133       671.6649760 Mbps   743.7218240 Mbps

01-03-2013    06:52:53      10.1.0.133       813.7234480 Mbps   768.3375760 Mbps

01-03-2013    06:52:54      10.1.0.133       699.6499120 Mbps   761.7959360 Mbps

01-03-2013    06:52:55      10.1.0.133       833.3064240 Mbps   801.3984400 Mbps

01-03-2013    06:52:56      10.1.0.133       555.0648640 Mbps   56.9256400 Mbps

01-03-2013    06:53:00      10.1.0.133       556.3440960 Mbps   757.8441600 Mbps

01-03-2013    06:53:01      10.1.0.133       705.3840160 Mbps   746.0181280 Mbps

01-03-2013    06:53:02      10.1.0.133       743.9673520 Mbps   735.0482000 Mbps

01-03-2013    06:53:03      10.1.0.133       790.8021760 Mbps   763.0558880 Mbps

01-03-2013    06:53:04      10.1.0.133       812.5589120 Mbps   108.4269680 Mbps

01-03-2013    06:53:07      10.1.0.133       771.0048240 Mbps   773.7856880 Mbps

This set of results seems out of whack.

Test Server On UnTagged VLAN port on SG300-28P, Client behind SG300-28P

Test Server has GW Address of Test SG300-28P, So both Test Subjects are directly connected to the SG300-28P and both have GW Address of said SG300-28P

01-03-2013    06:58:36      10.1.0.133       819.4404080 Mbps   99.9245040 Mbps

01-03-2013    06:58:38      10.1.0.133       118.0072080 Mbps   25.8097920 Mbps

01-03-2013    06:58:43      10.1.0.133       25.2115200 Mbps    52.2479760 Mbps

01-03-2013    06:58:49      10.1.0.133       74.2421440 Mbps    99.3851760 Mbps

01-03-2013    06:58:54      10.1.0.133       53.9801040 Mbps    692.5357600 Mbps

01-03-2013    06:59:00      10.1.0.133       497.1383440 Mbps   629.9311840 Mbps

01-03-2013    06:59:03      10.1.0.133       75.6506400 Mbps    764.3419200 Mbps

01-03-2013    06:59:08      10.1.0.133       799.8776160 Mbps   53.1832880 Mbps

01-03-2013    06:59:38      10.1.0.133       86.6323920 Mbps    751.2520080 Mbps

01-03-2013    06:59:41      10.1.0.133       588.5810880 Mbps   711.6690560 Mbps

01-03-2013    06:59:43      10.1.0.133       67.9986560 Mbps    732.9853520 Mbps

We are getting better, but not out of the woods?

Thank you!

[Tom Watts · Accepted solution]

How are the client and server gateways configured?

The default gateway of the connecting devices probably should be that of the SVI they are directly connecting to. So if you have a server connecting to a SVI of 192.168.1.100 then the gateway of the server should be 192.168.1.100, not an upstream router. That may be creating a route loop if it has to back traffic.

-Tom
Please mark answered for helpful posts

[stownsend]

I have Programmed all of the SG300 Switches to have Static routes to all of the other Subnets with the shortest path to that subnet.

I have a SG300-28P setup with 10.1.x.x and 10.25.x.x Subnets.  Configured untagged ports for each.

Connected  Laptop on 10.25.x.x with its gateway being the SVI IP of the SG300-28P's 10.25.x.x interface

Connected  Desktop on 10.1.x.x with its gateway being the SVI IP of the SG300-28P's 10.1.x.x interface

Ran the tests and there was the same inconsistant test results.

The only time I get consistant results is when the two servers are on the same subnet.

The Performance increase so far by removing the two 2620 series 10/100 routers from the Mix has been huge.

My thought was that I wanted a Single place to have all of hte Routes to other subnets. Point everyone to that router so they know where to go.

From what I gather you are saying, Put the Routes for all of the subnets into every SG300 Switch and then point what ever is connected to that switch to that Switch's VLAN IP for the subnet it is on.

Since I have two Firewalls, I cannot use the one Switch as I had two Cisco 2620's doing the routing for the different firewalls.  One firewall is for incoming services (Web/Mail) the other is for External Clients. Though everyone is on the same internal LAN Subnet.  

Should I get another SG300 for the 5 servers that need to point to the 2nd firewall, or is there something else that I should be using?

Thank you!

  Scott<-

[Tom Watts]

The switch will build adjacent route through direct connection. You may use default routes to send traffic to a specific router/gateway if you want to sculpt traffic this way.

Example Host A 192.168.1.10 wants to go to host B 192.168.2.10

Host A is 192.168.1.10 connecting to SVI 192.168.1.100

There is 1 route hop connecting, on 192.168.2.200 on my example 2620

Host B is 192.168.2.10 connecting to SVI 192.168.2.100

The default gateway of either host should be that of the SVI.  Without any route, if you check a show ip route, you will notice there will be the DC route 192.168.1.0 and 192.168.2.0 dynamically built within the switch. So there wouldn't be a need to tell where to go since the network has the connected route and the local switch has the requested host locally.

Now let's mix it up some,  on the switch connecting Host A, you have a default route 0.0.0.0 192.168.2.200. With such a statement, all traffic should be sent to 192.168.2.200 then of course get forwarded to the request destination. There's nothing wrong with this. But it does mean all request hit the SVI then route over to the 2620 which would already been done by default if the connection doesn't reside on the segment, instead routing locally on the switch.

Of course the reason you don't see any performance loss within the same vlan is because there is no route decision, just switched packet.

So, what I think, run a trace route from the each end, find out where the snag is getting caught up then assess the routes in place. Also may want to run multiple trace to see if different paths are taken, that would be the explanation why such a huge performance loss.

-Tom
Please mark answered for helpful posts

[stownsend]

In my Simple test case, its just the one SG300-28p and two PCs connected Directly to it. 

Here is a quick Diagram of the way I have things connected right now. This does NOT include my Test SG300-28P.

there used to be two Cisco 2610 (.1 and .2) Units next to 10.1.0.3 that Pointed to .3 and .7 and all of the Clients used to point to either .1 for clients and .2 for Web/Mail.

Thank you.

[Tom Watts]

And currently you have a switch standing alone and you're seeing bottle neck or inconsistent speeds?

                                 switch

                          Vlan 1    Vlan 2

                           --------------------

                            |             |

                            |             |

                        Host A     Host B

-Tom
Please mark answered for helpful posts

[stownsend]

Yes. Switch is latest Firmware too. Origonal Testing it had a Connection to the Corp Netwotk. Now that I've removed my Old Cisco 2620 Units I'm now uing my Test SG300 as a Gateway for the 2nd Firewall, so I cannot Disconnect it to try the Host to Host test without one of the ports on the HQ Subnet. But it would seem like the only thing that should affect Speed would be some sort of broadcast storm.   I can run Wireshark on the Host on the HQ Net and see what it shows.

Thank you for all your help. My Users are sooo much Happier!

Scott<-