Issues on 3550, LAG after some changes

[existhosting]

Hello,

We have been running on a 3550-48 port switch a about 2 years now with no problem. Recently, we got our new set of IP's and wanted to assign a subnet of 8 IP's to each customer. Therefore to plan ahead and to be ready for implementation, we went and created 512 VLANS on the switch as well as configuring each VLAN with its gateway IP. So we have on our switch 512 configured VLANS (along with the other vlans which are currently servicing over 400 servers).

Since we did that, there is a lag on the switch, even if we ping locally on our network from one subnet to the other we usually get something like this:

Reponses:

<1 MS

<1 MS

<1 MS

22 MS

<1 MS

<1 MS

15 MS

It fluctuates all the time maybe because there is a load or something ON the switch. Before doing those changes, it used to be ALL the time <1 MS (of course since it is inside the network)

Can that change we made affect our switch's performance that much?? Is it the fact that we added all those VLANs and routed them or maybe some other protocol that got more clustered because of that (like spanning tree or some other thing). CPU is running at 12 % and memory still has over half free... We are not sure where to check...

Any help would be greatly appreciated.

Thank You

[Amit Singh]

It could be the fact that you added those number of vlans and routed them on the switch. It could also be related to the SDM template you are running on the switch.

The default template balances the use of unicast MAC addresses, IGMP groups, QoS ACEs, security ACEs, unicast and multicast routes, routed interfaces, and Layer 2 VLANs.

Try changing the SDM template to routing to maximizes system resources for unicast routing.

Please see the link below for more info:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3550/software/release/12.2_25_seb/configuration/guide/swadmin.html#wp1219495

HTH,Please rate if it does.

-amit singh

[existhosting]

Hello,

I did this, and it did not really help.. We get the same lag on the switch.. Here is a ping from our network to the directly connected switch from our provider:

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=50ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=10ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=27ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time<1ms TTL=63

Reply from 64.XXX.XXX.9: bytes=32 time=1ms TTL=63

As you can see, it JUMPS sometimes and we are not sure why. Here is what we have now for template:

mtl-sw-gw1#show sdm prefer

The current template is the routing template.

The selected template optimizes the resources in

the switch to support this level of features for

8 routed interfaces and 1K VLANs.

number of unicast mac addresses: 5K

number of igmp groups: 1K

number of qos aces: 512

number of security aces: 512

number of unicast routes: 16K

number of multicast routes: 1K

Thank you SOO MUCH for helping us.

Thanks

[existhosting]

Hello,

Any reply on this please? This is URGENT !!. We need to get to the bottom of this ASAP

Thanks

[Amit Singh]

What is the IOS running on the switch. Did you try reloading another IOS on the switch. I really dont know why still the lag is there.Try reloading the new IOS and check if that helps.

If that doesnot help, I would suggest you to speak with TAC on this.

Here is the TAC information:

http://www.cisco.com/public/news_training/tac_overview.html

HTH,

-amit singh

[existhosting]

Hello,

We just recently updated the IOS running to this one 12.2(37)SE - IP-SERV

Basically, it was this file we downloaded from Cisco to update it:

c3550-ipservices-tar.122-37.SE.tar

Please let us know

Thanks

[Amit Singh]

That's the latest one. Are you really getting some performance issues on this box. It seems to be fine with me. CPU is OK, memory consumption is OK, its just a few lags that you are getting. Are you facing any application performance issues?

ICMP packets are given the least priority on Cisco devices.I think they treat most router-destined traffic in general with a lower priority. And router-originated traffic (which ss often ICMP) is process-switched rather than fast-switched, so it gets lower priority.

HTH,Please rate if it does.

-amit singh

[glen.grant]

Looks fine to me , the very few longer ones could be that the router is busy doing soemthing else at that particular moment , always ping thru the router to a device on the network .

[existhosting]

Hello,

yes that is what we mean. We ping OUR switch to the directly connected Switch of our provider connected through a GIGE uplink over fiber through our GBIC in port 2 on our switch.

So it is a DIRECTLY connected interface, we use to get <1 MS average and MAX which is normal. I just issued some tests and we get

1/4/80 MS

MIN = 1 MS

Average = 4 MS

MAX = 80 MS

An average of 4 MS is not normal. Those are directly connected to the switch, should be <1 MS

Please let us know, if you need me to put in switches output let me know the command and we will do it for you

Thanks

[existhosting]

hello,

any reply on this?

Thanks

[tbhagia]

Hi,

I would like to confirm certain details over here. You said that you have created 512 new vlans on 3550 switch. Are you using some other device for intervlan communication or is this switch doing routing as well? Also the link between the service provider switch and your switch is a direct link. Is it a trunk or layer 3 link? what is the port configuration? I will contact you as soon as I receive the above info.

[existhosting]

Hello,

No that switch is doing the routing as well. That switch is the switch directly connected to our provider, nothing else in between. Yes it is a DIRECT link over SC fiber using a GIGE port with a GBIC. It is not a trunk, it is basically a /30 in which they have the .9 and we have the .10 and we send all traffic 0.0.0.0 to the .9 IP address. It is a layer 3 link as far as I know. Port configuration is speed nonegotiate, full 1000 MBPS speed.

Please let me know

Thanks