3560G to 3750X/G LACP issues?

[bleucube]

Having a really weird isse and can't figure it out.  Turning to you guys for help. 

Issue(s)
* I mostly can't ping the managment interface of my leaf floor switches, but I can ping through them
* If I try to ping them, I get one ping response after that it times out or takes greater than 300 ms to respond
* Network seems sluggish

What we did recently:
* We upgraded the IOS on our leaf switches and core stack
* We setup LACP / STP so that each leaf switches has 2 uplink connections.  Each link goes into a different core switch.

Here's the new version;
Leaf
Cisco 3560G-48-PS, SW Version 12.2(58)SE2, SW Image C3560-IPBASEK9-M

Core Stack
1 WS-C3750X-24, SW Version 12.2(55)SE1, SW Image C3750E-UNIVERSALK9-M
2 WS-C3750X-24, SW Version 12.2(55)SE1, SW Image C3750E-UNIVERSALK9-M
3 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M
4 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M
5 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M
6 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M

Attached are the configs of the leaf and core.  Please tell me if you see anything weird.  I'm at a loss on this one.

[Reza Sharifi]

Looking at your config, you are using vlan 74 for both user data vlan and management vlan with Qos configured. Is this correct? 

Usually, your management vlan needs to be a separate vlan that is not used for data, voice, servers, etc...

HTH

[bleucube]

Thanks for the tip.  We tried changing the vlan for management to 6 but that did not resolve the issue.

[Zeeshan Siddiqui]

If not wrong you can ping the end devices on the leaf switch there are no time outs ?

When you ping the switch managment interface you can see time outs ?

By looking at the config you have allowed all vlans over the trunks ports which is not a good practice I would restrict then to the vlans needed.

Can you run a sh process memory sorted on the leaf switch and also check the stats on the interface on both ends (sh int port-channel) and (sh interface physical interfaces) please paste the output

[bleucube]

See attached

[Zeeshan Siddiqui]

We are good with free memory.

Can you please confirm the following

If not wrong you can ping the end devices on the leaf switch and there are no time outs ?

When you ping the switch managment interface you can see time outs ?

[bleucube]

I can ping a printer (172.22.74.30) that's on the leaf switch just fine.  Comes in under <1ms

If I ping the leaf switch (172.22.74.2).  The first response comes in at 13ms, after that it times out.  Sometimes it just times out right away.

I'm ping from a server on 172.22.1.142

[Zeeshan Siddiqui]

I had a simlar problem where the managment IP on the switch had delays a reboot fixed it for me.

As mentioned by others below can you configure the ports the same

Your config

nterface GigabitEthernet0/1

description "Device: Uplink to Sw0-core(Gi5/0/9)"

switchport trunk encapsulation dot1q

switchport mode trunk

speed 1000

duplex full

channel-protocol lacp

channel-group 6 mode active

!

interface GigabitEthernet0/2

description "Device: Uplink to Sw0-core(Gi6/0/9)"

switchport trunk encapsulation dot1q

switchport mode trunk

channel-protocol lacp

channel-group 6 mode active

It should be as follow

In the future I would restrict the vlan needed accross the trunks

nterface GigabitEthernet0/1

description "Device: Uplink to Sw0-core(Gi5/0/9)"

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 6 mode active

!

interface GigabitEthernet0/2

description "Device: Uplink to Sw0-core(Gi6/0/9)"

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 6 mode active

The ports should be configured exactly the same on the core switch to rule out all issues

You can also add the ip default gateway command on the leaf switch

[amikat]

Hi,

as the Etherchannel ports should be configured to operate both the same speed and duplex I would suggest to delete the "speed 1000" and "duplex full" commands under the Gi0/1 interface of your SW1-0 leaf switch (ie. configure both parameters to default "auto" as the other ports in the EC) and see if there is any improvement.

Best regards,

Antonin

[tdriscoll]

I am working (physically) with the original poster on this problem. I have set gi0/1 on sw1-0 to speed auto and duplex auto, and no change. Both legs of the etherchannel are up at 1G, full duplex both before and after the change and are members of the bundle. 

I am also debugging on the C1-R00 switch, Spanning-Tree general events, vtp events, PAgP Aggregation Control Protocol events(as we are using this protocol call also on this switch), LACP aggregation control protocol events, PAgP/LACP Shim details, and I am not seeing errors that would point me in a direction. We had been using PAgP between the stack and this leaf switch and this problem really only starting showing up after we rebuild the stack and datacenter and switched to LACP for the Etherchannels/Trunks. We did upgrade to 12.2(58) on the leaf switches and to 12.2(55) IP Services on the C1-R00 during this rebuild. The stack is a 6 switches stack and the legs of this switch (sw1-0) physically connect to switch 4 and 5 (gi4/0/9 and gi5/0/9) in the stack.

There are other switches connected to stack that are configured up and connected the same way and are not exhibiting this behavior. The key device is a monitoring system running Whats Up Gold on a windows server. It can monitor (ping) devices connected to the switches that are not themselves able to be pinged. In addition, the windows server is a Physical to Virtual server that now resides on a VMWare ESXi server that is trunked to our datacenter network to serve up the guests.

I hope this describes the layout with more detail. I am just not seeing errors that would help see this problem and I am hoping someone here may have seen similar problems and might be able to suggest a course of debugging/testing that might help generate an error that will help resolve this problem.

[glen.grant]

  Are you sure the stack itself is working correctly ?  It looks like you are not running the same version across your  switches .  I would also check to see where the spanning tree roots are .  Make sure they are not down on the access switches .    Good practice would be to manually prune off any vlans not needed on a switch with the " switchport trunk allowed vlan " command.   If you do a show switch detail on the core what does it say ? 

[Leo Laohoo]

Core Stack
1 WS-C3750X-24, SW Version 12.2(55)SE1, SW Image C3750E-UNIVERSALK9-M
2 WS-C3750X-24, SW Version 12.2(55)SE1, SW Image C3750E-UNIVERSALK9-M
3 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M
4 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M
5 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M
6 WS-C3750G-24T, SW Version 12.2(55)SE4, SW Image C3750-IPSERVICESK9-M

Seriously?  Your stack ain't working.  You have a version mis-match.

[tdriscoll]

Actually they are matched. the Universal Image contains the IP Services within and we purchased the license to unlock it. So they are running IP Services. And the stack does appear to be working correctly overall. I did have questions about the master for STP root so I did set the master on the stack (C1-R00) and a secondary our other main switch to be secondary. I understand the good practice of pruning and I have prunned in other location within our network where it was neccessary, but would pruning really be an issue? I have attached the switch detail info.

[amikat · Accepted solution]

Hi,

It appears that you have not enabled ip routing at your "sw1-0" leaf switch in which case the command specifying the default route "ip route 0.0.0.0 0.0.0.0 172.22.74.1" is inappropriate. You can either enable it via "ip routing" command or leave it as it is and use the command "ip default-gateway 172.22.74.1" instead.

Best regards,

Antonin

[tdriscoll]

Amikat,

That was the problem some of the leaf switches did not have "ip routing" I have corrected that and it appears to be working now. Thank you everyone for your assistance with this.

Thank you

Tad