Port-Channel Issues on CAT6509-E

[Benjamin Story]

I have a VSS pair setup as my datacenter switch.  We are starting to reconfigure some of our servers to use their NICs as port-channels.  So far most have gone well.  Unfortunately we have three that seem to be oddballs even though they are configured the same as the working ones.  Here is our basic config:

interface port-channel 401

description SERVERNAME

switchport

switchport access vlan 104

switchport mode access

logging event link-status

logging event trunk-status

logging event spanning-tree status

interface GigabitEthernet1/3/19

description SERVERNAME

switchport

switchport access vlan 104

switchport mode access

no logging event link-status

carrier-delay msec 0

wrr-queue bandwidth 5 25 70

wrr-queue queue-limit 5 25 40

wrr-queue random-detect min-threshold 1 80 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 2 80 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 3 50 60 70 80 90 100 100 100

wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100

wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100

wrr-queue random-detect max-threshold 3 60 70 80 90 100 100 100 100

wrr-queue cos-map 1 1 1

wrr-queue cos-map 2 1 0

wrr-queue cos-map 3 1 4

wrr-queue cos-map 3 2 2

wrr-queue cos-map 3 3 3

wrr-queue cos-map 3 4 6

wrr-queue cos-map 3 5 7

channel-protocol lacp

channel-group 401 mode active

spanning-tree portfast edge

interface GigabitEthernet2/3/19

description SERVERNAME

switchport

switchport access vlan 104

switchport mode access

no logging event link-status

carrier-delay msec 0

wrr-queue bandwidth 5 25 70

wrr-queue queue-limit 5 25 40

wrr-queue random-detect min-threshold 1 80 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 2 80 100 100 100 100 100 100 100

wrr-queue random-detect min-threshold 3 50 60 70 80 90 100 100 100

wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100

wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100

wrr-queue random-detect max-threshold 3 60 70 80 90 100 100 100 100

wrr-queue cos-map 1 1 1

wrr-queue cos-map 2 1 0

wrr-queue cos-map 3 1 4

wrr-queue cos-map 3 2 2

wrr-queue cos-map 3 3 3

wrr-queue cos-map 3 4 6

wrr-queue cos-map 3 5 7

channel-protocol lacp

channel-group 401 mode active

spanning-tree portfast edge

One one of the problem port-channels everything is up and working, but in the show commands you see Port-Channel 401 and 401A.  401A is the one that is working.  What is with the A?  The other two are having one of the two NICs marked as failed/standby by the server software.  One server is running HP's Teaming software and the other Intel's.  Both are using the same driver and software revisions as a working server elsewhere in the datacenter on the same switch. 

Any tips for troubleshooting these problems will be greatly appreciated.

Thanks,

Ben

[andtoth · Accepted solution]

Hi Benjamin,

The port-channel with the A letter means it's a secondary aggregator port. A secondary aggregator port is created in the LACP process when the ports you

are bundling are not compatible with each other, or with their remote peers.

Secondary Aggregator Ports Po1A or Po2A are Created

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_configuration_example09186a0080094470.shtml#po1a

Did you configure the port-channel between ports on the same type of linecards? Different linecards have different QoS settings (see in 'sh int capabilities') and it could cause these issues.

Might be an issue with remote peer as well. Try doing a shut/no shut on the original port-channel to see if that resolves the issue.

Best regards,

Andras

[ranraju]

If you unbundle the portchannel and rebundle, the issue would get resolved. This problem is seen when you are doing some changes on the portchannel, and if there is a mismatch in the configuration between the ports and the actual portchannel, then the ports will get bundled up under a different name. The new portchannel number would be - the old portchannel number appended by the letter A. If this problem is seen again, then the switch renames the portchannel to 401B, 401C so on till ..Z. after it continues to append with AA, BB... ZZ.

Creating the portchannel again should work.

Hope this answers your question on this post.

Regards,

ranraju

[Benjamin Story]

Thanks all on the 403A question.  That makes sense and cleared that issue.  Any ideas on the other port-channels that are not coming up correctly?

[andtoth]

Benjamin,

Could you please paste the actual error message and outputs you are seeing?

An output of the 'sh log' could help as well.

Thanks,

Andras

[Benjamin Story]

Unfortunately the only thing that shows in the logs is %LACP-SW1_SP-4-MULTIPLE_NEIGHBORS: Multiple neighbors detected on port Gi1/3/19

I've done a debug lacp events and either I didn't see a difference between the working port and the non-working one.

[andtoth]

Could you please verify if both links are going to the same device from the 6500 switch? It might be a miscabling and one member of the port-channel might go to one server and the other to another which could cause this issue.

Andras

[ranraju · Accepted solution]

Yes the logs will help us to tell y the portchannel is not coming up.. Can you also apply the QoS on the Portchannel and check rather on the interfaces. And have the same configuration on the interfaces and on the portchannel - thats the first step that we have to make sure if we have done it correctly. "show etherchannel summary" - output should also be helpful.. can you paste that here..

Logging event link-status is disabled on the ports but enabled on the portchannel... as said, try having the same configuration on all. Finally it would be best if you compare "show int po401" - "show int " (compare all the interfaces with the portchannel).

regards,

ranraju

[Benjamin Story]

Based on your comments, I went onto the switch and defaulted the interface configuration and port channel configuration and then recreated them without the QoS statements.  So far the port channel seems stable with both ports bundled and no errors in the logs.

[ranraju]

Thats great!

Try applying QoS configurations on the portchannel and see if it breaks.. If it does, add line by line of the QoS config and see what is breaking the portchannel. That might help you know what was wrong.

Regards,

ranraju

(TAC)