Re: Port-Channels not coming up until VSM reloaded

nashwj · ‎03-09-2010

We've had some real issues at this site getting the 1Kv up. We were on the phone with TAC for hours yesterday with other issues (one turning out to be a bug)... For the most part everything is going well today, except one thing. When I roll a server in to the dvSwitch its port-channels don't come up until I reload the VSM. Below is the output of show interface brief after I rolled two NICs (two separate uplinks) in to the dvSwitch:

Eth6/5        500    eth trunk link-up Link not connected         auto(D) 12
Eth6/9        501    eth trunk link-up Link not connected         auto(D) 11

--------------------------------------------------------------------------------
Port-channel VLAN Type Mode   Status Reason                    Speed Protocol
Interface
--------------------------------------------------------------------------------
Po1          3000 eth trunk up      none                      a-1000(D) none
Po2          501   eth trunk up      none                      a-1000(D) none
Po3          500   eth trunk up      none                      a-1000(D) none
Po4          3000 eth trunk up      none                      a-1000(D) none
Po5          500   eth trunk up      none                      a-1000(D) none
Po6          501   eth trunk up      none                      a-1000(D) none
Po7          501   eth trunk up      none                      a-1000(D) none
Po8          3000 eth trunk up      none                      a-1000(D) none
Po9          500   eth trunk up      none                      a-1000(D) none
Po10         3000 eth trunk up      none                      a-1000(D) none
Po11         501   eth trunk down    No operational members    a-1000(D) none
Po12         500   eth trunk down    No operational members    a-1000(D) none

You can see Eth6/5 and 6/9 and the resulting po11 and po12. When I reboot the VSM everything comes up correctly...but only after I do that. Here is the uplink configurations:

port-profile type ethernet Corp_Net_Uplink
vmware port-group
switchport mode trunk
switchport trunk native vlan 3000
switchport trunk allowed vlan 1-2,5,11,40,999-1000
channel-group auto mode on
no shutdown
system vlan 1,999-1000
state enabled

port-profile type ethernet DMZ_1_Uplink
vmware port-group
switchport mode trunk
switchport trunk native vlan 501
switchport trunk allowed vlan 50
channel-group auto mode on
no shutdown
system vlan 50
state enabled

Physical NICs go in to a switch stack (for Corp) and a single 3750 (for DMZ1) and channel-group mode on for the interfaces on those switches. After VSM reload all is happy.

nashwj · ‎03-09-2010

And after reload (and I added a third NIC before the reload):

Eth6/1        3000   eth trunk up      none                       1000(D) 10
Eth6/5        500    eth trunk up      none                       1000(D) 12
Eth6/9        501    eth trunk up      none                       1000(D) 11

--------------------------------------------------------------------------------
Port-channel VLAN Type Mode   Status Reason                    Speed Protocol
Interface
--------------------------------------------------------------------------------
Po1          3000 eth trunk up      none                      a-1000(D) none
Po2          501   eth trunk up      none                      a-1000(D) none
Po3          500   eth trunk up      none                      a-1000(D) none
Po4          3000 eth trunk up      none                      a-1000(D) none
Po5          500   eth trunk up      none                      a-1000(D) none
Po6          501   eth trunk up      none                      a-1000(D) none
Po7          501   eth trunk up      none                      a-1000(D) none
Po8          3000 eth trunk up      none                      a-1000(D) none
Po9          500   eth trunk up      none                      a-1000(D) none
Po10         3000 eth trunk up      none                      a-1000(D) none
Po11         501   eth trunk up      none                      a-1000(D) none

I didn't touch the physical switches or the ESX host itself..just "reload" on the VSM.

ryan.lambert · ‎03-09-2010

Quick question. Unfortunately, not an answer. What version of software are you on?

I am getting prepared to roll out our 1KV to a test environment in the next few days using 4.0(4)SV1(2). I'd be interested to know, so I can keep my eyes peeled for stuff like this. I have a slightly different setup, in that I'm going back to a pair of Nexus 5020s running vPC-MAC on my 1kv.

nenduri · ‎03-09-2010

It is nothing to do with port channels. Ports will get added to port channels when they come up. In this case the ports are not coming up.

Looks like they are stuck and waiting for a link up from VEM.

Do you see this even if you add a new pNIC?

If so, once the module is up enable the following debugs and send the logs across.

module vem <mod#> exec vemdpalog debug sfportagent all

module vem <mod#> exec vemdpalog start

module vem <mod#> exec vemdpalog stop

module vem <mod#> exec vemdpalog show all

-Naren

nashwj · ‎03-09-2010

I'll check when I get back onsite. The ports on these servers were in access-port mode on the switch before rolling them to the dvSwitch. It appeared..in my very limited testing...that if I reconfigured them for proper trunking BEFORE moving them to the dvSwitch they came up fine. But if I rolled them in to the dvSwitch first, and then did the trunk config they went in to that state. So not sure how long they'd stay like that..but I'll test it again.

Thanks!