Port-Channels not coming up until VSM reloaded

[nashwj]

We've had some real issues at this site getting the 1Kv up.  We were on the phone with TAC for hours yesterday with other issues (one turning out to be a bug)...  For the most part everything is going well today, except one thing.  When I roll a server in to the dvSwitch its port-channels don't come up until I reload the VSM.  Below is the output of show interface brief after I rolled two NICs (two separate uplinks) in to the dvSwitch:

Eth6/5        500    eth  trunk  link-up Link not connected         auto(D) 12
Eth6/9        501    eth  trunk  link-up Link not connected         auto(D) 11

--------------------------------------------------------------------------------
Port-channel VLAN  Type Mode   Status  Reason                    Speed  Protocol
Interface                                                               
--------------------------------------------------------------------------------
Po1          3000  eth  trunk  up      none                      a-1000(D)  none
Po2          501   eth  trunk  up      none                      a-1000(D)  none
Po3          500   eth  trunk  up      none                      a-1000(D)  none
Po4          3000  eth  trunk  up      none                      a-1000(D)  none
Po5          500   eth  trunk  up      none                      a-1000(D)  none
Po6          501   eth  trunk  up      none                      a-1000(D)  none
Po7          501   eth  trunk  up      none                      a-1000(D)  none
Po8          3000  eth  trunk  up      none                      a-1000(D)  none
Po9          500   eth  trunk  up      none                      a-1000(D)  none
Po10         3000  eth  trunk  up      none                      a-1000(D)  none
Po11         501   eth  trunk  down    No operational members    a-1000(D)  none
Po12         500   eth  trunk  down    No operational members    a-1000(D)  none

You can see Eth6/5 and 6/9 and the resulting po11 and po12.  When I reboot the VSM everything comes up correctly...but only after I do that.  Here is the uplink configurations:

port-profile type ethernet Corp_Net_Uplink
  vmware port-group
  switchport mode trunk
  switchport trunk native vlan 3000
  switchport trunk allowed vlan 1-2,5,11,40,999-1000
  channel-group auto mode on
  no shutdown
  system vlan 1,999-1000
  state enabled

port-profile type ethernet DMZ_1_Uplink
  vmware port-group
  switchport mode trunk
  switchport trunk native vlan 501
  switchport trunk allowed vlan 50
  channel-group auto mode on
  no shutdown
  system vlan 50
  state enabled

Physical NICs go in to a switch stack (for Corp) and a single 3750 (for DMZ1) and channel-group mode on for the interfaces on those switches.  After VSM reload all is happy.

[nashwj]

And after reload (and I added a third NIC before the reload):

Eth6/1        3000   eth  trunk  up      none                       1000(D) 10
Eth6/5        500    eth  trunk  up      none                       1000(D) 12
Eth6/9        501    eth  trunk  up      none                       1000(D) 11

--------------------------------------------------------------------------------
Port-channel VLAN  Type Mode   Status  Reason                    Speed  Protocol
Interface                                                               
--------------------------------------------------------------------------------
Po1          3000  eth  trunk  up      none                      a-1000(D)  none
Po2          501   eth  trunk  up      none                      a-1000(D)  none
Po3          500   eth  trunk  up      none                      a-1000(D)  none
Po4          3000  eth  trunk  up      none                      a-1000(D)  none
Po5          500   eth  trunk  up      none                      a-1000(D)  none
Po6          501   eth  trunk  up      none                      a-1000(D)  none
Po7          501   eth  trunk  up      none                      a-1000(D)  none
Po8          3000  eth  trunk  up      none                      a-1000(D)  none
Po9          500   eth  trunk  up      none                      a-1000(D)  none
Po10         3000  eth  trunk  up      none                      a-1000(D)  none
Po11         501   eth  trunk  up      none                      a-1000(D)  none

I didn't touch the physical switches or the ESX host itself..just "reload" on the VSM.

[ryan.lambert]

Quick question. Unfortunately, not an answer. What version of software are you on?

I am getting prepared to roll out our 1KV to a test environment in the next few days using 4.0(4)SV1(2). I'd be interested to know, so I can keep my eyes peeled for stuff like this. I have a slightly different setup, in that I'm going back to a pair of Nexus 5020s running vPC-MAC on my 1kv.

[nenduri]

It is nothing to do with port channels. Ports will get added to port channels when they come up. In this case the ports are not coming up.

Looks like they are stuck and waiting for a link up from VEM.

Do you see this even if you add a new pNIC?

If so, once the module is up enable the following debugs and send the logs across.

module vem <mod#> exec vemdpalog debug sfportagent all

module vem <mod#> exec vemdpalog  start

<add a new pNIC. recreate the issue>

module vem <mod#> exec vemdpalog stop

module vem <mod#> exec vemdpalog show all

-Naren

[nashwj]

I'll check when I get back onsite.  The ports on these servers were in access-port mode on the switch before rolling them to the dvSwitch.  It appeared..in my very limited testing...that if I reconfigured them for proper trunking BEFORE moving them to the dvSwitch they came up fine.  But if I rolled them in to the dvSwitch first, and then did the trunk config they went in to that state.  So not sure how long they'd stay like that..but I'll test it again.

Thanks!