Nexus FEX and PXE booting

[Daniel Davidson]

I have a new network I am setting up in our server room using two Nexus 5596 and several FEX connecting back to them, and in general, it works great.

However, our cluster needs to do PXE booting in order to reinstall nodes when they are first installed or fail,  and it is timing out and failing when the ports are in trunk.

Current config:

cfs eth distribute

feature lacp

feature vpc

feature lldp

feature vtp

feature fex

fex 123

  pinning max-links 1

  description "FEX0123"

interface port-channel123

  switchport mode fex-fabric

  fex associate 123

  vpc 123

interface Ethernet123/1/3

  switchport mode trunk

  switchport access vlan 170

  switchport trunk native vlan 170

  spanning-tree port type edge trunk

  no shutdown

Normally, on an IOS system the spanning tree line will fix the problem, and when I change to an access port on vlan 170, everything works fine as well.  However, it seems something is causing this to time out, and I cannot find what it is.  This will be a fairly large environment soon, so modifying the port entry for every system that gets reinstalled will get tedious very quickly.

Any help greatly appreciated,

Dan

[Jerry Ye]

Is the server configured to use trunk on the NIC? Or after the PXE, it will use trunk?

You have the interface set to switchport mode trunk and when that is configured, switchport access vlan 170 will not take affect (this command is for switchport mode access). I think you need to investigate where is the time out comes.

Regards,

jerry

[pille1234]

was my thought too at first, but the line

switchport trunk native vlan 170

should solve any problems with the vlan tag.

Do a show span int e123/1/3 detail and you should see something like this:

Timers: message age 0, forward delay 0, hold 0                               
   Number of transitions to forwarding state: 1                                 
   The port type is edge by port type edge trunk configuration          

If you stil have a timeout I would have a look with wireshark to investigate further.

regards,

pille

[Daniel Davidson]

To answer two posts in one, the access line is there because I keep having to switch back and forth.  Right now, it is in trunk mode, so the access line is just there inactive for when I change it back. 

Adding in a little more information that would have made things clearer earlier, the initial PXE boot works fine, but when the OS loads off PXE immediately after, it cannot get an IP and dies.

Leaving out the other vlans than 170 (which is the main one that matters here) :

sw-server-0-0# show spanning-tree int e123/1/3 detail

.....

Port 3971 (Ethernet123/1/3, vPC) of VLAN0281 is designated forwarding

   Port path cost 1, Port priority 128, Port Identifier 128.3971

   Designated root has priority 4096, address 000c.dbcf.97e0

   Designated bridge has priority 33049, address 547f.ee93.7041

   Designated port id is 128.3971, designated path cost 6

   Timers: message age 0, forward delay 0, hold 0

   Number of transitions to forwarding state: 2

   The port type is edge by port type edge trunk configuration

   Link type is point-to-point by default

   Bpdu guard is enabled

   Bpdu filter is enabled by default

   BPDU: sent 11, received 0

.....

[Jerry Ye]

Is VLAN281 tagged? Meaning is the PXE server image have VLAN281 tagged? You configuration have VLAN 170 as native VLAN (non-tagged) and everything else will be tagged.

Regards,

jerry

[Daniel Davidson]

170 is the internal wired network of the cluster, and should be the untagged (main) link for systems in the cluster.  Outside of the cluster 281 is our normal untagged vlan.  I made a mistake trying to be quick and copied the wrong one.  Machines in the cluster should never use vlan 281.

Port 3971 (Ethernet123/1/3, vPC) of VLAN0170 is designated forwarding

   Port path cost 1, Port priority 128, Port Identifier 128.3971

   Designated root has priority 32768, address 0017.0f6a.d8aa

   Designated bridge has priority 32938, address 547f.ee93.7041

   Designated port id is 128.3971, designated path cost 6

   Timers: message age 0, forward delay 0, hold 0

   Number of transitions to forwarding state: 2

   The port type is edge by port type edge trunk configuration

   Link type is point-to-point by default

   Bpdu guard is enabled

   Bpdu filter is enabled by default

   BPDU: sent 11, received 0

[Jerry Ye]

My question for you is, after login, is the server configured to use VLAN281 (trunked)?

Is VLAN281 configured with ip helper-address or any DHCP relay?

Regards,

jerry