Layer 3 switching on single switch

[NathanGau4996]

Hey guys, I'm not really a switching and routing expert, and I've been combing the internet looking for answers to what I suspect would be a simple problem for people who do this regularly, but I've had no luck so far.

I'm dealing with a couple of issues.

What I have is a NX 5548 switch with a daughter card to enable Layer 3. It also connected via FEX to an N2k switch for what that's worth since I have fiber cards on my servers.

I have 3 VLANS setup on said switch (vlan 1,2, and 3). I have a grand total of 2 physical servers (with about 8 VMs in total) and a couple more laptops connected to them. The servers are using teamed NICs and hyperV is running on both the servers and workstation (each workstation has 2 VMs). Trunking is enabled on the fiber ports for the servers, for what that's worth. Layer 3 switching is working for what that's worth, but there are a lot of dropped packets between the VLANs and (not sure it's related) I'm unable to relay DHCP addresses from a windows DHCP server (which sits on VLAN 1) to VLAN3. 

What I've done so far:

feature dhcp

feature ip routing

feature interface-vlan

interface vlan 1

description servers

ip address 192.168.1.1/24

no shutdown

interface vlan 2 (note this only for when I need to connect to the internet, nothing is on this presently but it does periodically get used)

description internet

ip address 192.168.2.1/24

no shutdown 

interface vlan 3

description workstations

ip address 192.168.3.1/24

no shutdown

ip dhcp relay address 192.168.1.10 <-- IP address of my Windows DHCP server.

I can, for what it's worth successfully ping between 1 and 3, but even ping -t is showing lag and occasional dropped packets. RDP across the VLANs is very slow, as is moving files between the workstations and servers. Within VLANs everything works fine. If I do a show interface of the specific ports connected to these VLANs, I'm seeing very slow through put in the statistics essentially confirming what I already know. DHCP to VLAN 3 does not work. I'm assuming at this point though that the problem is likely throughput related. Note that I've tried adding the broad cast address as dhcp relay as well. I'm using the default VRF with not much else configured.

I should note I'm a windows guy, so this is not something I've had to do a lot of. I'm assuming this is probably something simple that people who do this every day will catch, but I'm not seeing it. Any ideas as to why my throughput is bad and why DHCP isn't working?

[Richard Burts]

Thanks for posting the configuration. A couple of things in it surprise me. I note that you have 6 interfaces on the switch configured as trunk but the only vlan allowed on the trunk is vlan 1. Is there a reason for this? I note that interface vlan 2 has no ip address. Are there devices in vlan 2? If so what is routing for them?

I wonder if your issue may have something to do with getting back and forth using fex. Do you have devices that are connected to the main switch, not using fex? Could you put a device on a main switch port in vlan 1 and another device on a main switch port in vlan 3 and see if ping to them and between them has the problem?

HTH

Rick

[NathanGau4996]

Hi Richard,

There definitely could be something along the lines here of not knowing what we are doing :) We are windows guys.. 

Those fiber ports are all trunked to my hyper V servers with their nic team. I seem to remember communication not working until we trunked those lines, but I did remove the trunks. It's worth noting that my DHCP server is on one of those lines as was the large file I attempted to move. We have a virtual switch with hyper V setup, but every VM is on VLAN1 (and no plans to change that), so we shouldn't need to trunk, correct?

That said, I did try and a device on Vlan1 and Vlan 3. I thought I posted that up thread somewhere, but perhaps not. I used two of the copper ports and didn't use Fiber at all. I still saw issues. I didn't do anything fancy other an a continuous ping, but I wasn't seeing better latency results there. I don't remember if I was seeing dropped packets. I can run that test again though.

As for Vlan2, there's not much on it. I use it to slave a laptop to a mobile hotspot to download patches and update WSUS internally. The default gateway is technically the laptop when I'm doing that. Presently nothing is on it. But I've got to update patches at some point yet this month, so I'll fire it up at some point this week or next. 

[superego]

Nathan,

As you're doing NIC teaming on the server side, in the the Nexus side it should be a port-channel as well.

https://social.technet.microsoft.com/wiki/contents/articles/51163.windows-server-2016-nic-teaming-with-cisco-switch.aspx

[superego]

Another thing is, I see fex 100 on two ports Eth1/15 and Eth16. Nexus 2K-1 have 2 uplinks to 5K-1 right?

You should port-channel it as well.  Have a back up of your config before doing it.

Example on how to do it, but do it during maintenance window as it will require network outage.

interface Ethernet1/15
channel-group 100

interface Ethernet1/16
channel-group 100

!

interface po100
switchport mode fex-fabric
fex associate 100

to check:
show fex
show fex detail

[NathanGau4996]

In answer to both of your posts, setting a port channel on the team didn't help.

I started to do it on the FEX ports, but got an error saying they are in the wrong mode... So if I'm understanding this, I'll need to undo the Fex and trunk settings on those ports and then redo them for the port channel? What exactly am I doing here? Creating a team so to speak for the Fex trunk?

[superego]

Have a backup of your config as the fex 100 ethernet config might be lost.

On the global config mode:

default interface Ethernet1/15

default interface Ethernet1/16

!

interface Ethernet1/15
channel-group 100
!
interface Ethernet1/16
channel-group 100
!
interface po100
switchport mode fex-fabric
fex associate 100

[NathanGau4996]

This was part of the problem. Thanks! As soon as I turned that off, my latency issues went away.

DHCP, however, still does not work across the Vlans. I had hoped they were related. No such luck. 

[NathanGau4996]

This I've done... It hasn't changed anything as I turned on snooping to troubleshoot. 

[NathanGau4996]

To answer your question, yes there is connectivity between the 5k and 2k across the Vlans. I'm losing packets across them and for the life of me cannot get DHCP working across them either.

I'd note that I took too laptops and put them on the separate Vlans to eliminate NIC teaming. I still see high latency and packet loss when doing a continuous ping between them.

As for the setup, it's a single switch with an L3 daughter card. No routers.