topic Re: VMware VMXNET3 Adapter Remapping in Network Access Control

VMware VMXNET3 Adapter Remapping

noisey_uk — Wed, 20 Mar 2019 17:32:55 GMT

Cisco Identity Services Engine Installation Guide, Release 2.4 states that "If you choose VMXNET3, you might have to remap the ESXi adapter to synchronize it with the ISE adapter order."

My design requires 6 x VMXNET3 adapters and they're out of the expected order, as warned by this statement.

Does anyone have information on *how* to remap the ESXi adapter so that it realigns with the ISE adapter order?

As present, this is the mapping:

VMware Network Adapter 1 > ISE GE0
VMware Network Adapter 5 > ISE GE1
VMware Network Adapter 2 > ISE GE2
VMware Network Adapter 6 > ISE GE3
VMware Network Adapter 3 > ISE GE4
VMware Network Adapter 4 > ISE GE5

Whereas I'd like the more intuitive mapping:

VMware Network Adapter 1 > ISE GE0
VMware Network Adapter 2 > ISE GE1
VMware Network Adapter 3 > ISE GE2
VMware Network Adapter 4 > ISE GE3
VMware Network Adapter 5 > ISE GE4
VMware Network Adapter 6 > ISE GE5

Re: VMware VMXNET3 Adapter Remapping

Arne Bier — Wed, 20 Mar 2019 20:32:16 GMT

Good question. I am wondering why this even happens in the first place.

Strange that it only happens when you cross over a certain count (three?). I would still continue using vmxnet3 but I agree if you need to do this gymnastics for 50 nodes then you might be annoyed.

Re: VMware VMXNET3 Adapter Remapping

paul — Thu, 21 Mar 2019 01:48:24 GMT

I believe the threshold is 4 before renumbering occurs. I would be curious what design needs 6 NICS. In a 100+ installs I have never used more than 2.

Re: VMware VMXNET3 Adapter Remapping

noisey_uk — Fri, 22 Mar 2019 09:45:16 GMT

The threshold appears to be 4 - when I only had 4 configured they were in the correct order.

Re: VMware VMXNET3 Adapter Remapping

noisey_uk — Fri, 22 Mar 2019 09:49:57 GMT

Correct.
3 sets of 2 bonded NICs. bond0 for management; bond1 for internal; bond2 for DMZ guest services. Granted, the NIC bonding is arguably over-the-top/unnecessary given it's a virtual appliance so both member interfaces will be connected to the same vSwitch.

Re: VMware VMXNET3 Adapter Remapping

Nadav — Fri, 22 Mar 2019 14:25:17 GMT

Connecting your DMZ to a shared vSwitch for both management and internal networks? Quite risky.

As for your predicament, I'm unsure if it's possible to fix this without TAC involvement if you truly need more than 4 VMXNET3 interfaces.

Here are a few possible solutions. Maybe one can apply:

1) Involve TAC, this is a supported configuration afterall.

2) Seeing as this is on the same vSwitch, perhaps you can put management and internal VLANs on the same bond. That would mean you only have 4 interfaces to deal with. There may or may not be security concerns but that's up to your architecture and where you can place your controls.

3) Is there any particular reason why either the DMZ, Management or Internal networks will need more than 1Gbps at a time from this server for any one of these bonds? E1000 is an emulation of a 1Gbps NIC, which should be fine for most ISE deployments. For a PSN node the worst case scenario is that the Internal bond is used for both inter-ISE server traffic and AAA traffic. I'm unsure how many transactions per second you're expecting to hit, but I imagine 1Gbps worth full-duplex is a stretch for a single node. If you're hitting those kinds of numbers then you really should have more PSNs in place. If your server is an Admin node, then you are very unlikely to need VMXNET3.

Re: VMware VMXNET3 Adapter Remapping

noisey_uk — Fri, 22 Mar 2019 19:31:42 GMT

Segregating traffic using VLANs (including in a vSwitch) is common accepted practice in most environments. If it was a highly secure environment then I would agree... but I'd also be using a two tiered firewall with different vendors. Horses for courses.

TAC involvement was useless. They sent me information on VMware Workstation and vSphere 5.0, using the Client. It's always a lucky dip whether you get a useful response in my experience and unfortunately this time I didn't.

Unless I've missed something, it's not possible to subinterface an ISE bond. They're not a true bond anyway - they're active/passive.

Standardising on VMXNET3 across all VMs. It also gives the best room for future growth. It's one of those annoying situations where vendor Best Practice clashes - VMware's is to use VMXNET3 unless mandated otherwise; Cisco says that E1000 should be used, but only to avoid the situation I've come across.

Re: VMware VMXNET3 Adapter Remapping

noisey_uk — Mon, 25 Mar 2019 22:06:07 GMT

Re: VMware VMXNET3 Adapter Remapping

Nadav — Sat, 23 Mar 2019 20:01:37 GMT

I was thinking more along the lines of making the bond a couple of vNICs with the same access VLAN, and configuring the gateway's subinterface as dedicated for ISE. The bond could then serve both management and internal networks on a single subnet. You would add your security controls at the gateway.

Not ideal, but it may work for you.

Re: VMware VMXNET3 Adapter Remapping

noisey_uk — Mon, 25 Mar 2019 22:04:49 GMT

In ISE CLI, show interfaces. This will list the MAC addresses used by GigabitEthernet0-5.
Go to vSphere [your VM] Summary > Hardware Configuration. This will list the MAC addresses used by NETWORK ADAPTER X. Bear in mind that internal VMware enumeration (which is used later) is 1 number less than what is displayed. e.g. NETWORK ADAPTER 1 is ethernet0 internally to VMware.
By cross-referencing the MAC addresses, you can map each ISE GigabitEthernetX to its current VMware NETWORK ADAPTER.
In vSphere, go to [your VM] > Edit > VM Options > Configuration Parameters > Edit Configuration and you will see the current mapping of VMware Network Adapter to PCI Slot Number:
ethernet0.pciSlotNumber 160 (ethernet0 = NETWORK ADAPTER 1)
ethernet1.pciSlotNumber 192 (ethernet1 = NETWORK ADAPTER 2)
ethernet2.pciSlotNumber 224 (ethernet2 = NETWORK ADAPTER 3)
ethernet3.pciSlotNumber 256 (ethernet3 = NETWORK ADAPTER 4)
ethernet4.pciSlotNumber 1184 (ethernet4 = NETWORK ADAPTER 5)
ethernet5.pciSlotNumber 1216 (ethernet5 = NETWORK ADAPTER 6)
Now, looking at this *from the perspective of the vSphere ESXi hypervisor*, reorder the interfaces.

Re: VMware VMXNET3 Adapter Remapping

Nadav — Tue, 26 Mar 2019 13:42:25 GMT

If it works, awesome 🙂

Did you check that your mapping persists after you shutdown and power on the VM?