Do KVM connections have a MAC address?

[rmasteller]

I'm setting up a pair of UCS 6248 FIs in a DEV domain.  One 5108-AC2 chassis is attached via two I/O modules.  There are four B200-M4 blades in the chassis that I want to install ESXi 7.0.2 on (using SD cards for boot devices).

PROBLEM:  I can't get a KVM session on any of the blades.  They time out.  They all successfully get an IP address assigned from the Ext-Mgmt pool.  I asked one of our network engineers if they had any troubleshooting ideas and they tested an ARP command from the core network switch.  For that IP, the ARP results showed "INCOMPLETE" for the MAC. 

I try to ping the KVM IP from my desktop and it fails.  I got an SSH to the console of the UCS Central appliance, got into local-mgmt prompt, and tried to ping the KVM IP and it fails.

Is the KVM supposed to have a MAC?

What else in the UCS config could be misconfigured?

[rmasteller]

FIs are on FW 4.1(3j)A.

Blades are on FW 4.1(3J)B.

[rmasteller]

Oh, and the Physical IPv4 Outband KVM connection on a different vLAN/network works fine.  It's only the Service Profile provided IP that times out.  Thanks.

[Steven Tardy]

I'm confused. Your CIMCs are configured to have two different IPs (one out-of-band in ext-mgmt and one in-band CIMC IP from the service profile)?

It could be that traffic to the OOB IP goes through FI mgmt and gets NAT'd to CIMC internally.
But the return traffic gets sent back through the in-band CIMC IP (which may be in the wrong sub-net and/or VLAN).

Are the OOB IPs for ext-mgmt in the same IP range / sub-net as the FI mgmt IP?

Most of the time I prefer to only use the OOB IP which must be in the same sub-net as the FI mgmt IP as this is the most simple.

In-band CIMC IPs are used when something (a database fencing mechanism) must talk to CIMC directly (to STONITH a node of a cluster).
But when in-band CIMC is used, typically the OOB IP is not used.
Don't know that I've ever seen both used at the same time. Maybe that works, just don't remember seeing that configuration.

OOB CIMC access is (non-obviously) NAT'd through the FI mgmt port physical connection. The FI "owns" your CIMC IP, but when talking to CIMC the FI NATs traffic (internally) down to the CIMC. Return traffic from CIMC should leave the FI mgmt port and get NAT'd appropriately.

Don't know if that helps or confuses you more.

This issue is interesting. Reach out and I can help troubleshoot what's going on over a Webex.

[rmasteller]

Steven,

Message sent.  I'm sure I'm missing something basic or fundamental in trying to set up this environment.  I've been able to build an ESXi host, but I can't get it to communicate on the network either.

[rmasteller]

Steven got my inband/outband issues cleared up, and as a bonus helped with ESXi management network issue I had (ESXi was grabbing vNIC0, vNIC1 and labeling them as 2 and 3, and vice versa).  Thanks Steven!!!

[Steven Tardy · Accepted solution]

Thanks for the quick Webex. A few issues.

1. CIMC service profile IP was trying to use "OOB" (through FI mgmt port) instead of "In-band". Changed CIMC service profile IP from OOB to In-band and CIMC was ping-able.
2. ESXi was inaccessible due to VMware choosing a random vNICs instead of the UCS vNIC order. Checking MAC addresses to match UCS vNICs to ESXi vmnics. Simply changing to use vmnic2/3 instead of vmnic0/1 within ESXi got ESXi ping-able.

Happy to help.