Solved: Configure ESXi network connectivity through NIM E-Series NCE interfaces

leam2 · ‎06-06-2018

Hello.

I have an ISR4331 with a "NIM E-Series NCE" (UCS-EN140N-M2) module.

- Hardware information for the "NIM E-Series NCE" states that, as far as internal network interfaces are considered, it has two gigabit Ethernet interfaces.

- On the ISR4331 with the "NIM E-Series NCE" installed, there are two interfaces "ucse0/1/0" and "ucse0/1/1".

(This question might look stupid and/or irrelevant but:)

I am wondering why there are two network interfaces.

Are they similar?

Why not only one interface?

Do I need to configure both interfaces or just only one (like on a computer which would have two NICs and only one of them would be in service)?

I have and "ESXi host" installed on the "NIM E-Series NCE".

I see it as another machine in my LAN 192.168.X.0/24.

So, to me, it should have an IP address like 192.168.X.Y like any other machine on the LAN.

And it should be able to ping not only the other hosts on the LAN 192.168.X.0/24 but also external hosts.

I am trying to configure ESXi network connectivity as described above through the "NIM E-Series NCE" interfaces but I cannot find a configuration that works.

Mostly because the connections schema for the three elements "ISR4331 - NIM E-Series NCE - ESXi host" is quite esoteric to me and I can't find a proper guide to do it.

Can you explain how these connections are set up and maybe propose a configuration to enable this ESXi network connectivity?

Thank you in advance.

leam2 · ‎06-13-2018

Hello.

Setting "ip nat inside" on interface "ucse 0/1/0" solved my issue.
Best regards.

View solution in original post

Niko Nikas · ‎06-07-2018

Hello,

I don't specifically work on the E-series, so I can only really shed light on the ESXi configuration piece.

With two network interfaces we get some redundancy in case of a network failure, however to utilize this we will need to make sure that from an ESXi perspective that both of these interfaces belong to the same vSwitch or DVS (distributed virtual switch). There is also the option of having the interfaces on separate vSwitches/DVSs to keep specific traffic separate, but with only two interfaces there would be no redundancy per switch.

There's a few options on how to have these configured, but typically you will see Active-Active (so both interfaces are passing traffic load balanced between the two). Should one fail in this case, then the traffic on that interface is shifted over to the remaining interface.

Hopefully this helps at least get you on the right track.

Let me know if there's anything I can clarify for you.

--

Niko

leam2 · ‎06-07-2018

Hello.

Thank you for the explanation you gave me.

Concerning the other part of my questioning:

From the ESXi shell, do you think that, "normally", with the "appropriate" configuration, I should be able to reach the Internet (for example ping 8.8.8.8 (which is a Google's pingable public IP address))?

I am wondering if maybe, the ESXi host firewall is blocking that traffic or not.

I am not especially willing to allow that traffic if it's not a recommended practice, but I would like to understand which element in the network is blocking that traffic to be sure it's not some other configuration I did that is responsible for that traffic being dropped.

The command "esxcli network firewall ruleset list" shows a list of rules, is there one which blocks the traffic from the ESXi host to the Internet?

Thank you for your help.

Best regards.

Niko Nikas · ‎06-07-2018

Hello,

I would expect so, do you know if your host is able to ping other devices in your network (so not external/public), but in a different subnet?

Are you able to reach the gateway for that ESXi host from your laptop?

--

Niko

leam2 · ‎06-08-2018

Hello.

ninikas> I would expect so

Ah, ok, so the ESXi host should be able to ping a public pingable IP like 8.8.8.8.

ninikas> do you know if your host is able to ping other devices in your network (so not external/public), but in a different subnet?

Yes, it can. It is on subnet 192.168.X.0/24 and it can ping different hosts on subnet 192.168.X2.0/24.

ninikas> Are you able to reach the gateway for that ESXi host from your laptop?

I suppose, given the result of the command below, that 192.168.X.1 is the "gateway for that ESXi host" you are talking about?

esxi-host# esxcfg-route
VMkernel default gateway is 192.168.X.1

I'm not sure what laptop you are talking about.

So I'll suppose this is the laptop from which I connect to the ESXi host via SSH.

So the ESXi host IP being 192.168.X.Y, the laptop IP being 192.168.X.Z, I can ping 192.168.X.1 (gateway) from 192.168.X.Z (laptop) and also from the ESXi host (192.168.X.Y).

Thank you for helping me.

Best regards.

leam2 · ‎06-08-2018

Hello.

I am trying to find out where packets are dropped.

"isr4331" is the Cisco ISR in which there is the NIM E-Series NCE module on which the VMware ESXi host is installed.

192.168.X.Y is the ESXi host IP.

192.168.X.1 is the gateway on the "isr4331".

isr4331# debug platform condition ipv4 192.168.X.Y/32 both
isr4331# debug platform condition start
isr4331# debug platform packet-trace packet 1024
isr4331# debug platform packet-trace enable

I execute a ping from the ESXi host:

esxi_host# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

I try to check if packets were dropped on the ISR:

isr4331# show platform packet-trace statistics
Packets Summary
  Matched  75
  Traced   75
Packets Received
  Ingress  75
  Inject   0
Packets Processed
  Forward  75
  Punt     0
  Drop     0
  Consume  0

It looks like no packet was dropped on the ISR.

I try to check if packets were dropped on the ESXi host:

esxi-host# ethtool -S vmnic0 | grep -i drop
     dropped_smbus: 0
     tx_dropped: 0
     rx_queue_0_drops: 0

It looks like no packet was dropped on the ESXi host.

Still the ESXi host is not receiving echo replies from the host with external public pingable IP address 8.8.8.8

Please help me find out where the problem is coming from.

Best regards.

leam2 · ‎06-08-2018

Hello.

I just noticed I cannot ping 8.8.8.8 from CIMC (Cisco Integrated Management Controller) either:

EN140N-FOC22022YY0 /cimc/network # ping 192.168.X.1
Press CTRL+C to stop.
PING 192.168.X.1 (192.168.X.1): 56 data bytes
64 bytes from 192.168.X.1: seq=0 ttl=255 time=4.000 ms
64 bytes from 192.168.X.1: seq=1 ttl=255 time=2.000 ms
64 bytes from 192.168.X.1: seq=2 ttl=255 time=2.000 ms

--- 192.168.X.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 2.000/2.666/4.000 ms

I can ping the gateway (192.168.X.1).

EN140N-FOC22022YY0 /cimc/network # ping 8.8.8.8
Press CTRL+C to stop.
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
Hostname/IP Address 8.8.8.8 is not reachable

I can't ping the public pingable IP address 8.8.8.8

Is it normal?

Should I be able to ping a public pingable IP address like 8.8.8.8 from the NIM E-Series NCE CIMC?

Thank you for your help.

Niko Nikas · ‎06-08-2018

Hello,

This sounds like the issue is farther upstream from the ESXi host/CIMC.

If these devices are able to ping their gateway and other devices in your network, but in a different subnet, then I expect they are working fine. So you would want to focus your efforts at or beyond the gateway.

Are you able to ping the 8.8.8.8 address from your local machine?

--

Niko

leam2 · ‎06-08-2018

Hello.

ninikas> Are you able to ping the 8.8.8.8 address from your local machine?

Yes, I can.

Thank you.

Niko Nikas · ‎06-08-2018

Hello,

Alright, so then we may want to focus our efforts on the path from the ESXi host's gateway to the 8.8.8.8 address. Do you know what route that takes through your network?

Somewhere along that path the packet is being lost/dropped.

If you perform a packet capture on the device that connects your network out to the internet, do you see the pings coming in from the ESXi host? Do you see the response from 8.8.8.8?

--

Niko

leam2 · ‎06-08-2018

Hello.

I think traffic goes through these interfaces:

vmnic0
on ESXi host
->
ucse 0/1/0
on ISR4331                 
equivalent to (Nim E-Series NCE) UCSE GE0     
->
Gi0/0/1.1
on ISR4331
->
Dialer 0
on ISR4331
->
Internet

ninikas> If you perform a packet capture on the device that connects your network out to the internet

This is the ISR4331 which is connected to the Internet via its interface Gi0/0/0.835.
I am interested in learning how to do such a packet capture.
Can you tell me how to?

Thank you for your help.

leam2 · ‎06-12-2018

Hello.

I execute the command below.

I do a ping 8.8.8.8 from the esxi_host.

I do a Ctrl+C.

esxi-host# pktcap-uw --vmk vmk0 --proto 0x01 --capture PortOutput
The name of the vmk is vmk0
The session filter IP protocol is 0x01
The session capture point is PortOutput
No server port specifed, select 39914 as the port
Output the packet info to console.
Local CID 2
Listen on port 39914
Accept...Vsock connection from port 1027 cid 2
Join with dump thread failedDestroying session 3

Dumped 0 packet to console, dropped 0 packets.
Done.

I execute the command below.

I do a ping 8.8.8.8 from the esxi_host.

I do a Ctrl+C.

esxi-host# pktcap-uw --vmk vmk0 --proto 0x01 --capture PortInput
The name of the vmk is vmk0
The session filter IP protocol is 0x01
The session capture point is PortInput
No server port specifed, select 39920 as the port
Output the packet info to console.
Local CID 2
Listen on port 39920
Accept...Vsock connection from port 1028 cid 2
12:40:51.145595[1] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 98.
        Segment[0] ---- 98 bytes:
        0x0000:  682c 7b74 b498 00f6 63b9 6232 0800 4500
        0x0010:  0054 2d78 0000 4001 793b c0a8 033e 0808
        0x0020:  0808 0800 aa88 0f9c 0000 5b1f bf53 0002
        0x0030:  3863 0809 0a0b 0c0d 0e0f 1011 1213 1415
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
        0x0060:  3637
12:40:52.147445[2] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 98.
        Segment[0] ---- 98 bytes:
        0x0000:  682c 7b74 b498 00f6 63b9 6232 0800 4500
        0x0010:  0054 2d7c 0000 4001 7937 c0a8 033e 0808
        0x0020:  0808 0800 a33c 0f9c 0001 5b1f bf54 0002
        0x0030:  3fad 0809 0a0b 0c0d 0e0f 1011 1213 1415
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
        0x0060:  3637
12:40:53.149559[3] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 98.
        Segment[0] ---- 98 bytes:
        0x0000:  682c 7b74 b498 00f6 63b9 6232 0800 4500
        0x0010:  0054 2d7d 0000 4001 7936 c0a8 033e 0808
        0x0020:  0808 0800 9af4 0f9c 0002 5b1f bf55 0002
        0x0030:  47f3 0809 0a0b 0c0d 0e0f 1011 1213 1415
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
        0x0060:  3637
Join with dump thread failedDestroying session 4

Dumped 3 packet to console, dropped 0 packets.
Done

Does it give an indication?

Best regards.

leam2 · ‎06-13-2018

Hello.

Setting "ip nat inside" on interface "ucse 0/1/0" solved my issue.
Best regards.