Cisco Nexus 7010(NX-OS 6.1) not forwarding DHCP requests/responses across SVIs after power failure

[pvs]

We have Nexus 7010 switch configured as data center(DC) switch with various VLANs & SVIs.

DHCP server (run in VLAN 10) is connected to another layer 2 switch(3560) uplinked to Nexus 7010.

All our clients sitting in other VLANs get IP Address from this DHCP server (VLAN SVIs configured with dhcp relay)

Last weekend, all of a sudden due to power failure, Nexus Switch got powered off and came up by itself.

After which none of our clients are getting IP Address from DHCP server except the clients located in the VLAN10.

DHCP server functionality is fine & Nexus configuration is intact.

This Nexus switch is having dual supervisor engines configured in Active-Active state.

I am clueless where the things went wrong. Any help is greatly appreciated

[Georg Pauwen]

Hello,

hard to say what causes this. Do you have any logs, or debug output ? Can you post the full running configuration of our N7K ?

[balaji.bandi]

High level there is no communication between VLAN 10 to other vlan in terms of IP reachability.

1. For testing, configure 1 of the device on other VLAN with device has static IP and try to ping VLAN 10 Gateway and DHCP server is this works ?

2. make sure it was allowed all the VLAN between Switches.

3. check any Logs on DHCP Server., and Scope in DHCP Server for other VLAN.

post some configuration for our reference to guide correctly to understand the issue.

[pvs]

Systems accessibility/ping across VLANs is fine. No issues.

VLANs are allowed between switches

Enough IP addresses are available in DHCP server

[Georg Pauwen]

Hello,

chances are that this is somehow related to either the DHC server, or something in the path towards the DHCP server, rather than the Nexus. What are you using as DHCP server ? 

If this is just a one time thing related to an unscheduled reboot of the Nexus (something that should not happen too often, you might actually want to think of installing a redundant power supply system), troubleshooting might be hard...

[balaji.bandi]

is the DHCP Server also gone down due to power loss ?

what was the outcome when you setup a static IP for the user device and try to reach DHCP Server ?

[pvs]

Yes, DHCP server also went off during the Nexus outage.

I could ping/access DHCP server from static IPs from other VLANs

[balaji.bandi]

In that case you need to enable debug and see why DHCP not offering IP address.

can you post one of the SVI config wherre helper configured ?

[pvs]

interface Vlan10

ip address 192.168.10.1/24
ip router eigrp 123
ip passive-interface eigrp 123

ip dhcp relay address 192.168.40.2
ip dhcp relay address 192.168.40.10
description DHCP-Server-DNS
no shutdown

interface Vlan11
ip address 192.168.11.1/24
ip router eigrp 123
ip passive-interface eigrp 123
ip dhcp relay address 192.168.40.2
ip dhcp relay address 192.168.40.10
description HR
no shutdown

interface Vlan12
ip address 192.168.12.1/24
ip router eigrp 123
ip passive-interface eigrp 123
ip dhcp relay address 192.168.40.2
ip dhcp relay address 192.168.40.10
description Finance
no shutdown

[Christopher Hart]

Hello!

We can use a tool on the Nexus platform called "Ethanalyzer" to assist us with troubleshooting this issue. Ethanalyzer is essentially a packet capture performed on the supervisor engine of the chassis that will allow us to inspect control-plane traffic observed by the supervisor. DHCP traffic that ingresses the Nexus 7000 via the data plane should hit a redirect ACL installed in TCAM, which will punt the DHCP traffic to the supervisor so that the control plane can relay the DHCP traffic to one or more DHCP servers. Because the DHCP traffic is punted to the control plane, we should see DHCP traffic in an Ethanalyzer packet capture.

A detailed guide on Ethanalyzer can be found in this article. Specifically, you can configure these capture-filter options to only capture DHCP traffic.

To troubleshoot this issue, I recommend the following high-level procedure:

1. Identify the MAC address of one single host that cannot receive an IP address via DHCP.
2. Use Wireshark or some other packet capture utility to verify that this host is sending DHCP Discover packets to the network.
3. Use Ethanalyzer to verify whether the DHCP Discover packets from this specific host are being punted to the control plane.

If the DHCP Discover packets are being punted to the control plane, you can also use the same Ethanalyzer capture-filter provided above to verify whether the DHCP server is responding to DHCP Discover packets as expected. You should see a DHCP Offer packets from the DHCP server in the Ethanalyzer capture. If you don't, it might be worth performing a packet capture via Wireshark on the DHCP server to verify it receives the DHCP Discover packet.

I hope this helps - please do update us with what you find!

Thank you!

-Christopher

[pvs]

Hi Christopher,

As suggested, I used ethanalyzer to capture traffic relating to DHCP using the command 

switch#ethanalyzer local interface inband capture-filter "port 67 or port68"

I could not find any packets.

switch#ethanalyzer local interface inband capture-filter "host 192.168.46.45"

Using the above command i could see traffic relating to internet access as i was browsing internet

Used wireshark to capture the traffic of DHCP server, and found no DHCP discover packets touching the DHCP server.

I powered off Cisco Nexus 7000 and powered on again, no use.

My DHCP server is perfectly working fine within the same VLAN and issues IPs to the clients.

Issue is with the clients located in other VLANs and trying to get DHCP IP address

Thanks & Regards,

Vishnu Sankar

[balaji.bandi]

Just observed below :

As you mentioned VLAN 10 works, DHCP Server not belong to VLAN 10 subnet, what VLAN is that belong to ?

i do not suspect any configuration here, Looks some VLAN information missing over path

After verified all vlan allowed in the path, Debug is the option for to identify the issue.

[pvs]

Some typing mistakes in my earlier mail. DHCP server (IP Address 192.168.10.2(primary) & 192.168.10.10(secondary))  is in VLAN 10 with the following configuration. Other VLANs are 11 & 12. There are nearly 200 SVIs configured on this switch

interface Vlan10

ip address 192.168.10.1/24
description DHCP-Server-DNS
no shutdown

interface Vlan11
ip address 192.168.11.1/24
ip dhcp relay address 192.168.10.2
ip dhcp relay address 192.168.10.10
description HR
no shutdown

interface Vlan12
ip address 192.168.12.1/24
ip dhcp relay address 192.168.10.2
ip dhcp relay address 192.168.10.10
description Finance
no shutdown

[pvs]

Some typing mistakes in my earlier mail. DHCP server (IP Address 192.168.10.2(primary) & 192.168.10.10(secondary))  is in VLAN 10 with the following configuration. Other VLANs are 11 & 12. There are nearly 200 SVIs configured on this switch

interface Vlan10

ip address 192.168.10.1/24
description DHCP-Server-DNS
no shutdown

interface Vlan11
ip address 192.168.11.1/24
ip dhcp relay address 192.168.10.2
ip dhcp relay address 192.168.10.10
description HR
no shutdown

interface Vlan12
ip address 192.168.12.1/24
ip dhcp relay address 192.168.10.2
ip dhcp relay address 192.168.10.10
description Finance
no shutdown

[balaji.bandi]

is this single nexus or vPC ?

please post below command output :

show system internal access-list vlan 11

show system internal access-list vlan 11 input config