3750x DHCP issues

[tyler.perkey]

Hi all,

I ran into a weird issue while making some network changes last night.  We have a 3750x connected to a firewall on a flat network, devices on Vlan1 get an IP from the firewall and everything works fine.  Last night I created a new vlan to connect the switch and firewall together, Vlan 100, and left all other devices on vlan 1.  After this change DHCP would not work for any device on Vlan1, regardless if I had the pool on the firewall and use an IP Helper on the vlan interface or if I set up a local DHCP pool on the switch. 

We have a few other vlans on the switch, if I moved any vlan 1 port to another vlan and shut/no shut the port the device would pull an IP from the firewall no issues.  I even attempted to shut down vlan 1, move it's whole network to a new vlan I created and assign ports to the new vlan.  I ran debugs on dhcp events, I would see messages like this:

cisco DHCPD: no subnet configured for 10.10.20.51 (this network doesn't exist anywhere, no idea where this IP came from)

DHCPD: due to: CLIENT CHANGED SUBNET

Another odd thing I noticed is very rarely some device would get an IP if the pool was on the switch, but the lease was always set to expire in 5min, no matter what lease time I configured on the pool.

[balaji.bandi]

can you post the configuraiton of switch show run and also what FW is this ?

suggest to clear ARP and test again.

[tyler.perkey]

Here is the switch config on firmware c3750e-universalk9-mz.150-2.SE11.bin:

ip routing
no ip cef optimize neighbor resolution
ip dhcp excluded-address 10.100.220.1 10.100.220.100
ip dhcp excluded-address 10.100.220.250 10.100.220.254
ip dhcp excluded-address 10.10.6.100 10.10.6.250
ip dhcp excluded-address 192.168.22.1 192.168.22.99
ip dhcp excluded-address 192.168.22.200 192.168.22.254
!
ip dhcp pool VOICE
network 10.10.6.0 255.255.255.0
default-router 10.10.6.1
dns-server 8.8.8.8

!
ip dhcp pool SECURITY
network 10.100.220.0 255.255.255.0
dns-server 8.8.8.8

default-router 10.100.220.1

spanning-tree mode pvst
spanning-tree loopguard default
spanning-tree portfast bpduguard default
spanning-tree extend system-id
!
!
!
!
!
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause security-violation
errdisable recovery cause channel-misconfig (STP)
errdisable recovery cause pagp-flap
errdisable recovery cause dtp-flap
errdisable recovery cause link-flap
errdisable recovery cause sfp-config-mismatch
errdisable recovery cause gbic-invalid
errdisable recovery cause l2ptguard
errdisable recovery cause psecure-violation
errdisable recovery cause port-mode-failure
errdisable recovery cause dhcp-rate-limit
errdisable recovery cause pppoe-ia-rate-limit
errdisable recovery cause mac-limit
errdisable recovery cause vmps
errdisable recovery cause storm-control
errdisable recovery cause inline-power
errdisable recovery cause arp-inspection
errdisable recovery cause loopback
errdisable recovery cause small-frame
errdisable recovery cause psp
errdisable recovery interval 30
!
!
!
!
vlan internal allocation policy ascending
!
lldp run

interface Vlan1
ip address 192.168.22.1 255.255.255.0
!
interface Vlan6
description Voice
ip address 10.10.6.1 255.255.255.0
!
interface Vlan100
ip address 10.100.22.1 255.255.255.0
!
interface Vlan220
description Security
ip address 10.100.220.1 255.255.255.0
!
interface Vlan997
ip address 192.168.197.1 255.255.255.0
ip helper-address 10.100.22.2
!
router ospf 1
redistribute connected subnets
network 10.10.6.0 0.0.0.255 area 0
network 10.100.22.0 0.0.0.255 area 0
network 10.100.220.0 0.0.0.255 area 0
network 192.168.22.0 0.0.0.255 area 0
!
ip http server
ip http secure-server
!
!
ip sla enable reaction-alerts
!

!
!
!
vstack

!
line con 0
line vty 0 4
exec-timeout 0 0
password 7 04760C0B0825401C04
length 0
history size 256
transport input telnet ssh
line vty 5 15
password 7 07222641490D15571A
!
ntp server 24.56.178.140
ntp server pool.ntp.org
ntp peer 192.168.22.254
end

3750_CORE#

[Georg Pauwen]

Is that an ASA dishing out the DHCP addresses ?

[tyler.perkey]

It's actually a sonicwall firewall.  We have this same layout at multiple sites, this is the first one to have this kind of behavior.  All other networks seem to function right, but when I put the stub network between the switch and firewall the 192.168.22.x network will not process DHCP request regardless where the pool is.  Other networks behave fine. 

[Georg Pauwen]

Hello,

what is the physical port connecting the 3750x to the SonicWall configured like ? 

[paul driver]

Hello
You dont say how the fw and the 3750 are physically connected, routed port, routed svi, or trunk.
Is the Fw running OSPF or do you have static routes pertaining to vlan 1 on there.

Suggest remove dhcp from the 3750 as the Fw is servicing dhcp, and also the helper address from vlan 997 

Lastly looking at your ospf stanza as you are locally advertising the connected interfaces so not sure why you are redistributing them also and i dont see a network statement of vlan 997 either.

Post the following from the 3750:
sh ip protocols
sh ip ospf neighbors
sh ip ospf inter brief
sh ip arp
sh ip route

[tyler.perkey]

So the switch had to be reverted to the flat network, but below is the output.  The firewall and switch are connected via routed SVI, the switch and firewall are OSPF peers.  We often use "redistribute connected subnets" incase a new network is added and someone forgets to update the OSPF route statements, as is the case with vlan 997 here.  It really feels like this is a bug in the switch IOS because everything else works, until leases expire 192.168.22.xx has full network and internet access, all other subnets can get DHCP regardless of the DHCP server location.  It's just an issue with network 192.168.22.x getting DHCP.

sh ip prot
*** IP Routing is NSF aware ***

Routing Protocol is "ospf 1"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 192.168.197.1
It is an autonomous system boundary router
Redistributing External Routes from,
connected, includes subnets in redistribution
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
10.10.6.0 0.0.0.255 area 0
10.100.22.0 0.0.0.255 area 0
10.100.220.0 0.0.0.255 area 0
192.168.22.0 0.0.0.255 area 0
Routing Information Sources:
Gateway Distance Last Update
192.168.22.254 110 19:51:33
Distance: (default is 110)

sh ip ospf neigh

Neighbor ID Pri State Dead Time Address Interface
192.168.22.254 1 FULL/BDR 00:00:35 192.168.22.254 Vlan1

sh ip ospf int brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Vl1 1 0 192.168.22.1/24 1 DR 1/1
Vl220 1 0 10.100.220.1/24 1 DR 0/0
Vl100 1 0 10.100.22.1/24 1 DR 0/0
Vl6 1 0 10.10.6.1/24 1 DR 0/0

sh ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.10.6.1 - 00da.55bd.8941 ARPA Vlan6
Internet 10.10.6.2 0 805e.c04f.0671 ARPA Vlan6
Internet 10.10.6.3 0 805e.c056.aa32 ARPA Vlan6
Internet 10.10.6.4 0 805e.c056.b672 ARPA Vlan6
Internet 10.10.6.5 0 805e.c056.ab90 ARPA Vlan6
Internet 10.10.6.6 0 805e.c056.aec5 ARPA Vlan6
Internet 10.10.6.7 0 805e.c056.aa04 ARPA Vlan6
Internet 10.10.6.8 0 805e.c056.b2da ARPA Vlan6
Internet 10.10.6.9 0 805e.c02f.bf1b ARPA Vlan6
Internet 10.10.6.10 0 805e.c052.3e23 ARPA Vlan6
Internet 10.10.6.11 1 805e.c056.aa1f ARPA Vlan6
Internet 10.10.6.12 1 805e.c056.b586 ARPA Vlan6
Internet 10.10.6.13 0 805e.c052.2fdd ARPA Vlan6
Internet 10.10.6.14 0 805e.c056.b41f ARPA Vlan6
Internet 10.10.6.15 0 805e.c056.aacc ARPA Vlan6
Internet 10.10.6.16 0 805e.c056.b2be ARPA Vlan6
Internet 10.10.6.17 0 805e.c056.ac45 ARPA Vlan6
Internet 10.10.6.18 0 805e.c056.b5f1 ARPA Vlan6
Internet 10.10.6.19 1 805e.c052.8ec0 ARPA Vlan6
Internet 10.10.6.20 0 805e.c056.abeb ARPA Vlan6
Internet 10.10.6.21 0 805e.c056.aa2e ARPA Vlan6
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.10.6.22 0 2829.860d.2ac7 ARPA Vlan6
Internet 10.10.6.23 0 0015.65b4.5dd8 ARPA Vlan6
Internet 10.10.6.24 0 805e.c056.a96f ARPA Vlan6
Internet 10.10.6.25 0 0015.65ac.58b0 ARPA Vlan6
Internet 10.100.22.1 - 00da.55bd.8942 ARPA Vlan100
Internet 10.100.22.2 0 Incomplete ARPA
Internet 10.100.22.10 201 2c01.b56a.5020 ARPA Vlan100
Internet 10.100.22.11 201 2c01.b56a.394c ARPA Vlan100
Internet 10.100.22.12 201 2c01.b56a.44e2 ARPA Vlan100
Internet 10.100.22.13 201 2c01.b56a.505c ARPA Vlan100
Internet 10.100.22.254 0 Incomplete ARPA
Internet 10.100.220.1 - 00da.55bd.8943 ARPA Vlan220
Internet 10.100.220.10 58 b42e.993a.1e8b ARPA Vlan220
Internet 10.100.220.11 56 b42e.9937.96d2 ARPA Vlan220
Internet 10.100.220.12 0 4447.cca6.befd ARPA Vlan220
Internet 10.100.220.13 1 142f.fd14.2934 ARPA Vlan220
Internet 10.100.220.14 0 4447.cca6.bfa0 ARPA Vlan220
Internet 10.100.220.15 1 142f.fd14.2344 ARPA Vlan220
Internet 10.100.220.16 1 142f.fd14.241d ARPA Vlan220
Internet 10.100.220.17 1 142f.fd14.2445 ARPA Vlan220
Internet 10.100.220.18 1 142f.fd14.2466 ARPA Vlan220
Internet 10.100.220.19 1 142f.fd14.23cc ARPA Vlan220
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.100.220.20 1 142f.fd14.2415 ARPA Vlan220
Internet 10.100.220.21 1 142f.fd14.2418 ARPA Vlan220
Internet 10.100.220.22 1 142f.fd14.244a ARPA Vlan220
Internet 10.100.220.23 1 142f.fd14.2767 ARPA Vlan220
Internet 10.100.220.24 1 142f.fd14.2406 ARPA Vlan220
Internet 10.100.220.25 1 142f.fd14.2440 ARPA Vlan220
Internet 10.100.220.26 1 142f.fd14.2434 ARPA Vlan220
Internet 10.100.220.27 1 142f.fd14.2425 ARPA Vlan220
Internet 10.100.220.28 1 142f.fd14.240f ARPA Vlan220
Internet 10.100.220.29 1 142f.fd14.2420 ARPA Vlan220
Internet 10.100.220.30 1 142f.fd14.24e0 ARPA Vlan220
Internet 10.100.220.31 1 142f.fd14.2423 ARPA Vlan220
Internet 10.100.220.32 1 142f.fd14.246d ARPA Vlan220
Internet 10.100.220.33 1 142f.fd14.2624 ARPA Vlan220
Internet 10.100.220.34 1 142f.fd14.243d ARPA Vlan220
Internet 10.100.220.35 1 142f.fd14.2428 ARPA Vlan220
Internet 10.100.220.36 1 142f.fd14.2435 ARPA Vlan220
Internet 10.100.220.37 1 142f.fd14.2458 ARPA Vlan220
Internet 10.100.220.38 1 142f.fd14.242b ARPA Vlan220
Internet 10.100.220.39 1 142f.fd14.2422 ARPA Vlan220
Internet 10.100.220.40 1 142f.fd14.241f ARPA Vlan220
Internet 10.100.220.41 1 142f.fd14.24df ARPA Vlan220
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.100.220.42 0 1c82.5910.2098 ARPA Vlan220
Internet 10.100.220.43 1 142f.fd14.241c ARPA Vlan220
Internet 10.100.220.44 1 4447.cc6e.d8e3 ARPA Vlan220
Internet 10.100.220.45 1 4447.cc6e.d4e9 ARPA Vlan220
Internet 10.100.220.46 1 142f.fd14.2622 ARPA Vlan220
Internet 10.100.220.47 1 142f.fd14.2664 ARPA Vlan220
Internet 10.100.220.48 1 142f.fd14.2649 ARPA Vlan220
Internet 10.100.220.49 1 142f.fd14.2419 ARPA Vlan220
Internet 10.100.220.50 32 b8a4.4f0a.15fb ARPA Vlan220
Internet 10.100.220.51 1 142f.fd14.27e1 ARPA Vlan220
Internet 10.100.220.52 1 686d.bc45.73dc ARPA Vlan220
Internet 10.100.220.53 1 142f.fd14.2921 ARPA Vlan220
Internet 192.168.22.1 - 00da.55bd.8940 ARPA Vlan1
Internet 192.168.22.117 3 9cc0.7700.a80a ARPA Vlan1
Internet 192.168.22.254 0 1ab1.69b3.730c ARPA Vlan1
Internet 192.168.197.1 - 00da.55bd.8944 ARPA Vlan997

sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is 192.168.22.254 to network 0.0.0.0

O*E2 0.0.0.0/0 [110/10] via 192.168.22.254, 19:56:14, Vlan1
10.0.0.0/8 is variably subnetted, 6 subnets, 2 masks
C 10.10.6.0/24 is directly connected, Vlan6
L 10.10.6.1/32 is directly connected, Vlan6
C 10.100.22.0/24 is directly connected, Vlan100
L 10.100.22.1/32 is directly connected, Vlan100
C 10.100.220.0/24 is directly connected, Vlan220
L 10.100.220.1/32 is directly connected, Vlan220
12.0.0.0/29 is subnetted, 1 subnets
O E2 12.xxx.xxx.xxx [110/20] via 192.168.22.254, 19:56:14, Vlan1
192.168.22.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.22.0/24 is directly connected, Vlan1
L 192.168.22.1/32 is directly connected, Vlan1
192.168.197.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.197.0/24 is directly connected, Vlan997
L 192.168.197.1/32 is directly connected, Vlan997

[confreg01]

The Cisco Suggested image is Catalyst 3750 Series Switch -Release 12.2.55-SE12 MD

https://software.cisco.com/download/home/282526529/type/280805680/release/12.2.55-SE12

Suggested

12.2.55-SE12

12.2SE

MD

09-Oct-2017

[balaji.bandi]

Not sure what is the point here, since this was an old thread.

That is cisco suggested on 12.X Code ( that is the final version Cisco released before end of support), there is no harm going to 15.X version which the device has already?

Here is the switch config on firmware c3750e-universalk9-mz.150-2.SE11.bin.

[tyler.perkey]

This issue remains unsolved.  I had to drop the plans to change the network until we get someone on site.