Simple IP Routing not working correctly

[Uplink90]

   Hello,

   I have what I think to be a simple configuration but for some reason routing is not working properly. The configuration is:

[Firewall] 10.10.10.1 (10.10.10.0/24) <-> [L3 Switch Interface] 10.10.10.2 (10.10.10.0/24)

   Config on the L3 Interface is:

no switchport

ip address 10.10.10.2 255.255.255.0

   The switch has multiple VLans, 10.10.20.0/24, 10.10.30.0/24, 10.10.40.0/24. All VLans have a respective SVI at 10.10.x.1.

In the config I have:

IP Routing

IP Route 0.0.0.0 0.0.0.0 10.10.10.1

   Running a sh ip route displays '0.0.0.0 via 10.10.10.1' as well as all VLans correctly listed and the L3 Interface '10.10.10.0 is directly connected at Interface'.

   Running a sh arp displays all of the SVIs and both 10.10.10.1 and 10.10.10.2.

   Devices on all VLans can ping each other successfully which shows the IP Routing is functioning. They can also ping 10.10.10.2 so I know they are getting to that network. They can NOT ping 10.10.10.1 which makes no sense. The Firewall and Switch can ping each other but the Firewall can NOT ping any of the SVIs. I replaced the Firewall with a test PC at the same address and even made 10.10.10.2 the default gateway on the PC and it can not get to the SVIs.

   On the switch if I do 'ping 10.10.10.1' it is successful, but doing that with a source of any SVI address, the ping fails.

   There is no ACL on the 10.10.10.2 interface. As a test I even tried creating one that was 'permit ip any any' and applied it to the Interface both in and out with no luck. Something at the Interface still seems to be blocking the traffic. What could I be missing? It's pretty much a by-the-book config.

   Thanks in advance!

[Seb Rupik]

Hi there,

the firewall which has the interface 10.10.10.1 does not know how to reach the L3 VLANs on the switch, just the directly attached interface.

You need to either configure static routes on the firewall for each VLAN with a next hop of 10.10.10.2, or preferably run an IGP between the firewall and switch.

cheers,

Seb.

[Uplink90]

Thanks for the quick response Seb! Unfortunately, that's not the issue. I forgot to mention that when I replaced the Firewall with a PC with a gateway address of the L3 Interface (which should send all traffic that way regardless of VLan list) I also tested adding forced route statements:

route add 10.10.10.20.0 mask 255.255.255.0 10.10.10.2

route add 10.10.10.30.0 mask 255.255.255.0 10.10.10.2

route add 10.10.10.40.0 mask 255.255.255.0 10.10.10.2

   and the PC was still unable to get to the L3 VLans.

   That also doesn't explain why traffic from the SVIs can't get out to the GW address even though they can get to the GW network. I further validated the test PC by connecting it to the various VLans with proper IP info and it was pingable back and forth so there is nothing on the device preventing communication.

   To simplify, PC-A 10.10.20.5 and PC-B 10.10.30.5 can ping each other with no issue. They can also both ping 10.10.10.2 with no issue. Communication and routing is working.

   Now re-IP PC-B to 10.10.10.5 and connect it to the L3 Interface and it can no longer communicate with PC-A and PC-A can no longer communicate with it (regardless of forced static route settings on the PCs).

[MHM Cisco World]

there two are option 
A-
1- disable ip routing 
2- run default-gateway toward the ASA IN interface 
Note:- option A you can not config port as router port, config SVI and connect ASA to port with same VLAN of SVI you config
B-
1- run ip routing 
2- run ip route 0.0.0.0 toward ASA IN inteface in L3SW
3- run route IN <VLAN IP> toward the <router port of L3SW>

[Uplink90]

Thanks MHM! Just to clarify, it's not an ASA, it's a SonicWall but the switch is a 9200.

Under those 2 options:

A- Disabling IP Routing breaks the ability for the devices on the various VLans to communicate with each other on the switch backplane through Inter-VLan routing. That would shift all L3 routing to the Firewall over a single 1G port for every VLan in the company. Not ideal.

B- I did steps 1 & 2 in my initial configs above. I also did step 3 with network objects on the SonicWall and alternatively via static route statements on the test laptop. No success.

   Something is happening where Inter-VLan routing is working perfectly fine internally, including getting to the L3 Interface IP from any SVI/VLan but not anywhere else on the L3's network. I've even cleared my config and tried building the example here: https://study-ccna.com/layer-3-switch-intervlan-routing/ with no luck.

   It almost seems like there's some other unrelated component that's preventing the traffic. I even went down the rabbit hole of creating an OSPF router on the 9200 and grouping all the networks that way. Still no go.

[KJK99]

@Uplink90 

„Devices on all VLans can ping each other successfully which shows the IP Routing is functioning. They can also ping 10.10.10.2”

Perfect

“I replaced the Firewall with a test PC at the same address and even made 10.10.10.2 the default gateway on the PC and it can not get to the SVIs.”

That should work and it should work without any addition routes since 10.10.19.2 is the default gateway on the PC. If it does not work, there must be a problem with the link. Maybe that’s another case of 'Auto Smartport' playing tricks?

Unless you do some tracing, you can’t be sure that nothing comes out of that L3 port. All you can say is that you do not receive replies to your pings. I think Seb Rupik gave you a good advice. Those static routes are necessary.

[Uplink90]

Thanks Kris! I was using ping as the basic test to keep the message post simple but mapping drives doesn't work either. Also, if I put an IP helper on the L3 Interface, devices in the 10.10.10.0/24 will not get addresses but devices on any other VLan with the helper will get their proper DHCP addressing. When I used the test PC on 10.10.10.0/24 I had already added the static routes which is what Seb suggested and it didn't fix the problem.

   I like your idea about the Auto Smartport. It really feels like this is an issue outside the basic routing commands. I'm going to try unboxing another switch and see if it has the same issue with these configs. That may isolate it to something else in the IOS.

[paul driver]

Hello
Try the following:

L3 switch:
default interface x/x
interface x/x
description link to FW
switchport host
switchport access vlan 10
no shut

int vlan 10
ip address 10.10.10.2 255.255.255.0
no shut
exit

[Uplink90]

Thanks Paul! Great tip on that switchport host command for a port reset! Unfortunately this didn't resolve the issue either. After running those commands the config for the port and VLan are:

Interface GigabitEthernet1/0/37

switchport access vlan 10

switchport mode access

spanning-tree portfast

end

interface Vlan10

ip address 10.10.10.2 255.255.255.0

end

   Devices attached to any switch VLan can still hit the 10.10.10.2 address (which is now an SVI) but traffic is still not able to access 10.10.10.1 and traffic from 10.10.10.1 can only get as far as 10.10.10.2 even with static routes assigned. I've also tried doing this on an alternate port on the off chance there was a problem with this physical link and used different known good Cat5e cables. No luck. Next step is trying on a different switch.

[MHM Cisco World]

please do tracereoute see if the next-hop is FW or not 

[Uplink90]

From the CLI default a 'traceroute 8.8.8.8' goes through the FW out the far side and resolves correctly, so does a ping. So L3 Interface can get out to the Internet and recognizes that route. However, if I set the source to any SVI 'traceroute 8.8.8.8 source 10.10.20.1' all I get is * * * on every line and it never resolves the next hop.

I have also taken a new switch out, done a 'factory-reset config' and put in just the basic commands for this setup. Exact same results.

[MHM Cisco World]

Config defualt-gw with ip of FW and check result 

I see same issue before that SW use defualt gateway not default route 

[paul driver]

Hello
This does seem strange, especially given what you say about attaching a pc to the same port as the firewall , I assume that pc had any software fw disabled for the ping?

Can you post the output of the below please, and have you tried as different switchport/cable  etc...?

access-list 110 permit ip host 10.10.10.1 host 10.10.10.2
debug ip packet detail 110

[Uplink90]

I need to take a moment to thank everyone for their help and input! Your troubleshooting tips were helpful, a great sanity check, and very much appreciated!

Sadly, this may be one of the mysteries of the unexplained forever. I took out a 3rd switch, did a 'factory-reset config' input the same base settings and everything worked perfectly. I copied the full config over to the other 2 switches and tried them independently... they now work! I have absolutely NO idea what could have been the hangup. Nothing was changed on the Firewall or the Test PC, only the switch config being rewritten. It can now ping out to the Internet and traffic comes back in as expected from any SVI and any attached device. I even tested attaching a second switch to the first and traffic travels correctly over the trunk, routes to the L3 Interface out to the Internet and back. {Happy Dance}

Thank you all again!

[MHM Cisco World]

You are so so welcome

Have a nice day