IP connectivity between Nexus and Palo Alto firewall

[jeff20]

I'm struggling to understand connectivity problem between a switch and firewall.  The Nexus is located between the Core switch and the edge firewall. It's purpose is to divert traffic via ITD for content filtering.  My problem right now is that I'm not able to ping from the Nexus to the Palo Alto. Both interfaces have an IP in the same network. 

When there is an IP on the Cisco side L3 interface the IP ARP looks like below.  I'm trying to ping 164.104.10.1 connected to Ethernet 1/1.  I can ping it if there is no IP configured on the Cisco interface, but when I put 164.104.10.2 on it, I can't ping and see the INCOMPLETE in the table.

IP ARP Table for context default
Total number of entries: 9
Address Age MAC Address Interface Flags
10.112.11.2 00:16:29 3cec.effa.dc98 Vlan3
10.112.11.3 00:16:29 3cec.ef62.a318 Vlan3
10.112.11.4 00:16:29 3cec.effa.dc1a Vlan3
10.112.11.5 00:16:29 3cec.effa.dc48 Vlan3
10.112.11.6 00:16:29 3cec.effa.dc88 Vlan3
10.112.11.7 00:16:29 3cec.effa.8eea Vlan3
10.112.11.8 00:16:29 3cec.effa.db5c Vlan3
164.104.10.130 00:03:10 8030.e0ba.a600 Vlan1
164.104.10.1 00:00:10 INCOMPLETE Ethernet1/1

[Reza Sharifi]

Are you planning on configuring the Nexus as layer 2 or layer 3? If the Nexus is located between the firewall and the core switch, I assume you just want to use it as a layer-2 device with a transit vlan spanning between the core, Nexus, and the firewall. Is that what you are trying to do?

HTH

[jeff20]

That's basically how I had it set up before, but Nexus is running ITD to divert traffic for content filtering, so I needed to have separate vlans.  The interface on the firewall is L3, so it seemed that I needed to make the Cisco interface L3 (no switchport), and it was working like that.  But I needed to fail over to the other firewall and traffic stopped running through the Cisco, and when I failed back the default route which depends on the Cisco IP to be avaialble, I found I couldn't ping it.

I'm attaching a diagram and the Cisco config if you want to take a look.  Thanks for your reply!

[Reza Sharifi]

Thank you for the diagram. It is very helpful. The design between the PAs and the Junipers looks correct, as you only need one IP on the firewalls since they are clustered. I am not familiar with iBoss-switch but why does the core switch connects to iBoss-switch with one uplink while the other goes directly to the firewalls?

HTH

[jeff20]

The iBoss switch is the Cisco Nexus.  The PAs are in an Active/Passive configuration.  So the idea is that when we failover to the 2nd firewall, the traffic to the Internet bypasses the Cisco and goes straight to the firewall from the Core.  This is how we're operating now.  It's failing back that's giving me trouble, because the 164.104.10.1 IP doesn't give a reply if I have 164.104.10.2 on the Cisco's connected interface.  However, it was working before failover.

[Reza Sharifi]

10.1 and 10.2 IP are connecting the active firewall to the Nexus. So, when you fail that connection, the active firewall on the 5220-1 will switch the traffic forwarding to 5220-2 but the traffic from the core switch gets blackholed because the core still sees the 10.129 as an active connection, and when traffic gets to Nexus, it has nowhere to go because the link to the firewall is down. 

What if you remove the uplink from the core to the firewall and connect it to the nexus instead? So the Nexus to both firewalls using one transit vlan/subnet (164.104.10.0/25) and then connect the core with 2 uplinks to the Nexus as well using a separate transit vlan/subnet (164.104.10.128/25) and test the failover?

HTH

[jeff20]

Right, that makes sense to me.  Thank you for understanding the scenario. We did discuss runnning both links through the Nexus, but wanted to be able to bypass the iBoss filtering setup if needed.  But now that wer're using ITD, that might not really be necessary.

As for the routing issue, to address that problem we have 2 routes in the core with different metrics.  This has seemed to work before, though with hiccups.  I suppose we should be able to test the theory by just manually changing the static default route.

ip route 0.0.0.0 0.0.0.0 164.104.10.1 metric 2
ip route 0.0.0.0 0.0.0.0 164.104.10.129 metric 3

[Reza Sharifi · Accepted solution]

I agree. The static default route should work fine. If you make the design change, the core will only have one static default route pointing to .129. Also, with this design change, the hiccups should go away, but testing will tell.

HTH

[MHM Cisco World]

164.104.10.2 <<- referring to you topoly why this IP appear twice ?

MHM

[jeff20]

The PAs operate as Active/Passive, so only one is passing traffic at a time. The PAs configurations stay in sync so being able to leave the static routes pointing to 164.104.10.2 is helpful. We've discussed using different IPs and configuring route monitoring to change the gateway dynamically, but this had been working.

[MHM Cisco World]

ethanalyzer local interface inband capture-filter "host 0.0.0.0"
do this in Nexus 
and share result here 

MHM

[jeff20]

Thanks, I'm trying that but not getting any output.  It's the host 0.0.0.0 I want to use?  Sorry, I'm not experienced on Cisco.

[MHM Cisco World]

from palo ping to Nexus VLAN SVI or interface IP 
then do 
switch# ethanalyzer local interface inband capture-filter "host 100.0.0.2" <<- here the 100.0.0.2 as example of what palo interface use

MHM

[jeff20]

Okay, thanks.  I used this format:  ethanalyzer local interface inband capture-filter "host 164.104.10.1" while pinging from the PA.  But I am logged into the management interface of the PA.

1 2024-04-26 10:18:09.513568027 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
2 2024-04-26 10:18:11.513656089 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
3 2024-04-26 10:18:15.513688883 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
4 2024-04-26 10:18:23.513768013 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
5 2024-04-26 10:18:39.576200271 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
6 2024-04-26 10:18:41.576446641 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
7 2024-04-26 10:18:45.576412469 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
8 2024-04-26 10:18:53.576351232 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
9 2024-04-26 10:19:09.633216248 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
10 2024-04-26 10:19:11.633434277 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2

If I source the ping from the PA as coming from 164.104.10.1, I get no replies and the output looks like this:

1 2024-04-26 10:24:54.812523482 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
2 2024-04-26 10:25:10.915329003 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
3 2024-04-26 10:25:12.915462758 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
4 2024-04-26 10:25:16.915486598 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
5 2024-04-26 10:25:24.915653455 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
6 2024-04-26 10:25:41.013636360 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
7 2024-04-26 10:25:43.013748037 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
8 2024-04-26 10:25:47.013805178 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
9 2024-04-26 10:25:55.013977427 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2
10 2024-04-26 10:26:11.216308195 80:27:6c:36:0e:87 → ff:ff:ff:ff:ff:ff ARP 60 Who has 164.104.10.1? Tell 164.104.10.2

[MHM Cisco World]

show ip interface brief 

check if the VLAN SVI is UP/UP