Intervlan communication issue

[grapevine]

I have a Meraki that has a SVI for vlan 5, 172.18.5.2 and it's trunk to a firewall that also has an SVI for vlan 5 172.18.5.1. There is a default route from Meraki pointing to 172.18.100.1 which is on the firewall.

Meraki has SVI  for vlan 2 172.18.2.1. Server 172.18.5.76 is unable to reach IDRAC 172.18.2.75 via https though ANY is allowed on firewall. I have limited access to Palo Alto.

I ran packet captures on Meraki switchports where firewall and IDRAC is connected, I see SYN and ACK but no SYN,ACK . Also on the switchport where IDRAC is connected, I see SYN and SYN,ACK but no ACK.

Can you advise how to fix this issue. 172.18.2.75 is accessible via https through VPN

Default gateway of 172.18.2.75 is 172.18.2.1 and default gateway of 172.18.5.76 is 172.18.5.1

[Jens Albrecht]

Hello @grapevine,

"I have a Meraki that has a SVI for vlan 5, 172.18.5.2 and it's trunk to a firewall that also has an SVI for vlan 5 172.18.5.1. There is a default route from Meraki pointing to 172.18.100.1 which is on the firewall."

If the connection from your Meraki switch to the firewall is vlan 5, then the default route on the Meraki switch should point to the firewall IP 172.18.5.1.

In general, the next hop for any route you configure must be a reachable IP address. If you point the default route to 172.18.100.1 but the Meraki switch has no SVI in this network, then this default route will never be used.

HTH!

[paul driver]

Hello
Is the PA applying any inter-area policy's? 
The reason i'm asking is the Idrac/prod nics are part of the same physical server- meaning the production nic 172.18.5.76 & the idrac nic 172.18.2.75 connect to the same meraki switch which is routing for both vlans so unless the PA FW is performing some security policy you should have reachability between both subnets?

Also beware , PAs by default have a implicit deny any any rule on interzones so if you have applied permit a FW rule it needs to be above that implicit deny rule 

[Royalty]

Hi,

Is this correct?

• Your server in VLAN 5 is using the default gateway of the VL5 SVI on the PA.
• iDRAC NIC (supposedly on the same server?) is using the default gateway of the VL2 SVI on the Meraki switch.

I'm making an assumption that 172.18.5.76 is connected to the same switch as 172.18.2.75, which could be wrong... My thoughts are that 172.18.5.76 could be sending SYN packets that traverse the firewall (uses default gateway of 172.18.5.1 aka PA firewall) and is allowed by the FW policy. 172.18.2.75 (iDRAC NIC) receives the SYN and sends a SYN-ACK in response. SYN/ACK goes to its default gateway of 172.18.2.1 which is the Meraki switch and is routed (as destination is directly connected) to 172.18.5.76. This SYN-ACK does not go through the FW as the switch can route the packet directly. 172.18.5.76 responds and sends an ACK which goes to the PA, again because the PA's IP is configured as the default gateway. The PA did not see the SYN-ACK because the SYN-ACK was never routed through the firewall which means the ACK could be dropped at the firewall due to TCP reassembly. The firewall performs stateful inspection and is not having visibility of the entire path, it's common with asymmetric routing that drops like this can occur. If it missed the SYN-ACK but sees an ACK then this is a problem as the flow is not complete. Unsure of what security policies have been applied and how zones have been setup. If it is this which I'm not sure of as can't clearly picture the design, you can apply a Zone Protection Profile and configure a bypass for the check, but this is not a preferred resolution. I am unsure of what the complete current state is of the design and its intention though

I ran packet captures on Meraki switchports where firewall and IDRAC is connected, I see SYN and ACK but no SYN,ACK . Also on the switchport where IDRAC is connected, I see SYN and SYN,ACK but no ACK.

Is the ACK that is seen sourced from the MAC address of the PA or is the ACK seen the one sourced from the server? This would be a way to tell if the firewall is dropping it without having access to the PA directly

Can I ask what 172.18.2.76 is and the mention of VPN, this has confused me a little?

Hopefully this is relevant

[grapevine]

172.18.2.76 is a typo 

172.18.2.75 - IDRAC

172.18.5.76 - Windows server (Jumpbox)

[deep-dive]

Hi Royalty,

replaying for the same for more clarification

From the source IP 172.18.5.76(JUMBOX SERVER), TCP connectivity test (Test-NetConnection) to  172.18.2.75(IDRAC SERVER) on port 443 shows successful results. However, in the firewall session logs, the traffic is marked as incomplete, with the session stuck in the INIT state and eventually aged out. 

Jumbox initiates the HTTPS session and completes the 3-way handshake on its side, but the return packets (ACK and Client Hello) are not seen on the iDRAC's interface capture. 

[Royalty]

Hi @deep-dive @grapevine,

I am still of the belief as above that the issue is that your default gateways for the VLAN 5 server and the VLAN 2 server's iDRAC NIC are different and that this creates a problem with the PA being able to allow traffic.

• Traffic from the server in VLAN 5 to VLAN 2 uses the Palo Alto as the default gateway
• Traffic from the iDRAC NIC in VLAN 2 to VLAN 5 uses the Meraki switch as the default gateway

I believe if that is the case (which it seems to be based off your description) then the Palo Alto is not in the data path for the traffic from VLAN 2 to VLAN 5. This means the firewall is not seeing the complete flow and could be dropping the ACK. This would explain why the firewall session is showing as incomplete and being aged out. Is this making any sense? You have asymmetric routing (the path from VLAN 5 to VLAN 2 is not the same as the return path). The SYN-ACK from the iDRAC is never reaching the PA because it will only transit the Meraki switch and then be forwarded from the meraki switch to the server in VLAN 5 directly. This is why the server in VLAN 5 receives the SYN-ACK and responds with an ACK. I also can't see how the PA is configured to route back towards the VLAN 2 so this is still unclear for me

It would require some testing and it may be that I've misinterpreted the details in your initial post. If I am understanding correctly, I'm unsure as to the design choice that the iDRAC is using the default gateway of 172.18.2.1 (on the Meraki switch) instead of a default gateway on the PA. The setup would mean that the PA cannot apply proper policy or perform security filtering for inter-VLAN traffc... or at least traffic generated by the iDRAC whose DGW is the Meraki switch. If possible you should configure an SVI for VLAN 2 on the PA and point the iDRAC to that as it's default gateway, but it's hard to recommend without knowing the full design.

For the Test-NetConnection, that should be expected to work as otherwise the server in VLAN 5 would not be sending an ACK at all. Test-NetConnection is purely testing for a response to the open port and not verifying a TLS handshake can complete or that data can be sent and received. Essentially, it does not test that the remote host is even receiving the ACK.

Please let me know if this makes sense or you think otherwise if I'm barking up the wrong tree

[deep-dive]

Hi @Royalty 

You are right — the same behavior is observed here.

In the firewall, interface ethernet1/2 is configured with the LAN management IP 172.18.100.1, and a subinterface ethernet1/2.5 is configured with 172.18.5.1/24 for VLAN 5.

There is no SVI or subinterface for VLAN 2 on the firewall, and the client is not willing to add VLAN 2 to the firewall. They are also not ready to remove the VLAN 5 SVI from Meraki.

We have already suggested both of those options, but they were not approved. We are now looking for an alternate solution that works within the existing constraints.

[Royalty]

I'm glad that you think the same, sounds like you've got it worked out. I guess as a temp workaround you could configure a Zone Protection profile or enable a global bypass with 'set deviceconfig setting tcp asymmetric-path bypass' on the CLI to bypass the asymmetric routing checks, theoretically. You could also use options of the 'show counters' command to check/confirm the TCP drops which should be incrementing also when that ACK is sent and received at the PA

Let me know if there's anything else