Followup explanation/observation of your issue (as understood by me) is
>>>>For example, PC A 172.16.1.10 tries to ping PC B 192.168.15.20. The ICMP request reaches PC B and the response is sent. But it >>>never reaches the gateway.
>>>I'm not sure if I can monitor packets on the RV340 but it must be going there. What happens after that?
So what's happening here is:
i) the ping-request (icmp-ping-request) from PC-A 172.16.1.10 to 192.168.15.20 is forwarded to its Gateway (172.16.1.1)
ii) Here on the Gateway, it is also connected to the vlan1 network with a ip-addr 192.168.15.2
iii) now on Gateway, when it sees that the destination-ipaddr of the ping-request is 192.168.15.20, AND this network is directly-connected on it, and it has a ipaddr in the same-subnet 192.168.15.x/24
iv) and therefore, instead of forwarding/routing this icmp-ping-request packet to the RV340, it correctly (as per standard network behavior and design) does a ARP-request for finding out the "mac-address of 192.168.15.20"....and in this case the PC-B being in the same subnet does a ARP-reply with its mac-address as its configured with the ipaddr 192.168.15.20
v) So now once gateway gets to know the mac-addr of PC-B, it simply forwards/sends the icmp-ping-request packet (in a ethernet-frame) directly to PC-B
Note: Here please note, that till this point, the RV340 does not know anything about this icmp-ping sesssion...AND please note that on RV340, there is a full-fledged statefull-firewall running on it...and if you look closely at the icmp-ping-packets, it too also has a session-id for every ping-request-reply session between 2 hosts..
vi) So now, as expected, PC-B will process the icmp-ping-request packet recieved AND as expected it will prepare to send a icmp-ping-reply....BUT since the destination-ipaddr of this icmp-ping-reply is 172.16.1.10, and since this is NOT a local-network (which is 192.168.15.x), therefore PC-B will refer to its default-gateway/default-route which is configured as 192.168.15.1 (the RV340 vlan1 interface) and therefore correctly forwards this icmp-ping-reply to the RV340...
BUT on RV340, the firewall does not have any info of any "existing" icmp-ping "session" entry....and this is NOT a icmp-ping-request, therefore it is simply dropping it....which is correct behavior for the statefull-firewall
>>>Meanwhile, if I ping from PC B to PC A, it completes without issue.
Yes.
So what's happening in this case/direction is:
i) PC-B is sending a icmp-ping-request to 172.16.1.10 and since the destination is not a local-network, therefore its sending this to the default-gateway 192.168.15.1
ii) Now on RV340, there is a static-route that says "route traffic to 172.16.1.0 network to 192.168.15.2 on vlan1 interface"....and since it has to route to 192.168.15.2...on vlan1(192.168.15.1) interface, it would have done a ARP-request and got to know the mac-address of 192.168.15.2...
lets-say for example the mac-address of 192.168.15.2 is say "zzzz:zzzzz:zzzz"
iii) So as RV340 is a router too, and since the icmp-ping-request is from a host in the local-subnet 192.168.15.x, and since the gateway to reach the destination network (172.16.1.x) is also in the same local-subnet...
- for the purpose of efficient routing, RV340 router is simply sending a "IP-Redirect" message to PC-B, such as "IP-Redirect for 172.16.1.x to 192.168.15.2"...
- and i dont recall exactly now whether the macaddr of 192.168.15.2 is also included in this IP-Redirect message....but anyways the summary processing done is that RV340 has sent a IP-Redirect message to PC-B saying you should send your icmp-ping-request for 172.16.1.10 directly to the next-hop address 192.168.15.2....
iv) and that is what PC-B has done....it has simply "redirected" the icmp-ping-request for the destination 172.16.1.10 to 192.168.15.2
v) Now when the icmp-ping-reply is sent from PC-A to 192.168.15.20....and arrives at the Gateway, instead of routing it to RV340, a ARP-resolution is done for getting the mac-add of 192.168.15.20 and thus the icmp-ping-reply is sent directly via L2-forwarding in a Ethernet-frame to PC-B.....and thus in this case, the RV340 is no longer involved further in the ongoing icmp-ping-session (except for sending IP-Redirects to PC-B)
>>>>Additionally, if I add a static route on PC B - the same static route on the RV340 - then both ping scenarios work.
Correct. Thats becos, now you are completely avoiding the statefull-firewall on the RV340 (and therefore the PC-A to PC-B works now)
Due to all that is happening (as described above) presently in your network-deployment, hence my recommendation to re-configure your network as suggested in my previous post
