Re: Cannot mtr through Fire Power Firewall

TheNetRunner · ‎12-29-2022

Hey all,
We use mtr (my traceroute) to troubleshoot issues and provide reports to ISPs during outages. My company recently installed some new FirePower 4115 firewalls running FXOS 2.10 and FDM 7.0.1 to replace our old pfsense firewalls.

However, since they were installed, we've been unable to run my traceroutes through the new firewalls (inside to outside). We get a response from the target but the hops in between show up as (waiting for reply), as you can imagine this is unhelpful. I've tried adding rules to allow ICMP traffic and setting the default policy to allow. This issue persisted despite these changes. I also confirmed it's not out network by sending mtr through our old pfsense firewalls.

Any help would be greatly appreciated.

server-a) (172.31.11.14)                       2022-12-29T16:22:35+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.31.11.1                        0.0%   584    0.4   0.3   0.3   1.4   0.1
 2. 172.31.11.1                        0.0%   584    0.3   0.3   0.2   5.6   0.2
 3. (waiting for reply)
 4. (waiting for reply)
 5. (waiting for reply)
 6. (waiting for reply)
 7. (waiting for reply)
 8. (waiting for reply)
 9. (waiting for reply)
10. 1.1.1.1                           0.0%   583    1.2   1.2   1.1   4.9   0.2

@Rob IngramIt seem to delete your message but I caught it before it disappeared. I followed the guide you provided with no success sadly.

https://integratingit.wordpress.com/2019/10/12/ftd-allow-traceroute/

Rob Ingram · ‎12-29-2022

@TheNetRunner refer to this post to enable traceroute through the FTD https://integratingit.wordpress.com/2019/10/12/ftd-allow-traceroute/

Rob Ingram · ‎12-29-2022

@TheNetRunner I saw the message disappear....it's back now.

Please provide a screenshot of your ACP rules relating to the traceroute rules.

From the CLI of the FTD run the command "system support firewall-engine-debug" filter on the IP address of the client running the traceroute, capture the output and upload here.

TheNetRunner · ‎12-30-2022

@Rob Ingramthank you for the support
Below is a copy of the requested info

Please specify an IP protocol: icmp
Please specify a client IP address: 172.16.0.14
Please specify a server IP address: 1.1.1.1
Monitoring firewall engine debug messages

172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=6 GR=4-1 New firewall session
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=6 GR=4-1 app event with app id changed, url no change, tls host no change, bits 0x25
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=6 GR=4-1 MidRecovery data sent for rule id: 268435499, rule_action:2, rev id:1008270430, rule_match flag:0x0
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=6 GR=4-1 using HW or preset rule order 2, 'some_vn', action Allow and prefilter rule 0
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=6 GR=4-1 allow action
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 New firewall session
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 app event with app id changed, url no change, tls host no change, bits 0x25
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 MidRecovery data sent for rule id: 268435499, rule_action:2, rev id:1008270430, rule_match flag:0x0
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 using HW or preset rule order 2, 'some_vn', action Allow and prefilter rule 0
172.31.11.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 allow action

Rob Ingram · ‎12-30-2022

@TheNetRunner don't specify the server address (1.1.1.1) otherwise you won't see the other addresses (of each hop) responding - this might provide a clue to why its not matching rule action 2.

I appear to have the same rules set up on my FTD (using FDM) and it is working.

TheNetRunner · ‎12-30-2022

Apologies, I am new to FTD. I assumed, incorrectly, that it was required lol.

Interesting that it is working for you. Which versions of FXOS and FDM are you running?

2nd times the charm...

> system support firewall-engine-debug

Please specify an IP protocol: icmp
Please specify a client IP address: 172.16.0.14
Please specify a server IP address:
Monitoring firewall engine debug messages

172.16.0.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 Deleting Firewall session
172.16.0.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 New firewall session
172.16.0.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 app event with app id changed, url no change, tls host no change, bits 0x25
172.16.0.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 MidRecovery data sent for rule id: 268435486, rule_action:2, rev id:3675751523, rule_match flag:0x0
172.16.0.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 using HW or preset rule order 1, 'ICMP Outbound', action Allow and prefilter rule 0
172.16.0.14 8 -> 1.1.1.1 0 1 AS=0 ID=26 GR=4-1 allow action

Rob Ingram · ‎12-30-2022

@TheNetRunner is that it? I'd expect to see some time exceeded events from each hop.

TheNetRunner · ‎12-30-2022

Yep, nothing else appears, which I also took as odd as well. I've raised a TAC with Cisco since I believe this might be a complex one. If I fix it then I shall update this thread with the fix.

Thank you again @Rob Ingram and happy new year.