FTD sub-optimal lookup/drop for only certain traffic

[bgoulet00]

recently a number of telephony devices have stopped working.  these devices are spread across multiple branch offices but all talk to the same cisco call manager back in the datacenter.  running a packet capture i can see the incoming tftp request from the client to the CM passes the firewall fine but the response from the CM gets dropped by the firewall.  it seems to have some confusion on where the traffic should go.  interestingly ping traffic from the CM to that same client goes through just fine. the problem is limited to the tftp response. the problem only just started happening about a month ago.  the only change made at all on the firewall was an OS upgrade over 2 months ago.  

i've been working with TAC and haven't gotten anywhere on this.   the FTD has 2 upstream OSPF neighbors on the outside interface.  both are reachable, show as ospf neighbors and have entries in the arp table.  again, most traffic between these 2 devices passes just fine.

attached are traces from both the dropped tftp traffic and icmp traffic that passes between the same 2 devices.  CM is behind the inside interface and the client device is on the outside

[Francesco Molino]

Hi

Can you share a quick drawing on how everything is connected? Also can you show the detail of the drop packet?

Can you run the command system support trace for traffic to CM and another output for traffic from CM? Will you be able to share those outputs?

[bgoulet00]

ran the trace both ways.  this is my first time working with this command but it appears to be just running down the policy rules for a match and hitting the internal comms rule and getting an allow which is what i would expect.  network diagram attached.  the phone is on the outside, the call manager on the inside

i'm not sure what you want for the detail of the dropped packet.  pcap output?  

[Francesco Molino]

Sorry for my late answer.
Can you run a packet capture and export it in pcap and share the file?

Also I believe you've checked the routing and the traffic (return traffic) goes through the firewall were the session has been initiated?

Can you create the rule (both direction) in prefilter and see if it works.

Let us know about your results tests.

[bgoulet00]

the detailed captures with trace are attached to the initial post.  here is the same in pcap format. nothing too exciting.  on the outside interface you see the incoming tftp request but no response going out.  on the inside interface you see the request and response.  

after creating a prefilter rule for all inbound and outbound traffic for 10.224.107.11 the client devices all started working.  i'm not sure what prefilters are so i'm not sure what this means

[Francesco Molino]

Pre-filter means the traffic stays on the LINA (asa code) and don’t go through SNORT for inspection.

Can you share a screenshot of your policies?

[bgoulet00]

TAC was able to come up with a fix by modifying the NAT rules.  the whole thing still seems odd to me.  our NAT rules haven't changed since they were built by contractors 3 years ago so why all of a sudden we had an issue is beyond me.  looking at the attachment, the change that fixed it was enabling 'route-lookup' on rule 5.  during initial troubleshooting with TAC a few weeks ago we created rule 1 as a test.  that rule very specifically targeted one of the ATA devices effected.  that rule had route-lookup enabled on it and even though we verified the traffic was hitting that rule it was still getting dropped so why enabling the feature on the broader rules works is a bit of a mystery as well.