Complex Network problem I need help solving.

[cameron.calvert]

Here is a bit of a complex issue I need assistance finding the answer to. We're directly connected to another office that more or less manages their own network. The handoff between their network and ours is our equipment. They have their own Internet and network that connects to our network via a Router and Firewall. The reason there is a firewall and a router is there was a pair of t1 lines on their network that went into that router. That is gone and left it as a go between the our firewall and their network.

It goes like this, (Their router/switch-> Our ISR4331->Our ASA5506 ->ComcastENSCircuit/Trunk->Nexus9300FX(layer 3 routing pair of switches) for our whole network using HSRP and VPC -> 9300X(virtual servers connection to our network)->Host

The Problem:
Up until two days ago, everything was working fine. The last thing that was changed was over 3 weeks ago. We recently swapped from two Catylst 3850s paired together with HSRP. We've been running on the new setup problem free and this particular network that hangs off our network was functioning as well. As of a few days ago, now all of a sudden the only thing from their network that will come back is ping and traceroute. All other protocol traffic doesn't seem to be coming back. I can see the traffic go across to our network on that border firewall but nothing appears to come back except for ping and tracert.

I have gotten on the firewall and made sure that everything is open Any any on both the outside and inside interface for troubleshooting. Also of interesting note, I can ping their gateway 192.168.0.1 from the edge router because it has its interface on that same range. If I go to the Firewall before it, I cannot ping their gateway but I can our edge interface.

All of our equipment is using OSPF, including the nexus, 5506 and ISR4331. The 9300x is not routing and is being used as layer 2.

We're at a loss to track down what could of changed. Both sides say nothing changed but clearly something did.

[Kasun Bandara]

@cameron.calvert hi, as per your explanation, i get some sense on asymmetric routing. if you can provide some rough diagram and setup details we may be able to help more. also check if there is any route changes or switch path changes. you can try tracing L2 Paths and L3 paths.

[paul driver]

Hello
Probably be beneficial if you could post a network diagram outlining the issue you have mentioned?

[ammahend]

if you can ping and trace from host connected to 9300 (as source) to router on the other side past firewall (as destination) then its obviously not a routing issue.

you can start troubleshooting from left to right.

Start with a capture on router/switch on port connected to ISR4331 and make sure you receiving traffic sent by host, if not then capture on ISR4331 on port connected to ASA5506 and keep moving right.

[MHM Cisco World]

Share the topology please 

MHM

[cameron.calvert · Accepted solution]

I wanted everyone to know that I found the issue. It turns out the other side had a firewall(I was under the impression it was just a router) as the next hop. According to them, "They didn't change anything", however after pushing the issue to get them to check their ACLs to see if they were blocking their own traffic... well they were. Sometimes it goes that way. Thank you for all your suggestions! I kept pushing through my equipment and Ultimately found out that if I got on their network, but set my end interface as the gateway (the next hop past their firewall) everything worked perfectly. This led me to believe it was their device!