SSH to Public IP ISR Problem

[CiscoMedMed]

I have a Cisco ISR 1111 with a public IP address. I have dozens of micro sites setup like this but it's got me stumped. Based on sanitized debug attached - you can see SSH SYN comes to the router, the router processes that and sends the reply traffic and says it's properly send the packet to the next hop. The problem is that the reply traffic never makes it back to the initiator. 

Would you mind having a look at the attached brief debug ip packet for attempted SSH connectivity and let me know if you think it looks proper? Somehow I am suspecting that the return traffic is taking an alternate path back to the sender. If you have a thought on proving that to the ISP I'm all ears. But in the meantime - if you could give a thumbs up or suspicious hmmmm on this debug from the target ISR - I'd appreciate the look-over.

[Georg Pauwen]

Hello,

I guess what you could do is send an extended traceroute from the ISR back to the source, and compare both paths.

ISR#traceroute
Protocol [ip]:
Target IP address: 199.99.110.229
Source address: 77.77.129.2
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:10
Port Number [33434]:22
Loose, Strict, Record, Timestamp, Verbose[none]:

[CiscoMedMed]

I have't been able to access the remote ISR to run your suggested traceroute. But I got some new data from my ASA behind which I initiate the SSH. Out of ideas, in ASDM logging/debug I filtered for source 199.99.110.229. What I found was that the SYN went out OK. But immediately I started getting logged replies with source port 577 and destination port of the original source port on the outbound TCP 1088. Normally an SSH session outbound would yield no logging from the destination IP (if filtering on source) until the conversation teardown. Anyhow the ASA is not recognizing those replies as part of a valid conversation and are denied. The packets are TCP ACKs and SYN ACKs but it's like the ASA is seeing them for the first time. I'd expect the reply traffic to have source port 22 and destination of our random source port.

Any guess what might be going on??

[Georg Pauwen]

Hello,

so if you filter for a source from a 'working' site you don't get these logs I assume ?

What if you SSH from the ASA to the ISR, and vice versa ? Also, are the 'working' ISRs the same model/type, and do they run the same IOS version ? It is always worth checking for bugs...which versions are your running on your ASA and the ISR that doesn't work ?

[CiscoMedMed]

Exactly. If I go to a working site and SSH to the public IP of the ISR I will see no logs if I filter using the destination address as the source. If I filter with the destination address as the destination address, then 
I see the expected PAT operation and TCP SYN to port 22.

The working ISRs include this same type - 1111 router with 16.6 IOS.

I'll try to do some bug searches on both ends. The other thing I need to review is my 

"debug ip packet {ACL}" on the ISR as that should verify what source port it was sending 

from and so on. I collected that on Friday. If its source port is 577 then something is 

jacked at the ISR. If it's 22 - and I think it was - then I am at a total loss as to how

the ASA would be seeing 577 unless there's some NAT device between the

spoke and my data center.

I appreciate your feedback and helping me think it through. It's a very noisy problem internally!

[Georg Pauwen]

Hello,

when I do a search for TCP/UDP 577, the below comes up. Could it be that the ISP on that particular segment does some sort of traffic security checking ? Might be worth asking them...

https://www.speedguide.net/port.php?port=577

[CiscoMedMed]

Debug confirms that something is modifying the source port after the ACK or SYN ACK have egressed the ISR. 

Dealing with Comcast is like banging my head against the wall. Buy cheap Internet, expect cheap service.

*Jul 10 23:26:15.797: IP: s=199.99.110.229 (local), d=77.77.129.2 (nil), len 40, local feature
*Jul 10 23:26:15.797: TCP src=22, dst=3543, seq=638660902, ack=200757013, win=4128 ACK, feature skipped, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Jul 10 23:26:15.797: FIBipv4-packet-proc: route packet from (local) src 199.99.110.229 dst 77.77.129.2
*Jul 10 23:26:15.797: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/0 199.99.110.230
*Jul 10 23:26:15.797: FIBipv4-packet-proc: packet routing succeeded

[CiscoMedMed · Accepted solution]

The problem turns out to be a bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf70102 

The NAT issue in this bug describes issues with port 500 but it turns out this impacts SSH as well.

To work around the issue I added DENY rules in my NAT statement. It's weird because the NAT

is for inside traffic out. But this was SSH TO the outside interface and IPsec from the router

itself to the hub. Nasty one!