Solved: SSH Connection Refused

Travis-Fleming · ‎12-03-2020

I have a dumb problem. We have several Cisco 881 routers deployed that are doing a simple site-to-site VPN back to us from users home offices. When I have them setup in my lab on our internet connection I can SSH to the LAN IP address (over the VPN) no problem. However when they are deployed to the field, I cannon SSH, putty gives me a "Network error: Connection refused" message. I believe it is in relation to my

access-list

but I'm not sure how. Below is the running config I have of one specific device.

They do have an ssh RSA key (4096) generated on them before shipping out. Let me know if more relevant config is needed. Unfortunately I can't get to any of the 18+ devices we have deployed, but I do have another on my bench with the same config that works.. My management PC is on the 172.17.14.0/24 subnet so that would fall within the 172.16.0.0 0.15.255.255 wildcard bit mask we have on the ACL as well. Running Version 15.7(3)M.

line con 0
logging synchronous
no modem enable
transport preferred none
stopbits 1
line aux 0
stopbits 1
line vty 0 4
access-class Management in
privilege level 15
logging synchronous
transport preferred none
transport input ssh
line vty 5 15
transport preferred none
transport input none

!

ip access-list standard Management
permit 172.16.0.0 0.15.255.255

!

interface FastEthernet4 (internet interface for home user)
ip address dhcp
ip virtual-reassembly in
duplex auto
speed auto
crypto map XXX
service-policy input pm-bandwidthlimit
service-policy output pm-bandwidthlimit

Travis-Fleming · ‎12-17-2020

Well, when I do a

sh ip ssh

it states that ssh is disabled. So I then copy and pasted my standard line to generate a new rsa key at 4096 and it froze. I then restarted and generated one at 2048 and it generated and enabled SSH. Perhaps this device was not capable of generating a key with a length of 4096 and that was my problem. My standard config I use for our Cisco 4331 routers and 9300 switches generates a 4096 rsa key. Perhaps these smaller devices can only do 2048?

at-lte-agent-53#sh ip ssh
SSH Disabled - version 2.0
%Please create RSA keys to enable SSH (and of atleast 768 bits for SSH v2).
Authentication methods:publickey,keyboard-interactive,password
Authentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsa
Hostkey Algorithms:x509v3-ssh-rsa,ssh-rsa
Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
MAC Algorithms:hmac-sha1,hmac-sha1-96
Authentication timeout: 120 secs; Authentication retries: 3
Minimum expected Diffie Hellman key size : 1024 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded): NONE
at-lte-agent-53#config t
Enter configuration commands, one per line. End with CNTL/Z.
at-lte-agent-53(config)#crypto key generate rsa modulus 4096
The name for the keys will be: at-lte-agent-53.xxxx.com

% The key modulus size is 4096 bits
% Generating 4096 bit RSA keys, keys will be non-exportable...

View solution in original post

balaji.bandi · ‎12-03-2020

Can you post full config of main office Router and one remote location config to look VPN config ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Travis-Fleming · ‎12-03-2020

Sure, here is the full config of the remote 881 router. I will piece together the ASA main office firewall config if still needed.

version 15.7
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname AT-AgentRtr-5
!
boot-start-marker
boot-end-marker
!
aqm-register-fnf
!
enable secret 5 XXXX
!
aaa new-model
!
!
aaa group server tacacs+ XXX
server-private 172.17.98.80 timeout 3 key 7 XXX
!
aaa authentication login default group XXX local
aaa authentication login network group XXX local
aaa authorization network default group XXX local
aaa accounting exec default start-stop group XXX
aaa accounting commands 1 default start-stop group XXX
aaa accounting commands 15 default start-stop group XXX
aaa accounting network default start-stop group XXX
!
!
!
!
!
aaa session-id common
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!


!
ip dhcp excluded-address 10.30.5.1 10.30.5.29
!
ip dhcp pool AgentRtr
network 10.30.5.0 255.255.255.0
domain-name XXX.com
dns-server 172.18.98.78
default-router 10.30.5.1
option 150 ip 172.17.60.11 172.17.60.10
!
!
!
ip domain name XXX.com
ip cef
no ipv6 cef
!
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
license udi pid C881-K9 sn XXX
!
!
username admin password 7 XXX
!
redundancy
!
crypto ikev2 proposal XXX-PROP
encryption aes-cbc-256
integrity sha512
group 21
!
crypto ikev2 policy XXX-POLICY
proposal XXX-PROP
!
crypto ikev2 keyring L2L-Keyring
peer vpn
address X.X.X.X
pre-shared-key local XXX
pre-shared-key remote XXX
!
!
!
crypto ikev2 profile XXX
match identity remote address X.X.X.X 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local L2L-Keyring
!
!
!
!
class-map match-all cm-bandwidthlimit
match access-group 103
!
policy-map pm-bandwidthlimit
class cm-bandwidthlimit
police 5000000 conform-action transmit exceed-action drop
!
!
!
crypto isakmp policy 1
encr aes 256
authentication pre-share
group 21
!
crypto isakmp client configuration group XXX
!
!
crypto ipsec transform-set XXX-TS esp-aes 256 esp-sha-hmac
mode tunnel
!
!
!
crypto map XXX 10 ipsec-isakmp
set peer X.X.X.X
set transform-set XXX-TS
set ikev2-profile XXX
match address VPN
!
!
!
!
!
!
interface FastEthernet0
no ip address
!
interface FastEthernet1
no ip address
!
interface FastEthernet2
no ip address
!
interface FastEthernet3
no ip address
!
interface FastEthernet4
ip address dhcp
ip virtual-reassembly in
duplex auto
speed auto
crypto map XXX
service-policy input pm-bandwidthlimit
service-policy output pm-bandwidthlimit
!
interface Vlan1
ip address 10.30.5.1 255.255.255.0
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip tftp source-interface Vlan1
ip route 0.0.0.0 0.0.0.0 dhcp
ip tacacs source-interface Vlan1
ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr
ip ssh client algorithm encryption aes128-ctr aes192-ctr aes256-ctr
!
ip access-list standard Management
permit 172.16.0.0 0.15.255.255
!
ip access-list extended VPN
permit ip 10.30.5.0 0.0.0.255 any
!
logging host 172.16.1.166
ipv6 ioam timestamp
!
snmp-server community XXX RO 20
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps config-copy
snmp-server enable traps config
access-list 103 permit ip any any
!
!
!
control-plane
!
!
!
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
!
!
!
!
!
vstack
!
line con 0
logging synchronous
no modem enable
transport preferred none
stopbits 1
line aux 0
stopbits 1
line vty 0 4
access-class Management in
privilege level 15
logging synchronous
transport preferred none
transport input ssh
line vty 5 15
transport preferred none
transport input none
!
scheduler allocate 20000 1000
!
!
!
!
!
!
end

Richard Burts · ‎12-03-2020

When your attempt to SSH gets connection refused, are you able to ping to the address you were trying to use for SSH? Can you verify that at the time that you attempt SSH that the vpn tunnel is up, active, and passing traffic?

Are users able to use this vpn and access resources in the corporate network?

Am I correct in understanding that when these routers are deployed in the field that they connect to an ISP router directly? When you have one of these routers in the lab, how do they connect to outside (is it a direct connection to an ISP router or do they connect through something else)?

Seeing that the remote router is using DHCP to get its IP address on outside I wonder what is in the config of the ASA for this peer. Is there a crypto map entry for each remote router or is it doing some type of remote peer 0.0.0.0 so that it can receive multiple connection requests from different devices, which is frequently the case when the remote devices are DHCP?

HTH

Rick

Travis-Fleming · ‎12-03-2020

Yes to all of the above. I can ping the LAN IP 10.30.5.1 (Vlan1 above). The VPN is active and up, and the home users are actively passing traffic and accessing corporates resources.

Yes they are connecting to home users own cable modems to get an IP address. In the main office it is similar, we have our live internet come into a DMZ Switch segregated by VLAN's. I extend that VLAN to my lab, and directly connect into port fa4 on the router. We send these to remote staff homes with the PoE card so they can plug a Cisco Phone into them. Then we direct all internet traffic over the VPN (throttled to 5 Mbps) so all traffic goes out our corporate firewall. (probably TMI for this)

The main office does use a dynamic map on the outside that does not call out individual remote IP's, you are correct. I believe this is the relevant config on our VPN ASA 5525 firewall:

crypto dynamic-map outside_dyn_map 20 set ikev2 ipsec-proposal AES-256_SHA
crypto dynamic-map outside_dyn_map 20 set reverse-route
crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map
crypto map outside_map interface outside

Meheretab Mengistu · ‎12-03-2020

Hi,

I saw AAA is configured. Please add the following line:

aaa authorization exec default group XXX local if-authenticated

HTH,
Meheretab

Travis-Fleming · ‎12-03-2020

Thank you for the reply, but I think the line I have should work, as it works to SSH in my lab:

aaa authorization network default group XXX local

Another piece of the puzzle is if I do a Wireshark while trying to SSH, I get a bunch of "RST, ACK" packets back as apposed to just an acknowledge. It's as if SSH isn't running or listening on port 22 on the remote routers?

balaji.bandi · ‎12-03-2020

thank you can you also post the one next to your desk.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Georg Pauwen · ‎12-03-2020

Hello,

try a different SSH client such as KiTTY:

https://www.fosshub.com/KiTTY.html

Travis-Fleming · ‎12-03-2020

I thought the same thing, I tried SecureCRT, as well as ssh from within our SolarWinds web console, both say similar messages about the remote system refusing the connection.

Travis-Fleming · ‎12-03-2020

I just checked, and of the 18+ remote routers I AM able to get into one of them at our 10.30.6.1 IP. The configuration is exactly the same as the others (we take a backup of the config before we ship them). Could it be something on the remote users ISP side? This one gets a dhcp address of a private 192.168.0.2, so guessing they plugged it in behind a firewall at their home office.

In fact if I use notepad ++ to compare the working one to a none working one, they are literally the same, except the LAN ip's and serials...

Travis-Fleming · ‎12-04-2020

Here is my final guess. I believe most home users are plugging these routers into their home routers\firewalls and those are blocking ssh somehow, even though it's over a VPN tunnel. No other explanation I can think of seeing how 3-4 of them I can SSH into, and the other 14+ I cannot, with them all having the exact same configuration.

Richard Burts · ‎12-04-2020

I have been wondering if it might be possible to run debug for SSH at one of the non working routers. And that might be a good suggestion. But I believe that this most recent post suggests that the issue is something in the environment at the remote site. If the remote routers are using exactly the same config (except for different IP address ranges) and if 3 or 4 do successfully SSH and 14 do not SSH then it certainly suggests that there is something in the remote environment, perhaps something in their ISP, that is causing this behavior.

I have been thinking about what might be different between sites, perhaps some different in the DHCP parameters, that would cause this. But I can not think of any DHCP parameter that would selectively impact traffic. I am quite puzzled about how the ISP would impact SSH if the SSH packet is forwarded in the vpn. A ping to the remote router address works but SSH to that address does not work. So it is not an issue about IP connectivity. And how would any ISP be able to distinguish SSH packet from other packets?

As a next step I would suggest that you get from one of the remote routers that is not working the output of

show ip ssh

And perhaps to run debug for SSH and post any output.

HTH

Rick

Travis-Fleming · ‎12-04-2020

Thanks Richard, I've had the same thoughts as you. Through luck I am actually getting one of the non-working (in terms of SSH) routers back from a home user here next week. I will do a

sh ip ssh

and a debug of SSH when it's on my desk so I can console it.

Richard Burts · ‎12-04-2020

We will be interested in what you find on the returned router. I wonder if the issue is something about the remote ISP (perhaps DHCP or something) if the problem will reproduce when it is on your desk? But get the router, check it out, and let us know the results.

HTH

Rick