Unable to SSH to AP joined to 9800

[pabloayalas]

Hello,

I'm facing an issue with our 9800 where we can't SSH into any of our access points; even though we have it configured in the AP Join profile, we still can't SSH. This is the output from the policy profile:

WLC-1#show ap profile all-profiles

AP Profile Name : AP_PROFILE
Description : AP Profile
Country code : US
Stats Timer : 180
Link Latency : DISABLED
Data Encryption : DISABLED
LED State : ENABLED
NTP server : 0.0.0.0
NTP Authentication : DISABLED
Jumbo MTU : DISABLED
24ghz Report Interval : 90
5ghz Report Interval : 90
bssid stats status : DISABLED
bssid stats frqncy interval : 30
bssid neighbor stats status : DISABLED
bssid neighbor stats interval : 180
POE :
PreStandard 802.3af Switch : DISABLED
Power Injector State : DISABLED
Power Injector Selection : Unknown
Injector Switch Mac : Not Configured
Device Management :
Telnet : DISABLED
SSH : ENABLED
Serial Console : ENABLED
User Management :
Username : admin

As you can see, SSH is enabled and with an admin account we configured. Now, I'm showing an output from one of our access points:

WLC-1#show ap config general

Cisco AP Name : AP
=================================================

Cisco AP Identifier : ...
Country Code : US
Regulatory Domain Allowed by Country : 802.11bg:-A 802.11a:-AB 802.11 6GHz:-B
Radio Authority IDs : None
AP Country Code : US - United States
AP Regulatory Domain
802.11bg : -A
802.11a : -B
MAC Address : 002a.101e.e6d0
IP Address Configuration : DHCP
IP Address : IP_ADDRESS
IP Netmask : 255.255.255.0
Gateway IP Address : DEFAULT-GATEWAY
CAPWAP Path MTU : 1485
Capwap Active Window Size : 1
Telnet State : Disabled
CPU Type : ARMv7 Processor rev 1 (v7l)
Memory Type : DDR4
Memory Size : 1028096 KB
SSH State : Enabled
Serial Console State : Enabled
Cisco AP Location : default location

And, still we can't SSH. Finally if I run a remote command from any of our access points, you can see that SSH has been enabled to the remote AP:

Feb 1 03:16:15.671: %AP_LOG-6-WAP : Chassis 1 AP Name : WAP
Feb 1 03:16:15.671: %AP_LOG-6-WAP : Chassis 1 Admin State : Enabled
Feb 1 03:16:15.671: %AP_LOG-6-WAP : Chassis 1 AP Mode : Local
Feb 1 03:16:15.671: %AP_LOG-6-WAP : Chassis 1 AP Submode : AWIPS FORENSIC
Feb 1 03:16:15.671: %AP_LOG-6-WAP : Chassis 1 Location : default location
Feb 1 03:16:15.671: %AP_LOG-6-WAP : Chassis 1 Primary controller name : WLC-1
Feb 1 03:16:15.671: %AP_LOG-6-WAP : Chassis 1 Primary controller IP : IP_ADDRESS
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Secondary controller name :
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Tertiary controller name :
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Controller from DHCP offer : IP_ADDRESS
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Controller from DNS server : IP_ADDRESS
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 AP join priority : 1
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 IP Prefer-mode : IPv4
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 CAPWAP UDP-Lite : Unconfigured
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Last Joined Controller name: WLC-1
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 DTLS Encryption State : Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Discovery Timer : 10
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Heartbeat Timer : 30
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 CDP State : Enabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Watchdog monitoring : Enabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 IOX : Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 RRM State : Enabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 LSC State : Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 SSH State : Enabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 AP Username : admin
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Session Timeout : 300
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Extlog Host : 0.0.0.0
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Extlog Flags : 0
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Extlog Status Interval : 0
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Syslog Host : 255.255.255.255
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Syslog Facility : 0
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Syslog Level : informational
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Core Dump File Compression : Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Core Dump Filename :
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Client Trace Status : Disabled(All)
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Client Trace All Clients : Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Client Trace Filter : 0x00000000
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Client Trace Out ConsoleLog: Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 WLC Link LAG status : Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 AP Link LAG status : Disabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 AP WSA Mode : Enabled
Feb 1 03:16:15.672: %AP_LOG-6-WAP : Chassis 1 Auxiliary-client Interface : Disabled

And still, I can't get anything from the AP. If I try to SSH in verbose mode from my device, this is the output:

user@MAV ~ % ssh <IP_ADDRESS> -vvvv
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug3: kex names ok: [diffie-hellman-group1-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: /etc/ssh/ssh_config line 62: Applying options for *
debug2: resolve_canonicalize: hostname <IP_ADDRESS> is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/user/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: ssh_connect_direct: entering
debug1: Connecting to <IP_ADDRESS> [<IP_ADDRESS>] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /Users/user/.ssh/id_rsa type -1
debug1: identity file /Users/user/.ssh/id_rsa-cert type -1
debug1: identity file /Users/user/.ssh/id_ecdsa type -1
debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/user/.ssh/id_ed25519 type -1
debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/user/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/user/.ssh/id_xmss type -1
debug1: identity file /Users/user/.ssh/id_xmss-cert type -1
debug1: identity file /Users/user/.ssh/id_dsa type -1
debug1: identity file /Users/user/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
kex_exchange_identification: Connection closed by remote host
Connection closed by IP_ADDRESS port 22

Any idea where the issue might be?

Thanks 

[marce1000]

               - Reboot the AP and check if that can help , 

 M.

[pabloayalas]

I have already tried multiple times with the same result. I bounced the port (these are PoE), but still, there was no SSH after the AP restored connectivity. This is very strange; I already have a TAC case; however, there is no path forward.

[marce1000]

          >... I already have a TAC case; however, there is no path forward.
 - Check software version being used on the controller , go for an advisory release such as 17.9.4a and check if that can help , 

   Also have an overall checkup of the 9800 configuration with the CLI command show tech wireless ; and feed the output into 
                                                                                                                       Wireless Config Analyzer

  Note that GUI changes are translated into the running config ; you may want to take diffs after subsequent GUI actions and compare the changes as seen in the running config. Further proceeding by then on using the CLI with the running configuration only may sometimes need to better results ,

 M,

[David Ritter]

That's where I do it..

[pabloayalas]

That's what I have configured, and it's still the same.

[Rich R]

1. What version of software are you using?
2. Can you ping the AP?
3. Are there any ACLs or firewalls between your PC and the AP?
4. Have you tried the SSH from a directly connected router or switch in the same subnet?

[pabloayalas]

Pfff, I figure this out. I was pushing a dACL from ISE, this was the dACL:

remark - Allow DHCP
permit udp any eq bootpc any eq bootps
remark - Allow DNS
permit udp any any eq domain
remark - Allow ICMP
permit icmp any any
remark permit ssh
permit tcp any any eq 22
remark permit CAPWAP (control channel)
permit udp any any eq 5246
remark permit CAPWAP (data channel)
permit udp any any eq 5247
remark permit CAPWAP (Mcast channel)
permit udp any any eq 5248
deny ip any any

I remove entirely the dACL and worked. Thanks everyone for your help and support! 

[Rich R]

The old shooting yourself in the foot trick <smile>

[pabloayalas]

hahahahaha yep!!!