Securing SSH access on routers, combined approach, NO AAA-servers

[Andreas Wittemann]

SSH Access Segmentation Challenge on ISR4300/IOS-XE 17.12.4a

Platform

• Hardware: ISR4331, ISR4321 (ISR4300 series)
• Software: IOS-XE 17.12.04a

Objective

Implement SSH access with two different security policies based on authentication method:

• Public key authentication: Accessible from any source IP address
• Password authentication: Restricted to specific management IP addresses only

Technical Challenge

IOS VTY line architecture appears to allocate SSH connections to the first available VTY line, regardless of the authentication method that will ultimately be used. This prevents implementing different access-control policies for different authentication types. No additional servers should be involved for AAA, like TACACS/RADIUS. What seemed like an easy task makes me nuts at the moment.
The PubKey-Part is not the problem, that´s easy...i want to have a way in which i can separate the SSH-login to the routers using usr/pwd only to a few IPs, and the pubkey for whichever IP.
Sounds simple? I thought so too...

Configuration Attempts

Attempt 1: ACL-based VTY Segmentation

! VTY 0-4: Password auth, IP-restricted
line vty 0 4
 access-class MGMT-IPs in
 login authentication SSH-MGMT

! VTY 5-15: Key auth, no restrictions
line vty 5 15
 login authentication SSH-PUBKEY

Result: SSH connections are attempted on VTY 0-4 first, triggering the ACL check before authentication method is known. Public key users from non-management IPs are blocked despite having valid keys configured.

Log evidence:

%SEC-6-IPACCESSLOGS: list MGMT-IPs denied <non-mgmt-ip> 1 packet

Attempt 2: SSH Rotary Groups (Separate Ports)

! Port 22: Password auth, IP-restricted
line vty 0 4
 access-class MGMT-IPs in
 login authentication SSH-MGMT

! Port 2222: Key auth via rotary group
ip ssh port 2222 rotary 1
line vty 16 20
 rotary 1
 login authentication SSH-PUBKEY

Result: Port 2222 configuration accepted but connections timeout. show ip ssh confirms SSH listening on both ports. WAN ACL permits both ports. Unclear why rotary group port doesn't accept connections. Is this a bug?

Attempt 3: Remove VTY ACLs Entirely

line vty 0 4
 login authentication SSH-MGMT

line vty 5 15
 login authentication SSH-PUBKEY

Result: Public key authentication works from any IP, but password authentication is also now accessible from any IP, defeating the security objective.

Attempt 4: Configure quiet-mode/BruteForce-protection

login block-for 300 attempts 2 within 600
login quiet-mode access-class MGMT-IPs
login on-failure log
login on-success log

line vty 0 4
access-class MGMT-IPs in
login authentication SSH-MGMT

line vty 5 15
login authentication SSH-PUBKEY

Combining this with brute-force protection...somehow sabotages the (legit!) PubKey-SSH-Logins. SSH public key authentication frequently registers an initial LOGIN_FAILED during key negotiation/algorithm matching, followed immediately by LOGIN_SUCCESS. Each failure increments the login blocking counter.

%SSH-5-SSH2_SESSION: SSH2 Session request from <ip> (tty = 1) Succeeded 
%SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: keyuser] [Source: <ip>] 
%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: keyuser] [Source: <ip>]

...and after logging in a few times you are simply locked out. Yes, you COULD adjust this to make it more bearable, but it makes not much sense. Legitimate key-based users get blocked after configured threshold (2-5 attempts) because normal SSH key negotiation counts as failures. This causes self-lockout even when authentication ultimately succeeds.The challenge is that login block-for operates at the login subsystem level before distinguishing between authentication methods, making it unsuitable for mixed password/key environments.

Question for Community

Is there a supported method in IOS-XE to:

1. Route SSH connections to different VTY line groups based on authentication method?
2. Apply ACLs only to password-based authentication while allowing key-based authentication from any IP?
3. Make SSH rotary groups work reliably for this use case?

Or is the only viable solution to:

• Disable password authentication entirely for remote access (console/VPN only for password users)
• Use SSH exclusively for public key authentication

Any insights or alternative approaches would be appreciated!