Intermittent Connectivity on 9200l Management Ports

[benweber]

I have a customer with three 9200l switches.  For management we connect to each switch via its G0/0 management port. (The switches are configured as layer-2 only so this is the only management access.)

The configuration is very simple.  Here's one as an example:

interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address 172.31.254.245 255.255.255.0
negotiation auto

ip default-gateway 172.31.254.20

(The gateway isn't really needed as the jump server we use to access them is on the same subnet.)

Ping is consistently fine to their management IPs.  And there are no errors on the interfaces either on the switches or the switch they connect into.  However, SSH access to them is intermittent.  It will work and keep you connected for a few minutes, then kick you out, not let you connect in for a few more minutes, then kick you out again, etc.  It's so intermittent that administering the switches is basically not feasible.

I've tried different code revs in the 16 and 17 trains with no luck.  I got Cisco to send a replacement switch but it's doing the same thing.  (So a total of four switches all doing this.)

Is there something I'm missing?  I haven't worked a ton with using management interfaces but I think I'm doing everything the documentation says to do.  I've been working with the TAC for a few weeks but they haven't had any insights.

It's not a radius issue because if I reconfigure the switch for local authentication only it still does the same thing.

B

[MHM Cisco World]

I suggest that and his reply that I can ping but there is no SSH.

[balaji.bandi]

@MHM Cisco World  appologies, i might have missed your comment.

they he may need to used SSH source.

ip ssh source-interface XXXXX

Note  : I have also observed OP mentioned he can not see VRF route config, OP can provide - show version

[MHM Cisco World]

Dont worry, i just want to update you.

[benweber]

That "ip ssh source-interface XXXXX" determines the IP you come from when you are SSH'ing from a device that has multiple addressed interfaces.  I'm not SSH'ing from this box, I'm connecting to it.  And each of these switches only has the one IP, on the management interface.  So the command wouldn't do anything.  There's no other address I could be coming from.

[benweber]

Yeah, I'm not sure why you guys keep thinking it's a routing issue.  If I can ping the IP of the management interface from 4-5 hops away then routing is working fine.

And again, these are layer-2 only switches.  For layer-2 only you use the "default-gateway" syntax.  The "ip route" command is for switches running layer-3.  And since my jump server is on the same subnet no routing is occurring. 

And to repeat, in the absence of dynamic routing protocols (which I'm not using here) it would be odd for routing issues to be intermittent.

[MHM Cisco World]

please just one min think if SW work as L2 SW, does L2 SW have VRF ??? is VRF for L3 device ??
that why Me and  @balaji.bandi  think that it is routing issue.

[MHM Cisco World]

https://community.cisco.com/t5/telepresence-and-video/no-ssh-on-c9200-48p-gigabit0-0-mgmt/td-p/4051105

[benweber]

Yeah, I checked that.  I actually completely removed my VTY acl with no luck.  This is something else.

[MHM Cisco World]

1-run the below command check if the port is receive the ping and SSH packet, after finish no command to disable it.
monitor capture CL interface GigabitEthernet x both

2-run the below commend see the counter do ping many times and check counter, repeat the same for SSH do it many time and see counter, the ping and SSH must process by CPU 
sh controllers cpu-interface

3-check the TCP is establish between client and SW
show tcp brief
4-see if the VTY is idle and if the "*" simple is appear with user 
show users

[balaji.bandi]

Yeah, I'm not sure why you guys keep thinking it's a routing issue.  If I can ping the IP of the management interface from 4-5 hops away then routing is working fine.

we are not thinking its a routing issue since VRF was configured, your SSH may be not using the right path, this was a guess here.

Since ping ok, SSH is an issue, you need to provide show run full config here for us to determine along with below information.

show IP ssh

show IP interface brief

show IP route

show IP route vrf all

[benweber]

I ended up going in a different direction to resolve this.  I took one of the free ports on the switch and used the "no switchport" command to turn it into a routed port. That worked fine when I did that.  I left it plugged into the same port on the inside network as the management port so was able to rule out any sort of issue with our internal network.

That was a test solution only as running it that way wouldn't provide the necessary level of security.  But we have an ASA5506 in the environment that we use to terminate backup VPN connections so I just set up three interfaces on that as dedicated /30 DMZs and put the three switches behind that.  That provides the necessary security barrier so, while it's a bit kludgy we're just going to run it that way.

Definitely something weird about the management ports on those switches.  Also interesting is that they intermittently won't recognize type 9 passwords.  On all four switches they work at first but then on some of them (not all of them) they stop recognizing them and give an invalid credentials error.  I'm not really into burning my time fixing Cisco's problems for them (and the TAC was thoroughly stumped by all of this) so I just went with type 8 passwords.

Thanks for the responses.

Ben