Intermittent Connectivity on 9200l Management Ports

[benweber]

I have a customer with three 9200l switches.  For management we connect to each switch via its G0/0 management port. (The switches are configured as layer-2 only so this is the only management access.)

The configuration is very simple.  Here's one as an example:

interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address 172.31.254.245 255.255.255.0
negotiation auto

ip default-gateway 172.31.254.20

(The gateway isn't really needed as the jump server we use to access them is on the same subnet.)

Ping is consistently fine to their management IPs.  And there are no errors on the interfaces either on the switches or the switch they connect into.  However, SSH access to them is intermittent.  It will work and keep you connected for a few minutes, then kick you out, not let you connect in for a few more minutes, then kick you out again, etc.  It's so intermittent that administering the switches is basically not feasible.

I've tried different code revs in the 16 and 17 trains with no luck.  I got Cisco to send a replacement switch but it's doing the same thing.  (So a total of four switches all doing this.)

Is there something I'm missing?  I haven't worked a ton with using management interfaces but I think I'm doing everything the documentation says to do.  I've been working with the TAC for a few weeks but they haven't had any insights.

It's not a radius issue because if I reconfigure the switch for local authentication only it still does the same thing.

B

[MHM Cisco World]

You must config defualt route with vrf aware 

It is not in global.

[benweber]

Ok.  But they are in the same subnet so are you saying the default route matters?  And how would that be intermittent?  In my experience routes, especially static routes, either work or they don't.

The switch IP is 172.31.254.245/24.  The IP of the jump box I'm connecting from (SSH) is 172.31.254.150.

[benweber]

And if it's a routing issue why does ping work?  I can reliably ping the management IPs of all three switches from anywhere in the network, even distant WAN locations that go through many layer-3 hops to get there.  That doesn't jibe with this being a routing issue.

[MHM Cisco World]

ok, 
only try 
ip route vrf Mgmt-VRF x.x.x.x

and see, try only in one SW.

[benweber]

I just checked the switches and there is no option to specify the vrf for the default gateway.

[MHM Cisco World]

this other issue same as you and ping but no SSH, finally the routing issue was the solution for it.
https://community.cisco.com/t5/network-management/remote-switch-management-routing/m-p/4619732#M146781

[benweber]

That post is about using static default routes instead of the default-gateway command in layer 3 vs layer 2 switches, and it's describing a situation where the user could not ping.

My switches are strictly layer 2 and ping works fine, from anywhere on the internal network.

[MHM Cisco World]

from any PC telnet using port 22
this make us sure that the port is not close in FW.

for FW do you config  
 ACL to all port 22 to inside ?
NAT?

[Flavio Miranda]

Hi

 It seems to me that the problem might not be on the switch as you replace it. But I´d make a test by creating a different connection like:

int vlan 10

ip add x.x.x.x

Or using a interface with IP address.  

From this jump server , the only switches that happens this problem is those switches?  It is the only switch running IOX-XE on the network?   Which SSH client do you use ?

[benweber]

Yeah, I've actually done that and it works.  I migrated the connections off of one switch so I could test with it.  And I set it up with an in-band vlan for management as you describe.  That works fine.  We have something like 30-40 Cisco routers, switches, and firewalls on this network and the rest of them are all working fine.  The problem is that these three switches are DMZ and extranet devices and the network is high enough security that we don't want them to have a footprint inside the perimeter.  That's why we are using the management interfaces.

For SSH I've tried both SecureCRT and Putty.  Same thing either way.

Ben

[Flavio Miranda]

 They are in a DMZ but the access is not through a firewall, righ?  This jump server have a leg on the DMZ network.?

I´d recommend you to delete ssh config and add again, force it to generate a new key but is a long shot as you replace the switch and the same problem happens.

 Pretty weird problem. 

[benweber]

Right.  The management (G0/0) interfaces of the switches are on the internal admin network.  The switchports themselves are all layer-2 only in the DMZs.  The firewalls connecting those DMZs to the internal network are controlled by a 3rd party so we don't want to do in-band management that flows through those firewalls.  So management interfaces are an ideal solution for us.

I've tried rebuilding the config several times and am now working with replacement hardware we got from Cisco.  So now all four switches are exhibiting the same problem.  I'm beginning to wonder if this model of 9200l doesn't just have a problem with its management interfaces.  The TAC hasn't had any ideas either.  They're stumped.

[Flavio Miranda]

Could be a platform problem as this switch is relativelly new and this scenario is not used usually. 

 Unless someone with the same scenario can help here, I have no more idea. 

[balaji.bandi]

nterface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address 172.31.254.245 255.255.255.0
negotiation auto

If you using VRF as MGMT, try below :

no ip default-gateway 172.31.254.20

ip route vrf Mgmt-vrf 0.0.0.0 0.0.0.0 172.31.254.20