SSH and Console issue on Cisco 3850s Stack

[Beacon Bits]

Hi Ya'll,

I have Cisco 3850s stack where somehow our NAC device occupied all the vty lines so I have lost ssh on the stack.

In troubleshooting this issue I consoled (...is that a word) the stack and cleared the vty lines but that didn't work. Then I cleared the TCP TCB block relate to each SSH session since SSH is run on TCP but this didn't work either. The sad part of this whole troubleshooting was I have lost my console access as well. Now when I console ... Console does not show anything.. just a blinking cursor.

This is our production stack, we have no idea how to resolve this unless I reboot the whole stack.

Any suggestion be much appreciated.

Regards,

S

[Georg Pauwen]

Hello,

obviously you want to avoid a reboot. The only way I can see is to use SNMP to clear the hung TCP sessions. Are you using a Network Management Tool ? The document below describes the procedure using NET-SNMP:

http://www.cisco.com/c/en/us/support/docs/ip/simple-network-management-protocol-snmp/61860-fixTCPhang.html

[Beacon Bits]

Hi  gpauwen,

Thanks for your reply.

I was wondering if lines are not clearing at first place when I tried through console then HOW net-snmp can clear those hung TCP connections. 

Technically, either console or SNMP doing the same job but through the different ways.

By the way, we have Solarwinds SSH session open on the cisco Stack as it is polling data regularly. Is there anything you think could be done?

I have polled the MIB for tcpConnectState.

Any advise?

Regards,

S

[paul driver]

Hello

How did you lose console access- did you make any changes to the mgt plane On the switchs there should be a mode button to highlight the stack master - have to tried consoling in to that particular switch?

It should also show the other stack members status within that stack 

res

paul

[Beacon Bits]

Hi Paul,

Thanks for your contribution. I'm fully aware of Stack Master and other members.

The issue I can see is that my Console is also not releasing. There's way being suggested to use SNMP. I did my research and I found it bit long, 

I think reboot is the only quick and easy solution ..

Regards,

S

[Georg Pauwen]

You might want to try the break key combination (usually Ctrl + Break), which is usually used for password recovery, but maybe you'll get your console connection released...

http://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/12818-61.html

[Beacon Bits]

gpauwen,

Done this.. but nothing works.

Thanks!

[paul driver]

Hello

so just to confirm -

you are physically attached to the master switch into the console port (not aux port) via console cable and you cannot access the stack via serial  and ip connection via telnet / ssh  or http ?

Have you tried using the AUX port and maybe also disconnecting and reconnecting a couple of ports that are showing high unitilsatuon to try and free up some resource?

res

Paul