Solved: Re: Only one user can log into a switch via Telnet

gordonderick · ‎04-09-2013

The switch will allow only one user to connect to it at any given time. Below are the configs and oputputs.

ol-dr-6509-01#show users

Line User Host(s) Idle Location

* 1 vty 0 gkeo idle 00:00:00 192.168.3.130

2 vty 1 idle 23w6d

3 vty 2 idle 37w4d

4 vty 3 idle 37w4d

5 vty 4 idle never

6 vty 5 idle never

7 vty 6 idle never

8 vty 7 idle never

9 vty 8 idle never

10 vty 9 idle never

11 vty 10 idle never

12 vty 11 idle never

13 vty 12 idle never

14 vty 13 idle never

15 vty 14 idle never

16 vty 15 idle never

Interface User Mode Idle Peer Address

net_mgmt>>192.168.1.2

Trying 192.168.1.2 ...

% Connection refused by remote host

-----

!

aaa new-model

!

aaa group server tacacs+ site-tacacs

server 192.168.3.4

server-private 192.168.3.4 key 7 xxxxxxx

ip vrf forwarding vpn_mgmt

ip tacacs source-interface Vlan23

!

aaa authentication login default group site-tacacs local

aaa authentication enable default group site-tacacsenable none

aaa authorization exec default group site-tacacs local if-authenticated

aaa accounting exec default start-stop group tacacs+

aaa accounting commands 15 default start-stop group tacacs+

aaa accounting connection default start-stop group tacacs+

!

line vty 0 4

session-timeout 1

exec-timeout 30 0

password 7 xxxxxxx

line vty 5 15

session-timeout 1

exec-timeout 30 0

password 7 xxxxxx

!

Steve Fuller · ‎04-10-2013

I've found a bug (CSCeg52893) where the symptoms are "VTY or TTY sessions may hang after unsuccessful authentication attempts to an external AAA server".

Although the show line info you provided does not really match, there's some additional information in the bug that states:

Symptom: A VTY session on a 7206VXR may become hung and be unable to be cleared. The session does not drop after the exec-timeout period. Conditions: This condition is noted using IOS version 12.3(3)B1.
Workaround: The workaround is to reload the router.
Further Problem Description: The hung VTY session cannot be cleared using "clear line". It has no TCP TCB entry in "show tcp brief". The detailed TCP information from "show tcp " shows the TCP state as CLOSED.

The above seems pretty close to what you're seeing so I think this is a possibility.

It's not great that the workaround is a reload, but the bug also states:

To prevent the symptom from occurring, configure the maximum authentication attempts on the Cisco platform to be lower than the maximum authentication attempts on the external AAA server by entering the aaa authentication attempts login number-of-attempts global configuration command, in which the number-of-attempts argument is a value that is smaller then the maximum authentication attempts that are configured on the external AAA server.

If you are hitting this you can at least avoid hitting it again.

I can't get a list of all known affected versions to confirm whether 12.2(33)SXI5 is listed or not so it's probably worth going to TAC with this and see and they can confirm whether you're definitely hitting this and getting a confirmation of the fix.

Regards

View solution in original post

turfsniffer0 · ‎04-09-2013

Looks like there are open sessions on lines 2,3, and 4 for over 23 weeks. You can clear them by using the clear line x (where x represents the line number).

the problem occurs because users are logging in and not disconnecting the session correctly. When you close the session type in the disconnect command (if its a cisco device). This will clear the vty line after being used.

gordonderick · ‎04-10-2013

Thanks turfsniffer. I forgot to mention that i did try that but that did not work as well. Below are the logs and some more info.

ol-dr-6509-01#clear line 2
[confirm]
[OK]
ol-dr-6509-01#clear line 3
[confirm]
[OK]
ol-dr-6509-01#clear line 4
[confirm]
[OK]
ol-dr-6509-01#sh users
    Line       User       Host(s)              Idle       Location
* 1 vty 0     gkeo   idle                 00:00:00 192.168.3.130
   2 vty 1                idle                    23w6d
   3 vty 2                idle                    37w4d
   4 vty 3                idle                    37w4d
   5 vty 4                idle                    never
   6 vty 5                idle                    never
   7 vty 6                idle                    never
   8 vty 7                idle                    never
   9 vty 8                idle                    never
10 vty 9                idle                    never
11 vty 10               idle                    never
12 vty 11               idle                    never
13 vty 12               idle                    never
14 vty 13               idle                    never
15 vty 14               idle                    never
16 vty 15               idle                    never

Interface User Mode Idle Peer Address

ol-dr-6509-01#clear line vty 2
[confirm]
[OK]
ol-dr-6509-01#clear line vty 3
[confirm]
[OK]
ol-dr-6509-01#clear line vty 4
[confirm]
[OK]
ol-dr-6509-01#sh users
    Line       User       Host(s)              Idle       Location
* 1 vty 0     gkeo   idle                 00:00:00 192.168.3.130
   2 vty 1                idle                    23w6d
   3 vty 2                idle                    37w4d
   4 vty 3                idle                    37w4d
   5 vty 4                idle                    never
   6 vty 5                idle                    never
   7 vty 6                idle                    never
   8 vty 7                idle                    never
   9 vty 8                idle                    never
10 vty 9                idle                    never
11 vty 10               idle                    never
12 vty 11               idle                    never
13 vty 12               idle                    never
14 vty 13               idle                    never
15 vty 14               idle                    never
16 vty 15               idle                    never

Interface User Mode Idle Peer Address

turfsniffer0 · ‎04-10-2013

you could try command show tcp tcb - this will show you a list of connections that TCP adds to a Transmission control block to keep track of local and remote connections.

You'll see something like:

TCB Local Address Foreign Address (state)

48735680 10.0.0.3.57494 10.10.20.8.22 ESTAB

487DA60C 10.0.2.35.22 10.10.1.54.18073 ESTAB

you can clear the reference TCB by using the clear tcp tcb x (where x represents the TCB eg 48735680)

now do the sh users command and you should see the corresponding line status change to CLOSED. Just clear that line again and it should work. Repeat process for each line until all cleared.

gordonderick · ‎04-10-2013

Thanks again.

Im guessing since this is on 12.2.(33) it does not give the o/p that is expected. See below the logs. I dont see another tcp tty session (except the non-tty) on the switch.

ol-dr-6509-01#show tcp tcb ?

<0x0-0xFFFFFFFF> TCB address

ol-dr-6509-01#show tcp tcb 0

tty1, virtual tty from host 192.168.3.130
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Mininum incoming TTL 0, Outgoing TTL 255
Local host: 192.168.1.2, Local port: 23
Foreign host: 192.168.3.130, Foreign port: 25155

Enqueued packets for retransmit: 2, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x85ECC09D0):
Timer Starts Wakeups Next
Retrans 72 0 0x85ECC0AFF

------Output ommited-------

'It just shows 1 tty session'

ol-dr-6509-01#ol-dr-6509-01#clear tcp tcb 2
[confirm]
%Clear TCP failed: TCB 0x2 not found
ol-dr-6509-01#

ol-dr-6509-01#clear tcp tcb ?
<0x0-0xFFFFFFFF> TCB address

ol-dr-6509-01#clear tcp tcb 1
[confirm]
[Connection to 192.168.1.2 closed by foreign host]

turfsniffer0 · ‎04-10-2013

Sorry Gordon, my mistake - the command should be show tcp brief not show tcp tcb - you'll see the TCB indentifier on the output.

Steve Fuller · ‎04-10-2013

Hi,

The method described above will obviously clear existing sessions, but is not really practical as a long term solution. An option you should consider is using the service tcp-keepalives-in and service tcp-keepalives-out global configuraiton commands to clear any hung sessions. Take a read of the Using service tcp-keepalives to Avoid Hung Telnet Sessions texchnote for more details.

The other question is from where and why the sessions are appearing in the first place.

Regards

gordonderick · ‎04-10-2013

Hi Steve, thanks,

but that config is already on the switch

ol-dr-6509-01#sh run | i service

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

gordonderick · ‎04-10-2013

Thanks turf, i did that but still nothing.

ol-dr-6509-01#show tcp brie

TCB Local Address Foreign Address (state)

52DD3F38 192.168.1.2.646 192.168.1.3.17442 CLOSED

5295B9C4 192.168.1.2.179 192.168.1.3.48704 CLOSED

52DD43FC 192.168.1.2.11104 192.168.1.1.646 CLOSED

52DD2BA8 192.168.1.2.23 192.168.3.130.50756 ESTAB

52DD6378 192.168.1.2.179 192.168.1.1.50541 CLOSED

ol-dr-6509-01#

turfsniffer0 · ‎04-10-2013

Hi Gordon looks like you did the following and then issued the show tcp brief command?

clear tcp tcb 52DD3F38

clear tcp tcb 5295B9C4

clear tcp tcb 52DD43FC

clear tcp tcb 52DD6378

if so just clear the line again doing the following:

ol-dr-6509-01#clear line 2

[confirm]

[OK]

ol-dr-6509-01#clear line 3

[confirm]

[OK]

ol-dr-6509-01#clear line 4

[confirm]

[OK]

Now check by doing show users to see if they're cleared. If followed correctly it should work unless there might be a IOS bug, or something up with the config somewhere?

Steve Fuller · ‎04-10-2013

The odd thing here is that only 1 of the 5 sessions seen with the show tcp brief should appear when you execute the show users command, that being the entry with the TCB of 52DD2BA8.

The other entries are for BGP and LDP and so shouldn't be using one of the VTY sessions. You should be very careful before clearing those sessions.

What IOS version are you running on this?

Regards

gordonderick · ‎04-10-2013

Hi turf, steve is right, its was not right to clear the sessions. :-(

anyway even after clearing the session and clearing the line , there is no diffrence.

this is on 12.2(33)SXI5 on a 6509. I have other swithces in the network with the same IOS and hardware but they dont have this problem.

Steve Fuller · ‎04-10-2013

Hi Gordon,

Can you paste the output of a show users all and show line where line-number is one of the lines that are hung?

Also, I'm curious that you have AAA configured here, but I don't see any login authentication command within line vty context. Do you get prompted for AAA authentication on the one line that is working?

Regards

gordonderick · ‎04-10-2013

Thanks Steve,

Yes there is AAA configured. And i do get request for username and pwd when logging in.

Below are the logs as requested.

ol-dr-6509-01#sh users all
    Line       User       Host(s)              Idle       Location
   0 con 0                                     00:00:00
* 1 vty 0     gpereira   idle                 00:00:00 192.168.3.130
   2 vty 1                idle                    24w0d
   3 vty 2                idle                    37w5d
   4 vty 3                idle                    37w5d
   5 vty 4                idle                    never
   6 vty 5                idle                    never
   7 vty 6                idle                    never
   8 vty 7                idle                    never
   9 vty 8                idle                    never
10 vty 9                idle                    never
11 vty 10               idle                    never
12 vty 11               idle                    never
13 vty 12               idle                    never
14 vty 13               idle                    never
15 vty 14               idle                    never
16 vty 15               idle                    never

Interface User Mode Idle Peer Address

ol-dr-6509-01#

ol-dr-6509-01#show line 3
Tty Typ Tx/Rx A Modem Roty AccO AccI Uses Noise Overruns Int
* 3 VTY - - - - - 10 0 0/0 -

Line 3, Location: "", Type: ""
Length: 24 lines, Width: 80 columns
Baud rate (TX/RX) is 9600/9600
Status: Ready, Active, No Exit Banner
Capabilities: none
Modem state: Ready, Carrier Dropped
Group codes:    0
Special Chars: Escape Hold Stop Start Disconnect Activation
                ^^x    none   -     -       none
Timeouts:      Idle EXEC    Idle Session   Modem Answer Session   Dispatch
               00:30:00       00:01:00                       none     not set
                            Idle Session Disconnect Warning
                              never
                            Login-sequence User Response
                             00:00:30
                            Autoselect Initial Wait
                              not set
Modem type is unknown.
Session limit is not set.
Time since activation: never
Editing is enabled.
History is enabled, history size is 10.
DNS resolution in show commands is enabled
Full user help is disabled
Allowed input transports are lat pad telnet rlogin ssh.
Allowed output transports are lat pad telnet rlogin ssh.
Preferred transport is lat.
No output characters are padded
No special data dispatching characters
ol-dr-6509-01#

Steve Fuller · ‎04-10-2013

I've found a bug (CSCeg52893) where the symptoms are "VTY or TTY sessions may hang after unsuccessful authentication attempts to an external AAA server".

Although the show line info you provided does not really match, there's some additional information in the bug that states:

Symptom: A VTY session on a 7206VXR may become hung and be unable to be cleared. The session does not drop after the exec-timeout period. Conditions: This condition is noted using IOS version 12.3(3)B1.
Workaround: The workaround is to reload the router.
Further Problem Description: The hung VTY session cannot be cleared using "clear line". It has no TCP TCB entry in "show tcp brief". The detailed TCP information from "show tcp " shows the TCP state as CLOSED.

The above seems pretty close to what you're seeing so I think this is a possibility.

It's not great that the workaround is a reload, but the bug also states:

To prevent the symptom from occurring, configure the maximum authentication attempts on the Cisco platform to be lower than the maximum authentication attempts on the external AAA server by entering the aaa authentication attempts login number-of-attempts global configuration command, in which the number-of-attempts argument is a value that is smaller then the maximum authentication attempts that are configured on the external AAA server.

If you are hitting this you can at least avoid hitting it again.

I can't get a list of all known affected versions to confirm whether 12.2(33)SXI5 is listed or not so it's probably worth going to TAC with this and see and they can confirm whether you're definitely hitting this and getting a confirmation of the fix.

Regards