Re: strange vty/telnet problem

gcdudley · ‎07-28-2009

Hi - I've just noticed a strange problem that's occurring on our network. However, I believe that it's a fairly new problem.

Some of our switches are not allowing concurrent telnet connections to their vty lines. This is in spite of having multiple lines configured and available.

The problem only occurs with telnet; multiple ssh sessions are successful. 'show users' shows that I'm the only person connected and that I'm using the first vty line.

Any ideas would be appreciated. Unfortunately, googling 'telnet' and 'problem' yielded too many results ;-)

Thanks,

Chris

jbrenesj · ‎07-28-2009

Hi,

We need some more info.

Enable the "terminal monitor" so we can see some outputs, run a "sh tcp statistics"

1. then do a "debug ip tcp transactions",

2. try to telnet to an affected switch, capture the debugs and also

3. run these commands:

- sh tcp brief

- sh tcp statistics

- sh line

gcdudley · ‎07-28-2009

Here is one more piece of info - sorry for not including it earlier - once the 1st session is established, the second attempt will receive a login prompt that immediately times out.

Such as:

telnet hostname

Trying xxx.xxx.xxx.xxx...

Connected to hostname.

Escape character is '^]'.

User Access Verification

Username:

Username: Connection to hostname closed by foreign host.

gcdudley · ‎07-28-2009

The following was gathered during a failed 2nd attempt:

debug ip tcp transactions

TCP special event debugging is on

hostname#

Jul 28 15:52:33.810: TCP0: state was LISTEN -> SYNRCVD [23 -> xxx.xxx.xxx.xxx0(54964)]

Jul 28 15:52:33.810: TCP0: tcb 2BDE714 connection to xxx.xxx.xxx.xxx:54964, received MSS 1460, MSS is 516

Jul 28 15:52:33.810: TCP0: Connection to xxx.xxx.xxx.xxx:54964, ignoring option 3

Jul 28 15:52:33.810: TCP: sending SYN, seq 2825390698, ack 2279829130

Jul 28 15:52:33.810: TCP0: Connection to xxx.xxx.xxx.xxx:54964, advertising MSS 536

Jul 28 15:52:33.810: TCP0: state was SYNRCVD -> ESTAB [23 -> xxx.xxx.xxx.xxx(54964)]

Jul 28 15:52:33.810: TCB02BDE714 setting property TCP_TOS (1) 1C255C4

Jul 28 15:52:35.932: TCP2: state was ESTAB -> FINWAIT1 [23 -> xxx.xxx.xxx.xxx(54964)]

Jul 28 15:52:35.932: TCP2: sending FIN

Jul 28 15:52:35.932: TCP2: state was FINWAIT1 -> FINWAIT2 [23 -> xxx.xxx.xxx.xxx(54964)]

Jul 28 15:52:35.932: TCP2: FIN processed

Jul 28 15:52:35.932: TCP2: state was FINWAIT2 -> TIMEWAIT [23 -> 137.131.20.80(54964)]

cTPC77-3a#u all

Jul 28 15:52:52.944: TCP2: state was TIMEWAIT -> CLOSED [23 -> xxx.xxx.xxx.xxx(41052)]

Jul 28 15:52:52.944: TCB 0x2BA37BC destroyed

gcdudley · ‎07-28-2009

1 more thing - those 3 "Username" prompts occur immediately and do not accept input - type telnet hostname and BAM - the door is shut

gcdudley · ‎07-28-2009

After 1st login (successful)

sho tcp brief

TCB Local Address Foreign Address (state)

02C40020 switch.hostname.23 telnet.client..56768 ESTAB

After 2nd login attempt (failed)

switch.hostname#sho tcp brief

TCB Local Address Foreign Address (state)

032A5D34 switch.hostname.23 telnet.client..62662 TIMEWAIT

02C40020 switch.hostname.23 telnet.client..56768 ESTAB

gcdudley · ‎07-28-2009

sho tcp statistics

Rcvd: 175617 Total, 4990 no port

0 checksum error, 0 bad offset, 0 too short

63947 packets (758682 bytes) in sequence

2891 dup packets (260489 bytes)

20 partially dup packets (126 bytes)

0 out-of-order packets (0 bytes)

0 packets (0 bytes) with data after window

0 packets after close

0 window probe packets, 7 window update packets

320 dup ack packets, 0 ack packets with unsend data

108169 ack packets (9195786 bytes)

Sent: 187602 Total, 0 urgent packets

3964 control packets (including 17 retransmitted)

168700 data packets (9226310 bytes)

310 data packets (29413 bytes) retransmitted

5 data packets (591 bytes) fastretransmitted

14625 ack only packets (9704 delayed)

0 window probe packets, 2 window update packets

16 Connections initiated, 2546 connections accepted, 2562 connections established

2565 Connections closed (including 1302 dropped, 1 embryonic dropped)

327 Total rxmt timeout, 0 connections dropped in rxmt timeout

0 Keepalive timeout, 0 keepalive probe, 0 Connections dropped in keepalive

gcdudley · ‎07-28-2009

Line 1 increments on 2nd attempt; I'm on line 0

-----------------------

sho line

Tty Typ Tx/Rx A Modem Roty AccO AccI Uses Noise Overruns Int

0 CTY - - - - - 5 0 0/0 -

* 1 VTY - - - 1 1 1351 0 0/0 -

2 VTY - - - - 1 1095 0 0/0 -

3 VTY - - - - 1 105 0 0/0 -

4 VTY - - - - 1 4 0 0/0 -

5 VTY - - - - 1 3 0 0/0 -

6 VTY - - - - 1 0 0 0/0 -

7 VTY - - - - 1 0 0 0/0 -

8 VTY - - - - 1 0 0 0/0 -

9 VTY - - - - 1 0 0 0/0 -

10 VTY - - - - 1 0 0 0/0 -

11 VTY - - - - 1 0 0 0/0 -

12 VTY - - - - 1 0 0 0/0 -

13 VTY - - - - 1 0 0 0/0 -

14 VTY - - - - 1 0 0 0/0 -

15 VTY - - - - 1 0 0 0/0 -

16 VTY - - - - 1 0 0 0/0 -

jbrenesj · ‎07-28-2009

I just did the same thing. My PC is 10.198.39.76 and I let the telnet prompt timeout when it was asking me for the password but as you can see it waited 1 1/2 minutes before ESTAB -> FINWAIT1

In your case, this happened 2 seconds after getting to ESTAB.

Core3560#deb ip tcp transactions

TCP special event debugging is on

Core3560#term mon

Core3560#

000652: *Mar 15 02:10:29.903: TPA: Reserved port 0 in Transport Port Agent fo

CP IP type 1

000653: *Mar 15 02:10:29.903: TPA: Released port 0 in Transport Port Agent fo

CP IP

pe 1

000654: *Mar 15 02:10:29.903: TPA: Reserved port 23 in Transport Port Agent f

TCP IP type 1

000655: *Mar 15 02:10:29.903: TCP0: state was LISTEN -> SYNRCVD [23 -> 10.198

.76(2304)]

000656: *Mar 15 02:10:29.903: TCP0: tcb 38EFA3C connection to 10.198.39.76:23

received MSS 1260, MSS is 516

000657: *Mar 15 02:10:29.903: TCP: sending SYN, seq 3749767707, ack 567419721

000658: *Mar 15 02:10:29.903: TCP0: Connection to 10.198.39.76:2304, advertis

MSS 1260

000659: *Mar 15 02:10:29.903: TCP0: state was SYNRCVD -> ESTAB [23 -> 10.198.

76(2304)]

000660: *Mar 15 02:10:29.912: TCB038EFA3C setting property TCP_TOS (1) 2323E5

000661: *Mar 15 02:12:02.958: TCP2: state was ESTAB -> FINWAIT1 [23 -> 10.198

.76(2304)]

000662: *Mar 15 02:12:02.958: TCP2: sending FIN

000663: *Mar 15 02:12:02.958: TCP2: state was FINWAIT1 -> FINWAIT2 [23 -> 10.

.39.76(2304)]

000664: *Mar 15 02:12:02.958: TCP2: FIN processed

000665: *Mar 15 02:12:02.958: TCP2: state was FINWAIT2 -> TIMEWAIT [23 -> 10.

.39.76(2304)]

000666: *Mar 15 02:13:02.962: TCP2: state was TIMEWAIT -> CLOSED [23 -> 10.19

9.76(2304)]

000667: *Mar 15 02:13:02.962: TPA: Released port 23 in Transport Port Agent f

TCP IP

pe 1

000668: *Mar 15 02:13:02.962: TCB 0x38EFA3C destroyed

I will investigate further but what is the config under vty? do you have radius/tacacs? Any special settings?

Maybe send a sh line vty 2

jbrenesj · ‎07-28-2009

The switch model + IOS could help as well

gcdudley · ‎07-28-2009

the problem has occurred on 3560Gs and 3750s (24 port/48 port/12 port sfp) and it doesn't happen consistently. I just tried it on 2 switches that both had problems earlier today - this time around the problem occurred on only 1.

gcdudley · ‎07-28-2009

Nope, no special configs - login is local, access list allows our major network and lines 5-15 are disabled.

line vty 0 4

access-class 1 in

exec-timeout 20 0

password 7 PASSWORD

login local

Leo Laohoo · ‎07-28-2009

Hi Chris,

What's your Telnet config? Can you post the outputs for "sh run | b line vty"?

gcdudley · ‎07-28-2009

line vty 0 4

access-class 1 in

exec-timeout 20 0

password 7 PASSWORD

login local

lines 5-15 do not have login allowed

the access list just allows our major network