I’ve learned something today and would like to share it with others.
Problem
User 192.168.1.1 complaint that he can’t telnet to IP Address 10.10.10.10 anymore.
There is no response from 10.10.10.10 and the telnet session just disconnected. He was able to telnet to the same server last week.
Troubleshooting Steps
I did a quick check on the firewall and found the following error message.
ASA#sh log | i 10.10.10.10
Jun 24 2011 02:19:48 172.16.1.1 : %ASA-2-106001: Inbound TCP connection denied from 10.10.10.10/23 to 192.168.1.1/1068 flags SYN ACK on interface outside
Jun 24 2011 02:19:54 172.16.1.1 : %ASA-2-106001: Inbound TCP connection denied from 10.10.10.10/23 to 192.168.1.1/1068 flags SYN ACK on interface outside
Jun 24 2011 02:20:06 172.16.1.1 : %ASA-2-106001: Inbound TCP connection denied from 10.10.10.10/23 to 192.168.1.1/1068 flags SYN ACK on interface outside
ASA#
Seems like the firewall has dropped the packet. Quickly I checked the firewall rules to make sure it’s there
access-list acl-out extended permit tcp host 192.168.1.1 host 10.10.10.10 eq telnet
Yes, the ACL is there.
Then, I run a packet-tracer to make sure the firewall rule is working fine.
ASA# packet-tracer input outside tcp 192.168.1.1 1024 10.10.10.10 23
Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list
Phase: 2
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
static (inside,outside) 10.10.10.10 10.10.10.10 netmask 255.255.255.255
match ip inside host 10.10.10.10 outside any
static translation to 10.10.10.10
translate_hits = 0, untranslate_hits = 68
Additional Information:
NAT divert to egress interface inside
Untranslate 10.10.10.10/0 to 10.10.10.10/0 using netmask 255.255.255.255
Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group acl-out in interface outside
access-list acl-out extended permit tcp host 192.168.1.1 host 10.10.10.10 eq telnet
Additional Information:
Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 6
Type: AAA
Subtype: aaa-auth
Result: ALLOW
Config:
aaa authentication match acl-auth-out outside RSA
Additional Information:
Phase: 7
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Phase: 8
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
static (inside,outside) 10.10.10.10 10.10.10.10 netmask 255.255.255.255
match ip inside host 10.10.10.10 outside any
static translation to 10.10.10.10
translate_hits = 0, untranslate_hits = 68
Additional Information:
Phase: 9
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
static (inside,outside) 10.10.10.10 10.10.10.10 netmask 255.255.255.255
match ip inside host 10.10.10.10 outside any
static translation to 10.10.10.10
translate_hits = 0, untranslate_hits = 68
Additional Information:
Phase: 10
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 124969, packet dispatched to next module
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: allow
ASA#
Based on packet-tracer, I can see that it allow this traffic. Hmm...
I also googled the error message and found the following information.
106001
Error Message %PIX|ASA-2-106001: Inbound TCP connection denied from IP_address/port
to IP_address/port flags tcp_flags on interface interface_name
Explanation This is a connection-related message. This message occurs when an attempt to connect to an inside address is denied by your security policy. Possible tcp_flags values correspond to the flags in the TCP header that were present when the connection was denied. For example, a TCP packet arrived for which no connection state exists in the security appliance, and it was dropped. The tcp_flags in this packet are FIN and ACK.
The tcp_flags are as follows:
•ACK—The acknowledgment number was received.
•FIN—Data was sent.
•PSH—The receiver passed data to the application.
•RST—The connection was reset.
•SYN—Sequence numbers were synchronized to start a connection.
•URG—The urgent pointer was declared valid.
Recommended Action None required.
http://www.cisco.com/en/US/docs/security/asa/asa70/system/message/logmsgs.html
Since there is no information that can help me, I did a packet capture and found this.
9 packets captured 1: 13:37:05.331449 192.168.1.1.1337 > 10.10.10.10.23: S 2194094828:2194094828(0) win 64512 <[|tcp]>
2: 13:37:05.331785 10.10.10.10.23 > 192.168.1.1.1337: S 1102669092:1102669092(0) ack 2194094829 win 8192 <[|tcp]>
3: 13:37:05.332273 10.10.10.10.23 > 192.168.1.1.1337: S 1102669092:1102669092(0) ack 2194094829 win 8192 <[|tcp]>
4: 13:37:08.370662 192.168.1.1.1337 > 10.10.10.10.23: S 2194094828:2194094828(0) win 64512 <[|tcp]>
5: 13:37:11.326124 10.10.10.10.23 > 192.168.1.1.1337: S 1102669092:1102669092(0) ack 2194094829 win 8192 <[|tcp]>
6: 13:37:11.326704 10.10.10.10.23 > 192.168.1.1.1337: S 1102669092:1102669092(0) ack 2194094829 win 8192 <[|tcp]>
7: 13:37:14.386469 192.168.1.1.1337 > 10.10.10.10.23: S 2194094828:2194094828(0) win 64512 <[|tcp]>
8: 13:37:23.325804 10.10.10.10.23 > 192.168.1.1.1337: S 1102669092:1102669092(0) ack 2194094829 win 8192 <[|tcp]>
9: 13:37:23.326322 10.10.10.10.23 > 192.168.1.1.1337: S 1102669092:1102669092(0) ack 2194094829 win 8192 <[|tcp]>
9 packets shown
TCP 3 way handshake is not completed due to last ACK packet from user is not sent to the server 10.10.10.10.
I’ll let you guys figure out what is the answer for this, I’ll share the solution later
