Telnet drops over VPN

[00u18jg7x27DHjRMh5d7]

I have a server users must connect to with telnet over VPN threw the 1140 Firepower. The telnet connection drops after 1 hour per user, I have increased connection time on server still have same issue. The users are not idle they can be on the server interface and lose connection. Local users telnet to the server but do not get dropped only remote users. I increased VPN idle time to 3 hours same result. Is there a Telnet time out hidden in the FTD I need to change or were can I look on Firepower to see what might be dropping the traffic??

Firepower, Cisco Firepower Threat Defense (FTD)

Thanks

[balaji.bandi]

what is the device you trying to connect using telnet ? telnet to Firepower  or any other device.

[00u18jg7x27DHjRMh5d7]

@balaji.bandi 

It's remote clients over AnyConnect VPN that can only interface with the server over telnet its an old SB Client device.

[MHM Cisco World]

Check ftd log message i think the tcp connection is drop in ftd not from server nor client.

Increase tcp timeout if you see like this log.

[00u18jg7x27DHjRMh5d7]

@MHM Cisco World 

I have been looking for the TCP timeout in FTD unable to locate if you could point me in the right direction.  I have increased idle timeout on VPN connection it didn't help.

Thanks

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/215092-analyze-firepower-firewall-captures-to-e.html

[00u18jg7x27DHjRMh5d7]

I was already doing packet capture looking for odd traffic or out of place drops. See same behavior between local clients and remote clients. I thought it had to do with FTP quit commands but that is normal behavior we see it with on site users and they do not lose connection. The odd thing is that this worked without issue for 4 weeks then after a weekend and no changes made to FirePower device this started. 

So if anyone has any other suggestions to look at they are appreciated. 

Thanks.

[MHM Cisco World]

Can i see log from client or server ?
also share the packet-tracer.

[00u18jg7x27DHjRMh5d7]

Bellow is a trace I will look into getting some logs if needed. 

Thanks

WH01-FP-1# packet-tracer input outside tcp X.X.101.1 80 X.X.1.228 80 det$

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x1528f479e050, priority=13, domain=capture, deny=false
hits=581350015, user_data=0x1528f9a49d00, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=outside, output_ifc=any

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x1528f18871a0, priority=1, domain=permit, deny=false
hits=1993080061, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
input_ifc=outside, output_ifc=any

Phase: 3
Type: ROUTE-LOOKUP
Subtype: No ECMP load balancing
Result: ALLOW
Config:
Additional Information:
Destination is locally connected. No ECMP load balancing.
Found next-hop X.X.1.228 using egress ifc inside(vrfid:0)

Phase: 4
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,any) source static VPN_Network_10 VPN_Network_10 destination static Inside_Network Inside_Network
Additional Information:
NAT divert to egress interface inside(vrfid:0)
Untranslate X.X.1.228/80 to X.X.1.228/80

Phase: 5
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group NGFW_ONBOX_ACL global
access-list NGFW_ONBOX_ACL advanced permit object-group |acSvcg-268435486 ifc outside any ifc inside any rule-id 268435486
access-list NGFW_ONBOX_ACL remark rule-id 268435486: ACCESS POLICY: NGFW_Access_Policy
access-list NGFW_ONBOX_ACL remark rule-id 268435486: L5 RULE: Geo_Block_2
object-group service |acSvcg-268435486
service-object ip
Additional Information:
This packet will be sent to snort for additional processing where a verdict will be reached
Forward Flow based lookup yields rule:
in id=0x1528f54b4680, priority=12, domain=permit, deny=false
hits=335823, user_data=0x1528dd7ec5c0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, ifc=outside(vrfid:0)
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, ifc=inside(vrfid:0), vlan=0, dscp=0x0, nsg_id=none
input_ifc=any, output_ifc=any

Phase: 6
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside,any) source static VPN_Network_10 VPN_Network_10 destination static Inside_Network Inside_Network
Additional Information:
Static translate X.X.101.1/80 to X.X.101.1/80
Forward Flow based lookup yields rule:
in id=0x1528f46e9460, priority=6, domain=nat, deny=false
hits=71994, user_data=0x1528f97329d0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=X.X.101.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=X.X.1.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=outside(vrfid:0), output_ifc=any

Phase: 7
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x1528ef904430, priority=0, domain=nat-per-session, deny=false
hits=17854793, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=any, output_ifc=any

Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x1528f188c860, priority=0, domain=inspect-ip-options, deny=true
hits=12692043, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=outside(vrfid:0), output_ifc=any

Phase: 9
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x1528f39f1580, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=1186383, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=outside(vrfid:0), output_ifc=any

Phase: 10
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (outside,any) source static VPN_Network_10 VPN_Network_10 destination static Inside_Network Inside_Network
Additional Information:
Forward Flow based lookup yields rule:
out id=0x1528f4733d10, priority=6, domain=nat-reverse, deny=false
hits=70657, user_data=0x1528f68bfbc0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.10.101.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=192.168.1.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=outside, output_ifc=any(vrfid:0)

Phase: 11
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x1528ef904430, priority=0, domain=nat-per-session, deny=false
hits=17854795, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=any, output_ifc=any

Phase: 12
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x1528f19a6ff0, priority=0, domain=inspect-ip-options, deny=true
hits=13503424, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0, nsg_id=none
input_ifc=inside(vrfid:0), output_ifc=any

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 16162169, packet dispatched to next module
Module information for forward flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_snort
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Module information for reverse flow ...
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_snort
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_fp_tracer_drop
snp_ifc_stat

Phase: 14
Type: EXTERNAL-INSPECT
Subtype:
Result: ALLOW
Config:
Additional Information:
Application: 'SNORT Inspect'

Phase: 15
Type: SNORT
Subtype:
Result: ALLOW
Config:
Additional Information:
Snort Trace:
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435475, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queried. Got session type 2 rule id: 268435457, rule_action:2, rev id:4144222902, ruleMatch flag:0x5
MidRecovery data queriedSnort Verdict: (pass-packet) allow this packet

Phase: 16
Type: INPUT-ROUTE-LOOKUP-FROM-OUTPUT-ROUTE-LOOKUP
Subtype: Resolve Preferred Egress interface
Result: ALLOW
Config:
Additional Information:
Found next-hop X.X.1.228 using egress ifc inside(vrfid:0)

Phase: 17
Type: ADJACENCY-LOOKUP
Subtype: Resolve Nexthop IP address to MAC
Result: ALLOW
Config:
Additional Information:
Found adjacency entry for Next-hop X.X.1.228 on interface inside
Adjacency :Active
MAC address 6cae.8b02.ef64 hits 76 reference 6

Result:
input-interface: outside(vrfid:0)
input-status: up
input-line-status: up
output-interface: inside(vrfid:0)
output-status: up
output-line-status: up
Action: allow

[MHM Cisco World]

are this MAC address 6cae.8b02.ef64 map to interface or Server?

[00u18jg7x27DHjRMh5d7]

To the server.

[andrewmilne]

Sorry bit late to the party on this one, but still might help someone.

We use Firepower 1140 and SB client Telnet Connection and experience the same thing when we installed the 1140 in April last year

I raised a post on the Universe users forum and got a very useful reply SB Client VPN and Cisco Firewall | Rocket U2 | UniVerse & UniData (rocketsoftware.com)
The solution was to change the keepalive-timeout setting to 4 hours and we've not had any issues since

this change was deployed by our support company as i don't have a degree in Cisco and this change needed to be done via flexconfig and not via the webinterface for the 1140 Firewall

Thanks
Andy

[00u18jg7x27DHjRMh5d7]

Thanks for this information this is great!!! We will document this for future use. Also glad to see we were not alone with this problem. We found out it was our Geo-Block causing the issue.... But root cause was our Snort needed to be set back to 1.0 it was causing issues with a lot of different applications. After lowering that Geo-Block has worked normal and so have some other things

Thanks for the information I will put this in place even though we are moving off that server in the next couple of months.