ACE module is dropping packets and closing connection.

Amjad Hashim · ‎07-27-2012

Hello All,

I have a ACE module A2(3.5) installed, I am having a connectivity problem between two servers in my network. I have captured some traffic on different points in my network and from capture it seems like the problem is with this ACE module or somehow it is closing the connection. I have attached the syslog messages plus capture messages from ACE device, please keep in mind source ip address is 192.168.249.21 and destination is 192.168.249.69 when you check the log messages.

I am not good with ACE at all so any help will be really appriciated.

Regards,

Amjad Hashim.

Jorge Bejarano · ‎07-27-2012

Hello Amjad,

Please upload the configuration related to the issue or the #show run and specify the VIP in question.

Please get new capture like this:

Admin# show running-config access-list

access-list ACCESS-ANS line 8 extended permit ip any any

# capture CAPTURE-TAC all access-list ACCESS-ANS

# capture CAPTURE-TAC start

# capture CAPTURE-TAC stop

# copy capture CAPTURE-TAC disk0: CAPTURE-TAC

# copy disk0:CAPTURE-TAC ftp:

Enter Address for the ftp server[]? 10.198.16.93

Enter the destination filename[]? [CAPTURE-TAC]

Enter username[]? css

Enter the file transfer mode[bin/ascii]: [bin]

Enable Passive mode[Yes/No]: [Yes]

Password:

Passive mode on.

Hash mark printing on (1024 bytes/hash mark).

################

Admin#

Here you have a link about the entire process:

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Overview_of_ACE_Troubleshooting#Capturing_Packets_in_Real_Time

Is this happening during peak? How often do you experience this?

Jorge

Amjad Hashim · ‎07-30-2012

Hello Jorge,

Thanks for your reply, the issue is that the traffic is not for a VIP on the ACE itself. The ACE module has routes in it to pass the traffic through, in my case we have two firewall contexts and in between these two contexts in the ACE device.

The two firewall contexts are called front end and back end and traffic for all the backend servers go through ACE module, module simply has routes in it for BE firewall subnets and thats it.

And there is an access-list on ACE interface to allow any any traffic. This ACE has some VIPs on it for some services but it is also a layer 3 hop between FE and BE firewalls. If u want me to capture anything else please let me know.

Regards,

Amjad Hashim.

Jorge Bejarano · ‎07-30-2012

Hello Amjad,

I understand you just have traffic passing through the ACE, do you have a specific configuration which is matching that traffic?
How did you determine the ACE is dropping the connections?
Is this a new implementation? Has this ever worked before?
Do you get the same behavior if you bypass the ACE?

Jorge

Amjad Hashim · ‎07-31-2012

Hello Jorge,

Thanks for your reply, We have an access-list to allow traffic to come into the ACE from 192.168.249.69 and a route to send it to BE firewall.

I have attached some logs in my first post if u can spare some time to look into it. the file called "ace detail capture" is showing ACE having message type CON_CLOSE and PKT_DROP etc.

Please find attached my config from ACE.

Amjad Hashim · ‎07-31-2012

access-list ALLOW-TRAFFIC line 1 extended permit ip any any
access-list ALLOW-TRAFFIC line 2 extended permit icmp any any
access-list ICMP_ACL line 10 extended permit icmp any any

robe http HTTP_80
probe icmp ICMP
interval 15
passdetect interval 60
probe tcp TCP_80
interval 20
passdetect count 1
probe http TDB-ServerAvailability-80
description Probe for TDB Servers
interval 5
passdetect interval 5
receive 5
request method head url //Monitoring/Monitor.aspx
expect status 200 200

rserver host NETMAN1LDS_TEST
description test real server
ip address 192.168.239.12
probe TCP_80
inservice
rserver redirect TDBWEB-Redirect
description Redirect to https://abc.abc.com
webhost-redirection https://abc.abc.com/
inservice
rserver host TDBWEB1LV
description TDBWEB1LV real server
ip address 192.168.225.11
probe TDB-ServerAvailability-80
inservice
rserver host TDBWEB2LV
description TDBWEB1LV real server
ip address 192.168.225.12
probe TDB-ServerAvailability-80
inservice
rserver host TDBWEB3LV
description TDBWEB1LV real server
ip address 192.168.225.13
probe TDB-ServerAvailability-80
inservice

serverfarm host TDB_SF
rserver TDBWEB1LV 80
    inservice
rserver TDBWEB2LV 80
    inservice
rserver TDBWEB3LV 80
    inservice
serverfarm redirect TDB_SF_Redirect
description http to https redirect for TDB
rserver TDBWEB-Redirect
    inservice
serverfarm host TEST_SF
rserver NETMAN1LDS_TEST 80
    inservice

ssl-proxy service TDB-Web-SSL-PROXY
key abc.abc.ace.pem
cert abc.abc.ace.pem
chaingroup TDB-chain
ssl-proxy service TEST_ORION_PROXY
key healthspace-2048-key
cert HealthspaceSignedCert-V2
chaingroup Verisign-generic

class-map match-all ICMP_INSPECT_CLASS
2 match access-list ICMP_ACL
class-map match-any NAT_CLASS
2 match access-list NAT_ACCESS
class-map match-all TDB-Web-80
2 match virtual-address 10.97.88.12 tcp eq www
class-map match-all TDB_443_CM
2 match virtual-address 10.97.88.12 tcp eq https
class-map match-all TDB_CM
2 match virtual-address 10.97.88.13 tcp eq www
class-map match-all TEST_ORION_443_CM
2 match virtual-address 192.168.173.130 tcp eq https
class-map match-all TEST_ORION_CM
2 match virtual-address 192.168.173.130 tcp eq www

policy-map type management first-match mgmt-pm
class class-default
permit

policy-map type loadbalance first-match TDB_PM
class class-default
    serverfarm TDB_SF
policy-map type loadbalance first-match TDB_PM-80
class class-default
    serverfarm TDB_SF_Redirect
policy-map type loadbalance first-match TEST_ORION_PM
class class-default
    serverfarm TEST_SF

policy-map multi-match ICMP_INSPECT_POLICY
class ICMP_INSPECT_CLASS
    inspect icmp error
policy-map multi-match NAT_POLICY
class NAT_CLASS
    nat dynamic 1 vlan 300
class TEST_ORION_CM
    loadbalance vip inservice
    loadbalance policy TEST_ORION_PM
    loadbalance vip icmp-reply active
policy-map multi-match PM_MULTI_MATCH
class TEST_ORION_CM
    loadbalance vip inservice
    loadbalance policy TEST_ORION_PM
    loadbalance vip icmp-reply active
class TEST_ORION_443_CM
    loadbalance vip inservice
    loadbalance policy TEST_ORION_PM
    loadbalance vip icmp-reply active
    ssl-proxy server TEST_ORION_PROXY
class TDB_CM
    loadbalance vip inservice
    loadbalance policy TDB_PM
    loadbalance vip icmp-reply active
class TDB_443_CM
    loadbalance vip inservice
    loadbalance policy TDB_PM
    loadbalance vip icmp-reply active
    ssl-proxy server TDB-Web-SSL-PROXY
class TDB-Web-80
    loadbalance vip inservice
    loadbalance policy TDB_PM-80

service-policy input PM_MULTI_MATCH

interface vlan 300
ip address 192.168.62.36 255.255.255.248
alias 192.168.62.37 255.255.255.248
peer ip address 192.168.62.35 255.255.255.248
access-group input ALLOW-TRAFFIC
nat-pool 1 192.168.62.38 192.168.62.38 netmask 255.255.255.248 pat
service-policy input ICMP_INSPECT_POLICY
service-policy input mgmt-pm
service-policy input NAT_POLICY
no shutdown
interface vlan 301
ip address 192.168.62.44 255.255.255.248
alias 192.168.62.45 255.255.255.248
peer ip address 192.168.62.43 255.255.255.248
access-group input ALLOW-TRAFFIC
service-policy input ICMP_INSPECT_POLICY
service-policy input mgmt-pm
no shutdown

ip route 0.0.0.0 0.0.0.0 192.168.62.33
ip route 192.168.66.0 255.255.255.0 192.168.62.41
ip route 192.168.225.0 255.255.255.192 192.168.62.41
ip route 192.168.225.128 255.255.255.192 192.168.62.41
ip route 192.168.249.0 255.255.255.240 192.168.62.41
ip route 192.168.249.16 255.255.255.240 192.168.62.41

Gilles Dufour · ‎07-31-2012

Hello,

you can verify if your traffic will be permitted with the following command :

show np 1 access-list trace vlan 301 in proto 6 source 192.168.249.21 0 des 192.168.249.69 80

You should see something like :

action_flag 0x3 (permit yes ...

Also, get a show tech before and after a connection failure and send it to us so we can check the drop counters.

Could you also clarify if the connection works and then stops suddenly or the connection is never established ?

Could you export the sniffer trace in pcap format and not text so that we can analyse it with wireshark.

Thanks.

Gilles.