Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
2488
Views
0
Helpful
6
Replies

ACE module is dropping packets and closing connection.

Amjad Hashim
Level 1
Level 1

‎07-27-2012 03:19 AM

Hello All,

I have a ACE module A2(3.5) installed, I am having a connectivity problem between two servers in my network. I have captured some traffic on different points in my network and from capture it seems like the problem is with this ACE module or somehow it is closing the connection. I have attached the syslog messages plus capture messages from ACE device, please keep in mind source ip address is 192.168.249.21 and destination is 192.168.249.69 when you check the log messages.

I am not good with ACE at all so any help will be really appriciated.

Regards,

Amjad Hashim.                  

Labels:
132357-logging.rar.zip
0 Helpful
6 Replies 6

Jorge Bejarano
Level 4
Level 4

‎07-27-2012 10:04 AM

Hello Amjad,

Please upload the configuration related to the issue or the #show run and specify the VIP in question.

Please get new capture like this:

Admin# show running-config access-list

access-list ACCESS-ANS line 8 extended permit ip any any

# capture CAPTURE-TAC all access-list ACCESS-ANS

# capture CAPTURE-TAC start

# capture CAPTURE-TAC stop

# copy capture CAPTURE-TAC disk0: CAPTURE-TAC

# copy disk0:CAPTURE-TAC ftp:

Enter Address for the ftp server[]? 10.198.16.93

Enter the destination filename[]? [CAPTURE-TAC]

Enter username[]? css

Enter the file transfer mode[bin/ascii]: [bin]

Enable Passive mode[Yes/No]: [Yes]

Password:

Passive mode on.

Hash mark printing on (1024 bytes/hash mark).

################

Admin#

Here you have a link about the entire process:

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_%28ACE%29_Troubleshooting_Guide_--_Overview_of_ACE_Troubleshooting#Capturing_Packets_in_Real_Time

Is this happening during peak? How often do you experience this?

Jorge

0 Helpful

Amjad Hashim
Level 1
Level 1
In response to Jorge Bejarano

‎07-30-2012 04:17 AM

Hello Jorge,

Thanks for your reply, the issue is that the traffic is not for a VIP on the ACE itself. The ACE module has routes in it to pass the traffic through, in my case we have two firewall contexts and in between these two contexts in the ACE device.

The two firewall contexts are called front end and back end and traffic for all the backend servers go through ACE module, module simply has routes in it for BE firewall subnets and thats it.

And there is an access-list on ACE interface to allow any any traffic. This ACE has some VIPs on it for some services but it is also a layer 3 hop between FE and BE firewalls. If u want me to capture anything else please let me know.

Regards,

Amjad Hashim.

0 Helpful

Jorge Bejarano
Level 4
Level 4
In response to Amjad Hashim

‎07-30-2012 03:41 PM

Hello Amjad,

  • I understand you just have traffic passing through the ACE, do you have a specific configuration which is matching that traffic?
  • How did you determine the ACE is dropping the connections?
  • Is this a new implementation? Has this ever worked before?
  • Do you get the same behavior if you bypass the ACE?

Jorge

0 Helpful

Amjad Hashim
Level 1
Level 1
In response to Jorge Bejarano

‎07-31-2012 02:17 AM

Hello Jorge,

Thanks for your reply, We have an access-list to allow traffic to come into the ACE from 192.168.249.69 and a route to send it to BE firewall.

I have attached some logs in my first post if u can spare some time to look into it. the file called "ace detail capture" is showing ACE having message type CON_CLOSE and PKT_DROP etc.

Please find attached my config from ACE.

0 Helpful

Amjad Hashim
Level 1
Level 1

‎07-31-2012 02:18 AM

access-list ALLOW-TRAFFIC line 1 extended permit ip any any
access-list ALLOW-TRAFFIC line 2 extended permit icmp any any
access-list ICMP_ACL line 10 extended permit icmp any any

robe http HTTP_80
probe icmp ICMP
  interval 15
  passdetect interval 60
probe tcp TCP_80
  interval 20
  passdetect count 1
probe http TDB-ServerAvailability-80
  description Probe for TDB Servers
  interval 5
  passdetect interval 5
  receive 5
  request method head url //Monitoring/Monitor.aspx
  expect status 200 200


rserver host NETMAN1LDS_TEST
  description test real server
  ip address 192.168.239.12
  probe TCP_80
  inservice
rserver redirect TDBWEB-Redirect
  description Redirect to https://abc.abc.com
  webhost-redirection https://abc.abc.com/
  inservice
rserver host TDBWEB1LV
  description TDBWEB1LV real server
  ip address 192.168.225.11
  probe TDB-ServerAvailability-80
  inservice
rserver host TDBWEB2LV
  description TDBWEB1LV real server
  ip address 192.168.225.12
  probe TDB-ServerAvailability-80
  inservice
rserver host TDBWEB3LV
  description TDBWEB1LV real server
  ip address 192.168.225.13
  probe TDB-ServerAvailability-80
  inservice

serverfarm host TDB_SF
  rserver TDBWEB1LV 80
    inservice
  rserver TDBWEB2LV 80
    inservice
  rserver TDBWEB3LV 80
    inservice
serverfarm redirect TDB_SF_Redirect
  description http to https redirect for TDB
  rserver TDBWEB-Redirect
    inservice
serverfarm host TEST_SF
  rserver NETMAN1LDS_TEST 80
    inservice

ssl-proxy service TDB-Web-SSL-PROXY
  key abc.abc.ace.pem
  cert abc.abc.ace.pem
  chaingroup TDB-chain
ssl-proxy service TEST_ORION_PROXY
  key healthspace-2048-key
  cert HealthspaceSignedCert-V2
  chaingroup Verisign-generic

class-map match-all ICMP_INSPECT_CLASS
  2 match access-list ICMP_ACL
class-map match-any NAT_CLASS
  2 match access-list NAT_ACCESS
class-map match-all TDB-Web-80
  2 match virtual-address 10.97.88.12 tcp eq www
class-map match-all TDB_443_CM
  2 match virtual-address 10.97.88.12 tcp eq https
class-map match-all TDB_CM
  2 match virtual-address 10.97.88.13 tcp eq www
class-map match-all TEST_ORION_443_CM
  2 match virtual-address 192.168.173.130 tcp eq https
class-map match-all TEST_ORION_CM
  2 match virtual-address 192.168.173.130 tcp eq www

policy-map type management first-match mgmt-pm
  class class-default
    permit

policy-map type loadbalance first-match TDB_PM
  class class-default
    serverfarm TDB_SF
policy-map type loadbalance first-match TDB_PM-80
  class class-default
    serverfarm TDB_SF_Redirect
policy-map type loadbalance first-match TEST_ORION_PM
  class class-default
    serverfarm TEST_SF

policy-map multi-match ICMP_INSPECT_POLICY
  class ICMP_INSPECT_CLASS
    inspect icmp error
policy-map multi-match NAT_POLICY
  class NAT_CLASS
    nat dynamic 1 vlan 300
  class TEST_ORION_CM
    loadbalance vip inservice
    loadbalance policy TEST_ORION_PM
    loadbalance vip icmp-reply active
policy-map multi-match PM_MULTI_MATCH
  class TEST_ORION_CM
    loadbalance vip inservice
    loadbalance policy TEST_ORION_PM
    loadbalance vip icmp-reply active
  class TEST_ORION_443_CM
    loadbalance vip inservice
    loadbalance policy TEST_ORION_PM
    loadbalance vip icmp-reply active
    ssl-proxy server TEST_ORION_PROXY
  class TDB_CM
    loadbalance vip inservice
    loadbalance policy TDB_PM
    loadbalance vip icmp-reply active
  class TDB_443_CM
    loadbalance vip inservice
    loadbalance policy TDB_PM
    loadbalance vip icmp-reply active
    ssl-proxy server TDB-Web-SSL-PROXY
  class TDB-Web-80
    loadbalance vip inservice
    loadbalance policy TDB_PM-80

service-policy input PM_MULTI_MATCH

interface vlan 300
  ip address 192.168.62.36 255.255.255.248
  alias 192.168.62.37 255.255.255.248
  peer ip address 192.168.62.35 255.255.255.248
  access-group input ALLOW-TRAFFIC
  nat-pool 1 192.168.62.38 192.168.62.38 netmask 255.255.255.248 pat
  service-policy input ICMP_INSPECT_POLICY
  service-policy input mgmt-pm
  service-policy input NAT_POLICY
  no shutdown
interface vlan 301
  ip address 192.168.62.44 255.255.255.248
  alias 192.168.62.45 255.255.255.248
  peer ip address 192.168.62.43 255.255.255.248
  access-group input ALLOW-TRAFFIC
  service-policy input ICMP_INSPECT_POLICY
  service-policy input mgmt-pm
  no shutdown

ip route 0.0.0.0 0.0.0.0 192.168.62.33
ip route 192.168.66.0 255.255.255.0 192.168.62.41
ip route 192.168.225.0 255.255.255.192 192.168.62.41
ip route 192.168.225.128 255.255.255.192 192.168.62.41
ip route 192.168.249.0 255.255.255.240 192.168.62.41
ip route 192.168.249.16 255.255.255.240 192.168.62.41

0 Helpful

Gilles Dufour
Cisco Employee
Cisco Employee
In response to Amjad Hashim

‎07-31-2012 05:42 AM

Hello,

you can verify if your traffic will  be permitted with the following command :

show np 1 access-list trace vlan 301 in proto 6 source 192.168.249.21 0 des 192.168.249.69 80

You should see something like :

action_flag 0x3 (permit yes ...

Also, get a show tech before and after a connection failure and send it to us so we can check the drop counters.

Could you also clarify if the connection works and then stops suddenly or the connection is never established ?

Could you export the sniffer trace in pcap format and not text so that we can analyse it with wireshark.

Thanks.

Gilles.

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents