Troubleshooting connection failure from client.

Chris Thuys · ‎11-13-2012

Hi,

I have a ACE4710 setup to load balance a couple of web servers. The real servers all show as inservice as do the propbes and serverfarms/virtual servers. If I ping the Virtuual server ip address I get a reply but it I try to access VIP via telnet or web browser. I get a connection could not be open error on the client.

The question is how do i determine where the error is comming from so far I can not tell if the client is getting through the acl or not.

I have used the trouble shooting guide and nothing has helped to determine the cause so far.

show service-policy int479 detail does not show an increase in the hit count when I try to connect.

show stats conn does not show an increase in failed or timed out connections when i try to connect

my config is as below

logging enable

logging fastpath

logging console 4

logging timestamp

logging trap 4

logging history 4

logging buffered 7

logging persistent 4

logging monitor 7

logging device-id context-name

logging host 10.19.56.100 udp/514 format emblem

logging host 10.19.56.101 udp/514 format emblem

access-list ACL_Permit_all line 21 extended permit ip any host 10.19.204.22

access-list ACL_Permit_all line 31 extended permit ip any host 10.19.204.30

access-list ACL_Permit_all line 41 extended permit ip any host 10.19.204.21

access-list ACL_Permit_all line 51 extended permit ip any host 10.19.204.23

access-list ACL_Permit_all line 61 extended permit ip any any

access-list ESP-04-test-vip line 1 extended permit ip host 172.30.0.10 host 10.19.204.30

probe tcp TCP_Probe

interval 5

passdetect interval 20

open 10

rserver host sdc-esp4sgt01

ip address 10.19.204.25

inservice

rserver host sdc-esp4sgt02

ip address 10.19.204.26

inservice

serverfarm host esp-int-04-4848

failaction purge

probe TCP_Probe

rserver sdc-esp4sgt01 4848

inservice

rserver sdc-esp4sgt02 4848

inservice

serverfarm host esp-int-04-8080

failaction purge

probe TCP_Probe

rserver sdc-esp4sgt01 8080

inservice

rserver sdc-esp4sgt02 8080

inservice

sticky ip-netmask 255.255.255.255 address source esp-int-04-4848-sticky

serverfarm esp-int-04-4848

replicate sticky

class-map type management match-any REMOTE_ACCESS

description Remote access traffic match

2 match protocol ssh any

3 match protocol icmp any

5 match protocol https any

class-map match-all esp-int-04-4848

2 match virtual-address 10.19.204.30 tcp eq 4848

class-map match-all esp-int-04-8080

2 match virtual-address 10.19.204.30 tcp eq 8080

policy-map type management first-match mgmt-pm

class REMOTE_ACCESS

permit

class class-default

permit

policy-map type loadbalance first-match esp-int-04-4848-l7slb

class class-default

sticky-serverfarm esp-int-04-4848-sticky

policy-map type loadbalance first-match esp-int-04-8080-l7slb

class class-default

serverfarm esp-int-04-8080

policy-map multi-match int479

class esp-int-04-4848

loadbalance vip inservice

loadbalance policy esp-int-04-4848-l7slb

loadbalance vip icmp-reply active

nat dynamic 1 vlan 479

class esp-int-04-8080

loadbalance vip inservice

loadbalance policy esp-int-04-8080-l7slb

loadbalance vip icmp-reply active

nat dynamic 1 vlan 479

interface vlan 479

description "Webmethods Internal Test VLAN"

ip address 10.19.204.20 255.255.255.0

access-group input ACL_Permit_all

access-group output ACL_Permit_all

nat-pool 1 10.19.204.241 10.19.204.249 netmask 255.255.255.0 pat

service-policy input mgmt-pm

service-policy input int479

no shutdown

interface vlan 481

ip address 10.19.56.43 255.255.255.0

service-policy input mgmt-pm

no shutdown

ip route 0.0.0.0 0.0.0.0 10.19.204.1

Kanwaljeet Singh · ‎11-13-2012

Hi Chris,

When you try to access VIP, please do show conn and see if you get IN/OUT connection. No counters increasing on service policy may indicate that traffic is not hitting the ACE and i see that you have permit all ACL applied to the interface.

If ACL is denying the traffic you can easily see that in the logs. You can use show logging for the same. Also, please see show access-list detail.

If you are getting reply for the ping it means that servers are active and operational. Are you able to get the page when you access the servers directly?

Can you take a quick capture on client machine itself and see what do you get? Is TCP three way handshake completing or not etc.

Also, you can take capture on ACE itself using "capture" command and you can see what exactly is going on.

Configuration looks fine.

Regards,

Kanwal

Chris Thuys · ‎11-13-2012

Hi Kanwal,

After sleeping on the issue overnight I realised that as a ping of the VIP was responding this does not neccesarily mean that the VIP is responding to the ping. Especially when the show access-list did not show any increase in the hit count. The light then went on and I discovered I had a nother server on the network which someone had configured with the same ip. It was this server that was responding to the ping. I changed the VIP being used and everything now works.

I would still like to know how to do a packet capture though. I used the command

sdc-lbaint01/VC_WM_TEST# capture wm-test interface vlan 479 access-list ACL_Permit_all bufsize 5000

but what I get does not look like a packet capture I get from snoop/tcpdump/wireshark.

sdc-lbaint01/VC_WM_TEST# show capture wm-test

0001: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0002: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0003: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0004: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0005: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0006: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0007: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0008: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0009: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0010: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

0011: msg_type: ACE_HIT ace_id: 1739 action_flag: 0x13

Kanwaljeet Singh · ‎11-13-2012

Hi Chris,

When you ping the VIP on ACE the traffic is not sent at the backend to the servers in serverfarm and as a result you will not see any counter increasing for the service-policy.

But if you have configured "loadbalance icmp-reply active" you will only get the ping response when servers in serverfarm are "operational" otherwise you will not get the response. So in a way if you are getting the ping response it does indicate that servers are up and running. You can also configure the command like "loadbalance icmp-reply".

Regarding the pcaps, please see the link below;

http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_(ACE)_Troubleshooting_Guide_--_Overview_of_ACE_Troubleshooting

Please visit section: capturing packets in real time

After configuring capture name and acl you need to start the capture, stop the capture and save the capture in disk. Get the capture out of the disk using ftp/tftp and open it in wireshark. Should look like a normal capture.

Let me know if you have any questions.

Regards,

Kanwal