12-05-2010 11:25 PM - edited 03-11-2019 12:18 PM
Hi all. My HQ office has a server that pulls sql data from other database servers located in our overseas branches.
There is a particular site where my HQ server frequently fail to pull the sql data. My HQ traffic goes through a asa 5510 firewall while traffic from that particular oversea branch office goes through a cisco pix. Is there any tools from cco or any other tools that can be used to help to troubleshoot and find the root cause for this? THks in advance.
12-06-2010 05:58 AM
Hi Don,
One useful tool for troubleshooting this is packet captures. Here is a link that describes how to configure them on the firewall and then download them for analysis in Wireshark:
https://supportforums.cisco.com/docs/DOC-1222
I would recommend setting up simultaneous, bi-directional packet captures on the ingress and egress interfaces of the ASA and the PIX so you can see if/how the firewalls are affecting the flow. I would also suggest looking at syslogs generated by the firewall and the output of 'show asp drop' to see if any packets in the flow are being dropped.
Also, check to see if the SQLnet inspection is enabled for this traffic on either firewall. If so, you could try disabling/enabling it to see if that makes a difference.
Hope that helps.
-Mike
12-08-2010 06:19 PM
Hi mike,
Thk you very much for the prompt response. Apologies for the delay as i am away. Base on the example from the link you provided, buffer is set to 1000000. Does it mean that the capture is set to hold up to 1000000 packets after which it will stop? How do i clear the buffer after that? What is the suitable packet-length to set? Thks in advance.
12-09-2010 05:14 AM
Hi Don,
The buffer size is measured in bytes rather than packets, so 1000000 means that the buffer will hold about 1 MB of data. After the buffer is full (which you can check in the output of 'show capture'), the capture will simply stop. To clear the data and restart the capture, you can use the 'clear capture
The default packet-length is 1518 bytes (unless you are running an older version of code) and this is usually fine for most troubleshooting. Unless you are working in an environment that uses an MTU larger than 1500 you won't likely need to increase this. There are times when you might want to decrease the size if you are only looking for the IP or TCP headers and you don't care about capturing the payload of the packet. In most cases, the default is just fine.
Hope that helps.
-Mike
12-09-2010 05:26 PM
Another great tool is ASA packet tracer, which let you emulate certain traffic and verify if the different security policies (access lists , nat, etc) are going to allow the traffic or drop it.
You can find it in ASDM (menu "Tools > Packet tracer") or you can also use it from CLI with "packet-tracer" command.
This is an example, showing you that traffic is denied because of the implicit acl rule.
ciscoasa(config)#packet-tracer input outside icmp 172.22.1.6 8 0 172.16.10.1 detailed
Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 172.16.10.0 255.255.255.0 outside
Phase: 3
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
12-12-2010 11:30 PM
Hi Mirober2,
I got the output of my packet capture of the communication between the problematic site and my HQ as below. I believe they are mostly related to 3way handshake. What should i look out for to conclude possible problem? pls advise.
1: 14:03:17.410089 192.168.x.x.39428 > 222.x.x.x.1433: S 2270247581:2270247581(0) win 65535
2: 14:03:20.382548 192.168.x.x.39428 > 222.x.x.x.1433: S 2270247581:2270247581(0) win 65535
3: 14:03:20.657696 222.x.x.x.1433 > 192.168.x.x.39428: S 1403769533:1403769533(0) ack 2270247582 win 16384
4: 14:03:20.658733 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403769534 win 65535
5: 14:03:20.658779 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247582:2270247634(52) ack 1403769534 win 65535
6: 14:03:20.928342 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769534:1403769571(37) ack 2270247634 win 65483
7: 14:03:20.929502 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247634:2270247866(232) ack 1403769571 win 65498
8: 14:03:21.200184 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769571:1403769936(365) ack 2270247866 win 65251
9: 14:03:21.212162 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133
10: 14:03:23.604323 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133
11: 14:03:24.049741 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769571:1403769936(365) ack 2270247866 win 65251
12: 14:03:24.052746 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403769936 win 65133
13: 14:03:28.426506 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133
14: 14:03:37.986536 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133
15: 14:03:38.189473 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769936:1403770114(178) ack 2270248036 win 65081
16: 14:03:38.384394 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403770114 win 65535
17: 14:03:40.243166 192.168.x.x.39428 > 222.x.x.x.1433: P 2270248036:2270248168(132) ack 1403770114 win 65535
18: 14:03:49.754157 192.168.x.x.39428 > 222.x.x.x.1433: P 2270248036:2270248168(132) ack 1403770114 win 65535
19: 14:03:49.879547 222.x.x.x.1433 > 192.168.x.x.39428: P 1403770114:1403771387(1273) ack 2270248168 win 64949
20: 14:03:50.154548 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535
21: 14:04:05.103052 192.168.x.x.39431 > 222.x.x.x.1433: S 1317797996:1317797996(0) win 65535
22: 14:04:05.198491 222.x.x.x.1433 > 192.168.x.x.39431: S 2552889815:2552889815(0) ack 1317797997 win 16384
23: 14:04:05.198628 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552889816 win 65535
24: 14:04:05.198674 192.168.x.x.39431 > 222.x.x.x.1433: P 1317797997:1317798049(52) ack 2552889816 win 65535
25: 14:04:08.159247 192.168.x.x.39431 > 222.x.x.x.1433: P 1317797997:1317798049(52) ack 2552889816 win 65535
26: 14:04:08.253374 222.x.x.x.1433 > 192.168.x.x.39431: P 2552889816:2552889853(37) ack 1317798049 win 65483
27: 14:04:08.253694 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798049:1317798281(232) ack 2552889853 win 65498
28: 14:04:08.349911 222.x.x.x.1433 > 192.168.x.x.39431: P 2552889853:2552890218(365) ack 1317798281 win 65251
29: 14:04:08.350338 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798281:1317798593(312) ack 2552890218 win 65133
30: 14:04:08.454459 222.x.x.x.1433 > 192.168.x.x.39431: . 2552890218:2552891598(1380) ack 1317798593 win 64939
31: 14:04:08.454551 222.x.x.x.1433 > 192.168.x.x.39431: P 2552891598:2552891691(93) ack 1317798593 win 64939
32: 14:04:08.454932 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552891691 win 65535
33: 14:04:08.458457 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798593:1317798831(238) ack 2552891691 win 65535
34: 14:04:08.554872 222.x.x.x.1433 > 192.168.x.x.39431: P 2552891691:2552891745(54) ack 1317798831 win 64701
35: 14:04:08.555635 192.168.x.x.39431 > 222.x.x.x.1433: F 1317798831:1317798831(0) ack 2552891745 win 65481
36: 14:04:08.679850 222.x.x.x.1433 > 192.168.x.x.39431: . ack 1317798832 win 64701
37: 14:04:08.679881 222.x.x.x.1433 > 192.168.x.x.39431: F 2552891745:2552891745(0) ack 1317798832 win 64701
38: 14:04:08.680567 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552891746 win 65481
39: 14:04:19.840425 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
40: 14:04:19.932721 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
41: 14:04:50.104791 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
42: 14:04:51.117242 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
43: 14:04:52.219242 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
44: 14:04:52.311675 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
45: 14:05:22.195699 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
46: 14:05:22.291183 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
47: 14:05:52.179754 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
48: 14:05:52.273774 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
49: 14:06:22.354473 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
50: 14:06:22.447791 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
51: 14:06:52.526584 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
52: 14:06:53.535113 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
53: 14:06:53.628644 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
54: 14:07:23.622968 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
55: 14:07:24.620542 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
56: 14:07:24.713433 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
57: 14:07:54.603209 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
58: 14:07:54.697092 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
59: 14:08:24.669856 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
60: 14:08:24.763678 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
61: 14:08:50.199803 222.x.x.x.1433 > 192.168.x.x.39428: . 1403771386:1403771387(1) ack 2270248168 win 64949
62: 14:08:50.200642 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535
63: 14:08:54.746192 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
64: 14:08:54.875702 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
65: 14:09:24.931180 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
66: 14:09:25.027952 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
67: 14:09:55.004119 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
68: 14:09:55.102121 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
69: 14:10:25.077983 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
70: 14:10:25.174353 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
71: 14:10:55.157508 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
72: 14:10:55.253389 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
73: 14:11:25.247362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
74: 14:11:25.339749 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
75: 14:11:55.513446 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
76: 14:11:55.685191 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
77: 14:12:25.690500 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
78: 14:12:25.786504 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
79: 14:12:55.664989 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
80: 14:12:56.670909 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
81: 14:12:57.687525 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
82: 14:12:57.783956 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
83: 14:13:27.753913 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
84: 14:13:27.873093 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
85: 14:13:50.329954 222.x.x.x.1433 > 192.168.x.x.39428: . 1403771386:1403771387(1) ack 2270248168 win 64949
86: 14:13:50.331693 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535
87: 14:13:57.831362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
88: 14:13:58.843248 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
89: 14:13:59.942929 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
90: 14:14:00.060086 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
91: 14:14:30.120797 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
92: 14:14:30.237872 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
12-13-2010 07:33 AM
Hi Don,
It's hard to identify a cause just based on the capture, but you can see that there is packet loss between the client and the server. You can see that the client tries several times to contact the server and ask for more data, but nothing new ever comes. The server just resends his previous data, so he doesn't know the client is asking for more data:
87: 14:13:57.831362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
88: 14:13:58.843248 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
89: 14:13:59.942929 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
90: 14:14:00.060086 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
91: 14:14:30.120797 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535
92: 14:14:30.237872 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949
I assume this capture was taken on the firewall at the client's side. You'll need to trace this through the topology and find out where the packets are being lost. Check the firewalls to make sure the connections aren't being torn down ('show conn' and syslogs will help you with that). Also repeat the captures at various points in the network and find out which device is dropping the packets.
Hope that helps.
-Mike
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide