cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1125
Views
0
Helpful
6
Replies

tools for troubleshooting

donnie
Level 1
Level 1

Hi all. My HQ office has a server that pulls sql data from other database servers located in our overseas branches.

There is a particular site where my HQ server frequently fail to pull the sql data. My HQ traffic goes through a asa 5510 firewall while traffic from that particular oversea branch office goes through a cisco pix. Is there any tools from cco or any other tools that can be used to help to troubleshoot and find the root cause for this? THks in advance.

6 Replies 6

mirober2
Cisco Employee
Cisco Employee

Hi Don,

One useful tool for troubleshooting this is packet captures. Here is a link that describes how to configure them on the firewall and then download them for analysis in Wireshark:

https://supportforums.cisco.com/docs/DOC-1222

I would recommend setting up simultaneous, bi-directional packet captures on the ingress and egress interfaces of the ASA and the PIX so you can see if/how the firewalls are affecting the flow. I would also suggest looking at syslogs generated by the firewall and the output of 'show asp drop' to see if any packets in the flow are being dropped.

Also, check to see if the SQLnet inspection is enabled for this traffic on either firewall. If so, you could try disabling/enabling it to see if that makes a difference.

Hope that helps.

-Mike

Hi mike,

Thk you very much for the prompt response. Apologies for the delay as i am away. Base on the example from the link you provided, buffer is set to 1000000. Does it mean that the capture is set to hold up to 1000000 packets after which it will stop? How do i clear the buffer after that? What is the suitable packet-length to set? Thks in advance.

Hi Don,

The buffer size is measured in bytes rather than packets, so 1000000 means that the buffer will hold about 1 MB of data. After the buffer is full (which you can check in the output of 'show capture'), the capture will simply stop. To clear the data and restart the capture, you can use the 'clear capture ' command.


The default packet-length is 1518 bytes (unless you are running an older version of code) and this is usually fine for most troubleshooting. Unless you are working in an environment that uses an MTU larger than 1500 you won't likely need to increase this. There are times when you might want to decrease the size if you are only looking for the IP or TCP headers and you don't care about capturing the payload of the packet. In most cases, the default is just fine.

Hope that helps.

-Mike

Another great tool is ASA packet tracer, which let you emulate certain traffic and verify if the different security policies (access lists , nat, etc) are going to allow the traffic or drop it.

You can find it in ASDM (menu "Tools > Packet tracer") or you can also use it from CLI with "packet-tracer" command.

This is an example, showing you that traffic is denied because of the implicit acl rule.

ciscoasa(config)#packet-tracer input outside icmp 172.22.1.6 8 0 172.16.10.1 detailed
Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   172.16.10.0     255.255.255.0   outside

Phase: 3
Type: ACCESS-LIST
Subtype:

Result: DROP

Config:

Implicit Rule

Hi Mirober2,

I got the output of my packet capture of the communication between the problematic site and my HQ as below. I believe they are mostly related to 3way handshake. What should i look out for to conclude possible problem? pls advise.

   1: 14:03:17.410089 192.168.x.x.39428 > 222.x.x.x.1433: S 2270247581:2270247581(0) win 65535

   2: 14:03:20.382548 192.168.x.x.39428 > 222.x.x.x.1433: S 2270247581:2270247581(0) win 65535

   3: 14:03:20.657696 222.x.x.x.1433 > 192.168.x.x.39428: S 1403769533:1403769533(0) ack 2270247582 win 16384

   4: 14:03:20.658733 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403769534 win 65535

   5: 14:03:20.658779 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247582:2270247634(52) ack 1403769534 win 65535

   6: 14:03:20.928342 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769534:1403769571(37) ack 2270247634 win 65483

   7: 14:03:20.929502 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247634:2270247866(232) ack 1403769571 win 65498

   8: 14:03:21.200184 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769571:1403769936(365) ack 2270247866 win 65251

   9: 14:03:21.212162 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

  10: 14:03:23.604323 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

  11: 14:03:24.049741 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769571:1403769936(365) ack 2270247866 win 65251

  12: 14:03:24.052746 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403769936 win 65133

  13: 14:03:28.426506 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

  14: 14:03:37.986536 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

  15: 14:03:38.189473 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769936:1403770114(178) ack 2270248036 win 65081

  16: 14:03:38.384394 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403770114 win 65535

  17: 14:03:40.243166 192.168.x.x.39428 > 222.x.x.x.1433: P 2270248036:2270248168(132) ack 1403770114 win 65535

  18: 14:03:49.754157 192.168.x.x.39428 > 222.x.x.x.1433: P 2270248036:2270248168(132) ack 1403770114 win 65535

  19: 14:03:49.879547 222.x.x.x.1433 > 192.168.x.x.39428: P 1403770114:1403771387(1273) ack 2270248168 win 64949

  20: 14:03:50.154548 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535

  21: 14:04:05.103052 192.168.x.x.39431 > 222.x.x.x.1433: S 1317797996:1317797996(0) win 65535

  22: 14:04:05.198491 222.x.x.x.1433 > 192.168.x.x.39431: S 2552889815:2552889815(0) ack 1317797997 win 16384

  23: 14:04:05.198628 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552889816 win 65535

  24: 14:04:05.198674 192.168.x.x.39431 > 222.x.x.x.1433: P 1317797997:1317798049(52) ack 2552889816 win 65535

  25: 14:04:08.159247 192.168.x.x.39431 > 222.x.x.x.1433: P 1317797997:1317798049(52) ack 2552889816 win 65535

  26: 14:04:08.253374 222.x.x.x.1433 > 192.168.x.x.39431: P 2552889816:2552889853(37) ack 1317798049 win 65483

  27: 14:04:08.253694 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798049:1317798281(232) ack 2552889853 win 65498

  28: 14:04:08.349911 222.x.x.x.1433 > 192.168.x.x.39431: P 2552889853:2552890218(365) ack 1317798281 win 65251

  29: 14:04:08.350338 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798281:1317798593(312) ack 2552890218 win 65133

  30: 14:04:08.454459 222.x.x.x.1433 > 192.168.x.x.39431: . 2552890218:2552891598(1380) ack 1317798593 win 64939

  31: 14:04:08.454551 222.x.x.x.1433 > 192.168.x.x.39431: P 2552891598:2552891691(93) ack 1317798593 win 64939

  32: 14:04:08.454932 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552891691 win 65535

  33: 14:04:08.458457 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798593:1317798831(238) ack 2552891691 win 65535

  34: 14:04:08.554872 222.x.x.x.1433 > 192.168.x.x.39431: P 2552891691:2552891745(54) ack 1317798831 win 64701

  35: 14:04:08.555635 192.168.x.x.39431 > 222.x.x.x.1433: F 1317798831:1317798831(0) ack 2552891745 win 65481

  36: 14:04:08.679850 222.x.x.x.1433 > 192.168.x.x.39431: . ack 1317798832 win 64701

  37: 14:04:08.679881 222.x.x.x.1433 > 192.168.x.x.39431: F 2552891745:2552891745(0) ack 1317798832 win 64701

  38: 14:04:08.680567 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552891746 win 65481

  39: 14:04:19.840425 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  40: 14:04:19.932721 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  41: 14:04:50.104791 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  42: 14:04:51.117242 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  43: 14:04:52.219242 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  44: 14:04:52.311675 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  45: 14:05:22.195699 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  46: 14:05:22.291183 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  47: 14:05:52.179754 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  48: 14:05:52.273774 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  49: 14:06:22.354473 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  50: 14:06:22.447791 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  51: 14:06:52.526584 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  52: 14:06:53.535113 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  53: 14:06:53.628644 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  54: 14:07:23.622968 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  55: 14:07:24.620542 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  56: 14:07:24.713433 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  57: 14:07:54.603209 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  58: 14:07:54.697092 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  59: 14:08:24.669856 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  60: 14:08:24.763678 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  61: 14:08:50.199803 222.x.x.x.1433 > 192.168.x.x.39428: . 1403771386:1403771387(1) ack 2270248168 win 64949

  62: 14:08:50.200642 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535

  63: 14:08:54.746192 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  64: 14:08:54.875702 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  65: 14:09:24.931180 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  66: 14:09:25.027952 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  67: 14:09:55.004119 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  68: 14:09:55.102121 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  69: 14:10:25.077983 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  70: 14:10:25.174353 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  71: 14:10:55.157508 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  72: 14:10:55.253389 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  73: 14:11:25.247362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  74: 14:11:25.339749 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  75: 14:11:55.513446 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  76: 14:11:55.685191 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  77: 14:12:25.690500 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  78: 14:12:25.786504 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  79: 14:12:55.664989 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  80: 14:12:56.670909 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  81: 14:12:57.687525 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  82: 14:12:57.783956 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  83: 14:13:27.753913 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  84: 14:13:27.873093 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  85: 14:13:50.329954 222.x.x.x.1433 > 192.168.x.x.39428: . 1403771386:1403771387(1) ack 2270248168 win 64949

  86: 14:13:50.331693 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535

  87: 14:13:57.831362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  88: 14:13:58.843248 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  89: 14:13:59.942929 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  90: 14:14:00.060086 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  91: 14:14:30.120797 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  92: 14:14:30.237872 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

Hi Don,

It's hard to identify a cause just based on the capture, but you can see that there is packet loss between the client and the server. You can see that the client tries several times to contact the server and ask for more data, but nothing new ever comes. The server just resends his previous data, so he doesn't know the client is asking for more data:

87: 14:13:57.831362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  88: 14:13:58.843248 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  89: 14:13:59.942929 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  90: 14:14:00.060086 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

  91: 14:14:30.120797 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

  92: 14:14:30.237872 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

I assume this capture was taken on the firewall at the client's side. You'll need to trace this through the topology and find out where the packets are being lost. Check the firewalls to make sure the connections aren't being torn down ('show conn' and syslogs will help you with that). Also repeat the captures at various points in the network and find out which device is dropping the packets.


Hope that helps.

-Mike

Review Cisco Networking for a $25 gift card