tools for troubleshooting

donnie · ‎12-05-2010

Hi all. My HQ office has a server that pulls sql data from other database servers located in our overseas branches.

There is a particular site where my HQ server frequently fail to pull the sql data. My HQ traffic goes through a asa 5510 firewall while traffic from that particular oversea branch office goes through a cisco pix. Is there any tools from cco or any other tools that can be used to help to troubleshoot and find the root cause for this? THks in advance.

mirober2 · ‎12-06-2010

Hi Don,

One useful tool for troubleshooting this is packet captures. Here is a link that describes how to configure them on the firewall and then download them for analysis in Wireshark:

https://supportforums.cisco.com/docs/DOC-1222

I would recommend setting up simultaneous, bi-directional packet captures on the ingress and egress interfaces of the ASA and the PIX so you can see if/how the firewalls are affecting the flow. I would also suggest looking at syslogs generated by the firewall and the output of 'show asp drop' to see if any packets in the flow are being dropped.

Also, check to see if the SQLnet inspection is enabled for this traffic on either firewall. If so, you could try disabling/enabling it to see if that makes a difference.

Hope that helps.

-Mike

donnie · ‎12-08-2010

Hi mike,

Thk you very much for the prompt response. Apologies for the delay as i am away. Base on the example from the link you provided, buffer is set to 1000000. Does it mean that the capture is set to hold up to 1000000 packets after which it will stop? How do i clear the buffer after that? What is the suitable packet-length to set? Thks in advance.

mirober2 · ‎12-09-2010

Hi Don,

The buffer size is measured in bytes rather than packets, so 1000000 means that the buffer will hold about 1 MB of data. After the buffer is full (which you can check in the output of 'show capture'), the capture will simply stop. To clear the data and restart the capture, you can use the 'clear capture ' command.

The default packet-length is 1518 bytes (unless you are running an older version of code) and this is usually fine for most troubleshooting. Unless you are working in an environment that uses an MTU larger than 1500 you won't likely need to increase this. There are times when you might want to decrease the size if you are only looking for the IP or TCP headers and you don't care about capturing the payload of the packet. In most cases, the default is just fine.

Hope that helps.

-Mike

Eduardo Aliaga · ‎12-09-2010

Another great tool is ASA packet tracer, which let you emulate certain traffic and verify if the different security policies (access lists , nat, etc) are going to allow the traffic or drop it.

You can find it in ASDM (menu "Tools > Packet tracer") or you can also use it from CLI with "packet-tracer" command.

This is an example, showing you that traffic is denied because of the implicit acl rule.

ciscoasa(config)#packet-tracer input outside icmp 172.22.1.6 8 0 172.16.10.1 detailed
Phase: 1
Type: FLOW-LOOKUP
Subtype:
Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 172.16.10.0 255.255.255.0 outside

Phase: 3
Type: ACCESS-LIST
Subtype:

Result: DROP

Config:

Implicit Rule

donnie · ‎12-12-2010

Hi Mirober2,

I got the output of my packet capture of the communication between the problematic site and my HQ as below. I believe they are mostly related to 3way handshake. What should i look out for to conclude possible problem? pls advise.

1: 14:03:17.410089 192.168.x.x.39428 > 222.x.x.x.1433: S 2270247581:2270247581(0) win 65535

2: 14:03:20.382548 192.168.x.x.39428 > 222.x.x.x.1433: S 2270247581:2270247581(0) win 65535

3: 14:03:20.657696 222.x.x.x.1433 > 192.168.x.x.39428: S 1403769533:1403769533(0) ack 2270247582 win 16384

4: 14:03:20.658733 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403769534 win 65535

5: 14:03:20.658779 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247582:2270247634(52) ack 1403769534 win 65535

6: 14:03:20.928342 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769534:1403769571(37) ack 2270247634 win 65483

7: 14:03:20.929502 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247634:2270247866(232) ack 1403769571 win 65498

8: 14:03:21.200184 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769571:1403769936(365) ack 2270247866 win 65251

9: 14:03:21.212162 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

10: 14:03:23.604323 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

11: 14:03:24.049741 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769571:1403769936(365) ack 2270247866 win 65251

12: 14:03:24.052746 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403769936 win 65133

13: 14:03:28.426506 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

14: 14:03:37.986536 192.168.x.x.39428 > 222.x.x.x.1433: P 2270247866:2270248036(170) ack 1403769936 win 65133

15: 14:03:38.189473 222.x.x.x.1433 > 192.168.x.x.39428: P 1403769936:1403770114(178) ack 2270248036 win 65081

16: 14:03:38.384394 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403770114 win 65535

17: 14:03:40.243166 192.168.x.x.39428 > 222.x.x.x.1433: P 2270248036:2270248168(132) ack 1403770114 win 65535

18: 14:03:49.754157 192.168.x.x.39428 > 222.x.x.x.1433: P 2270248036:2270248168(132) ack 1403770114 win 65535

19: 14:03:49.879547 222.x.x.x.1433 > 192.168.x.x.39428: P 1403770114:1403771387(1273) ack 2270248168 win 64949

20: 14:03:50.154548 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535

21: 14:04:05.103052 192.168.x.x.39431 > 222.x.x.x.1433: S 1317797996:1317797996(0) win 65535

22: 14:04:05.198491 222.x.x.x.1433 > 192.168.x.x.39431: S 2552889815:2552889815(0) ack 1317797997 win 16384

23: 14:04:05.198628 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552889816 win 65535

24: 14:04:05.198674 192.168.x.x.39431 > 222.x.x.x.1433: P 1317797997:1317798049(52) ack 2552889816 win 65535

25: 14:04:08.159247 192.168.x.x.39431 > 222.x.x.x.1433: P 1317797997:1317798049(52) ack 2552889816 win 65535

26: 14:04:08.253374 222.x.x.x.1433 > 192.168.x.x.39431: P 2552889816:2552889853(37) ack 1317798049 win 65483

27: 14:04:08.253694 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798049:1317798281(232) ack 2552889853 win 65498

28: 14:04:08.349911 222.x.x.x.1433 > 192.168.x.x.39431: P 2552889853:2552890218(365) ack 1317798281 win 65251

29: 14:04:08.350338 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798281:1317798593(312) ack 2552890218 win 65133

30: 14:04:08.454459 222.x.x.x.1433 > 192.168.x.x.39431: . 2552890218:2552891598(1380) ack 1317798593 win 64939

31: 14:04:08.454551 222.x.x.x.1433 > 192.168.x.x.39431: P 2552891598:2552891691(93) ack 1317798593 win 64939

32: 14:04:08.454932 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552891691 win 65535

33: 14:04:08.458457 192.168.x.x.39431 > 222.x.x.x.1433: P 1317798593:1317798831(238) ack 2552891691 win 65535

34: 14:04:08.554872 222.x.x.x.1433 > 192.168.x.x.39431: P 2552891691:2552891745(54) ack 1317798831 win 64701

35: 14:04:08.555635 192.168.x.x.39431 > 222.x.x.x.1433: F 1317798831:1317798831(0) ack 2552891745 win 65481

36: 14:04:08.679850 222.x.x.x.1433 > 192.168.x.x.39431: . ack 1317798832 win 64701

37: 14:04:08.679881 222.x.x.x.1433 > 192.168.x.x.39431: F 2552891745:2552891745(0) ack 1317798832 win 64701

38: 14:04:08.680567 192.168.x.x.39431 > 222.x.x.x.1433: . ack 2552891746 win 65481

39: 14:04:19.840425 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

40: 14:04:19.932721 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

41: 14:04:50.104791 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

42: 14:04:51.117242 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

43: 14:04:52.219242 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

44: 14:04:52.311675 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

45: 14:05:22.195699 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

46: 14:05:22.291183 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

47: 14:05:52.179754 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

48: 14:05:52.273774 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

49: 14:06:22.354473 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

50: 14:06:22.447791 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

51: 14:06:52.526584 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

52: 14:06:53.535113 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

53: 14:06:53.628644 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

54: 14:07:23.622968 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

55: 14:07:24.620542 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

56: 14:07:24.713433 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

57: 14:07:54.603209 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

58: 14:07:54.697092 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

59: 14:08:24.669856 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

60: 14:08:24.763678 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

61: 14:08:50.199803 222.x.x.x.1433 > 192.168.x.x.39428: . 1403771386:1403771387(1) ack 2270248168 win 64949

62: 14:08:50.200642 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535

63: 14:08:54.746192 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

64: 14:08:54.875702 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

65: 14:09:24.931180 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

66: 14:09:25.027952 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

67: 14:09:55.004119 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

68: 14:09:55.102121 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

69: 14:10:25.077983 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

70: 14:10:25.174353 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

71: 14:10:55.157508 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

72: 14:10:55.253389 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

73: 14:11:25.247362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

74: 14:11:25.339749 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

75: 14:11:55.513446 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

76: 14:11:55.685191 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

77: 14:12:25.690500 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

78: 14:12:25.786504 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

79: 14:12:55.664989 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

80: 14:12:56.670909 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

81: 14:12:57.687525 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

82: 14:12:57.783956 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

83: 14:13:27.753913 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

84: 14:13:27.873093 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

85: 14:13:50.329954 222.x.x.x.1433 > 192.168.x.x.39428: . 1403771386:1403771387(1) ack 2270248168 win 64949

86: 14:13:50.331693 192.168.x.x.39428 > 222.x.x.x.1433: . ack 1403771387 win 65535

87: 14:13:57.831362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

88: 14:13:58.843248 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

89: 14:13:59.942929 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

90: 14:14:00.060086 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

91: 14:14:30.120797 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

92: 14:14:30.237872 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

mirober2 · ‎12-13-2010

Hi Don,

It's hard to identify a cause just based on the capture, but you can see that there is packet loss between the client and the server. You can see that the client tries several times to contact the server and ask for more data, but nothing new ever comes. The server just resends his previous data, so he doesn't know the client is asking for more data:

87: 14:13:57.831362 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

88: 14:13:58.843248 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

89: 14:13:59.942929 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

90: 14:14:00.060086 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

91: 14:14:30.120797 192.168.x.x.39428 > 222.x.x.x.1433: . 2270248167:2270248168(1) ack 1403771387 win 65535

92: 14:14:30.237872 222.x.x.x.1433 > 192.168.x.x.39428: . ack 2270248168 win 64949

I assume this capture was taken on the firewall at the client's side. You'll need to trace this through the topology and find out where the packets are being lost. Check the firewalls to make sure the connections aren't being torn down ('show conn' and syslogs will help you with that). Also repeat the captures at various points in the network and find out which device is dropping the packets.

Hope that helps.

-Mike