Introduction   This article aims at addressing a problem wherein snort may cause connection failures without dropping any traffic due to some snort rules   Problem   Snort cause connection failures for rules with "replace" keyword even if the rule state is set to "Generate Events".   Solution   The reason is, currently snort does not look at the rule state before enforcing the "replace" action, as a result if a rule state says "generate events", snort will still modify the packet and cause connection failures. An enhancement request has been opened to change this behavior https://tools.cisco.com/bugsearch/bug/CSCus41655/?reffering_site=dumpcr One such example is SID - 24097 to detect Team viewer. Customers would want to enable such signatures to monitor network and not necessarily drop such traffic.   List of signatures with replace keyword   (1:12031) CONTENT-REPLACE MSN deny in-bound file transfer attempts (1:12032) CONTENT-REPLACE MSN deny out-bound file transfer attempts (1:12033) CONTENT-REPLACE Jabber deny in-bound file transfer attempts (1:12034) CONTENT-REPLACE Jabber deny out-bound file transfer attempts (1:12035) CONTENT-REPLACE IRC deny in-bound file transfer attempts (1:12036) CONTENT-REPLACE IRC deny out-bound file transfer attempts (1:12037) CONTENT-REPLACE AIM deny in-bound file transfer attempts (1:12038) CONTENT-REPLACE AIM deny out-bound file transfer attempts (1:12039) CONTENT-REPLACE Yahoo Messenger deny in-bound file transfer attempts (1:12040) CONTENT-REPLACE Yahoo Messenger deny out-bound file transfer attempts (1:12041) CONTENT-REPLACE Yahoo Messenger V7 deny in-bound file transfer attempts (1:12042) CONTENT-REPLACE Yahoo Messenger V7 deny out-bound file transfer attempts (1:15415) CONTENT-REPLACE AIM or ICQ deny unencrypted login connection (1:15416) CONTENT-REPLACE ICQ deny http proxy login (1:15417) CONTENT-REPLACE AIM deny server certificate for encrypted login (1:15420) CONTENT-REPLACE MSN deny login (1:15421) CONTENT-REPLACE AIM or ICQ deny login for unencrypted connection (1:15429) CONTENT-REPLACE Yahoo Messenger deny outbound login attempt (1:15438) CONTENT-REPLACE QQ 2009 deny udp login (1:15439) CONTENT-REPLACE QQ 2009 deny tcp login (1:15440) CONTENT-REPLACE QQ 2008 deny udp login (1:15441) CONTENT-REPLACE QQ 2009 deny tcp login (1:15570) CONTENT-REPLACE Google Talk deny login (1:18469) CONTENT-REPLACE Microsoft Windows Encrypted DCERPC request attempt (1:24096) APP-DETECT Teamviewer remote connection attempt (1:24097) APP-DETECT Teamviewer remote connection attempt (1:24098) APP-DETECT Teamviewer remote connection attempt    Workaround   If you want to enable the above rules to only alert and not cause traffic failures, the only workaround available right now is to clone a local rule of the original rule and remove the "replace" keyword and enable this local rule instead of the rule provided by Sourcefire.   Please note: Removing the replace keyword may limit the effectiveness of the rule     ... View more

    What are Packet Captures - A Brief Introduction to Packet Captures   Packet capture is a activity of capturing data packets crossing networking devices There are 2 types - Partial packet capture and Deep packet capture   Partial packet capture just record headers without recording content of datagrams, used for basic troubleshooting upto L4   Deep packet capture will give us everything that a packet can tell, doing a deep packet analysis is like investigating in a forensic lab, it is used in advanced troubleshooting like troubleshooting at L7, troubleshooting for performance related issues, traffic patterns etc   There are 2 ways of looking at traffic coming to any device, either collect captures on the ingress of the device or collect captures on the egress interface of the device behind the device in question   It might be sometimes necessary to collect captures on the egress interface, for example in case our device is dropping packets even before it is processing it or if we have to collect captures for large data as captures on some devices are limited by buffer size   Collecting captures on ASA   You can enable captures on ASA either from CLI or from ASDM   Enable captures on ASDM   Go to wizards and select packet capture wizard, it will take you through 6 simple self explanatory steps, once done with captures select save captures. This has been illustrated in Scenario 1   Enable captures in CLI   This is the syntax to apply capture   capture <name of capture>   These are the options available     access-list       Capture packets that match access-list, when you specify access-list make sure that you specify the traffic in both direction if you want to capture bi-directional traffic buffer Default is 512 KB and you can configure it upto 32 MB, you do not need to change this in most cases. Just a note of caution - applying captures will add to  memory utilization so keep an eye on memory before enabling captures  with max buffer circular-buffer Overwrite buffer from beginning when full, default is non-circular ethernet-type EtherType is a two-octet field in an Ethernet frame. It is used to  indicate which protocol is encapsulated in the PayLoad of an Ethernet  Frame. Default is IP Here is link for iana assigned ethernet type numbers http://www.iana.org/assignments/ethernet-numbers headers-only Capture only L2, L3 and L4 headers of packet without data in them, useful for collecting partial packet capture interface Used to specify the interface on which you want to apply the capture match Capture packets matching five-tuple - 5 tuple consists of  ->   Type of protocol - eg ip, gre, esp, icmp etc>     ->   Source Destination IP ->   and other specific detail related to type of protocol specified for  example in case of tcp it would be src dst port or in case of icmp it  would be icmp type (optional) packet-length Defines maximum length of each packet to capture, default is 1518 bytes which is the mtu in most cases, maximum is 9216 bytes real-time Display captured packets in real-time. Warning: using this option with a slow console connection may result in an excessive amount of non-displayed packets due to performance limitations. This is very rarely useful trace This keyword enables you to check the output of packet tracer for  each packet, note that this will show packet tracer output only for  inbound packets. This is useful in cases when you want to check  the various checks in firewall for consecutive packets as the normal  packet tracer command will always show you output for new connection check the view packet capture section to learn how to check the trace output. type These are the various option available here             asp-drop  Capture packets dropped with a particular reason           isakmp    Capture encrypted and decrypted ISAKMP payloads           raw-data   Capture inbound and outbound packets on one or more interfaces           tls-proxy   Capture decrypted inbound and outbound data from TLS Proxy on one or more interfaces           webvpn    Capture WebVPN transactions for a specified user   You need to know what you are looking for when you want to collect  these captures, for example asp drop captures might generate lot of  output so unless you dont know what kind of drop you are looking for you  might end up looking at lot of packets   Example of capture   capture capin interface inside match ip host 1.1.1.1 host 2.2.2.2 ----> this will use defaults for other parameters     Viewing captures   You can view captures in 2 ways view it on CLI/ASDM or in other words view it on the device itself or you can view it on a packet analyser after exporting it in pcap form   Let us examine each of them closely   Viewing it on the device itself   You can watch the captures in real time when you enable it on asdm or you can watch it real time when you enable capture on cli using the option "real-time" (not really recommended as it may lead to excessive amount of non displayed packets in some cases)   Once you are done with capturing you can view them by issueing the command show capture <capname> this will display minimum information - src dst ip, src dst port, timestamp and ethertype   You can view some more information by using the extended form of show cap <capname>   show cap <capname> <one of the keywords below>   access-list Display packets matching access-list count Display <number> of packets in capture - lets you display specified number of packets detail Display more information for each packet - like src dst mac address, ttl, ip id - this has been illustrated in Scenario 1 dump Display hex dump for each packet - shows datagram in hex packet-number Display packet <number> in capture - lets you view captures starting from a specified packet number trace Display extended trace information for each packet - used if capture is set using the trace keyword as mentioned above, this will show the output of packet tracer for each packet in the inbound direction     Viewing it on a packet anaylser tool   You can export these captures and save them on your PC and view it using a packet anaylser tool like wireshark (open source tool available for free on internet). there are 2 ways of doing this   Export via https   For this you need to enable http server on your ASA and you need to know the credentials used to access asa via asdm (default is no username no password)   Comamnds to enable http server     asa#config t asa(config)# http server enable asa(config)# crypto key generate rsa modulus 1024   Note: This is for creating keys because we communicate with asa via https, if you have ssh access you probably have these keys       Once you have enabled http server on asa go to your browser and give the following in the url field     https:// <ip address of asa>/capture/<capname>/pcap   if it is in multiple context mode you have to specify the context   https://<ip address of asa>/capture/<context name>/<capname>/pcap   After you enter this you will be prompted for username password and once you enter that the captures are stored on your PC and you can open them in a packet anaylser tool   Export via copy command   copy /pcap capture:   disk flash ftp smb system tftp       As metioned before in some cases you might need to capture packets on devices directly connected to asa and in most cases it is a switch connected to ASA and in such cases you can also span the switchport to collect captures   Here is a link which will help you setup a span on your catalyst switch   http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml#config   Once done always make sure that you remove the captures using the command   no capture <capname>   Troubleshooting simple scenario's using packet capture   Scenario - 1 - Basic Connectivity   To verify if ASA is dropping any packet - simple connectivity issues   Topology:       So here we nat all users to interface ip   Commands to capture traffic   Inside:   access-list capi extended permit ip host 192.168.1.2 host 4.2.2.2 access-list capi extended permit ip host 4.2.2.2 host 192.168.1.2 capture capin interface inside access-list capi   Or   capture capin interface inside match ip host 192.168.1.2 host 4.2.2.2   [this is possible in asa 8.0 and above and we do not need to be in config mode to put apply an capture]   Outside:   access-list capo extended permit ip host a.b.c.d host 4.2.2.2 access-list capo extended permit ip host 4.2.2.2 host a.b.c.d capture capout interface outside access-list capo   Or   capture capout interface outside match ip host a.b.c.d host 4.2.2.2   [Note that we are using the natted ip - so for capture  use the ip addresses that you expect to see on the wire after all  processing is done for egress interface and before any processing is  done for ingress interface]   We can also apply capture using ASDM and the below screen shots show the steps for that   Ping from 192.168.1.2 to 4.2.2.2   Pings Succed         show capture capin     192.168.1.2> 4.2.2.2: icmp: echo request     4.2.2.2 > 192.168.1.2: icmp: echo reply   192.168.1.2 > 4.2.2.2: icmp: echo request     4.2.2.2 > 192.168.1.2: icmp: echo reply   show capture capout     a.b.c.d> 4.2.2.2: icmp: echo request      4.2.2.2 > a.b.c.d: icmp: echo reply   a.b.c.d > 4.2.2.2: icmp: echo request      4.2.2.2 > a.b.c.d: icmp: echo reply   So you can see we have captured bidirectional traffic   If you want to see more details you can use the detailed keyword at the end   show capture capin detail        000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo request ( ttl 128, id 52241 )     0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply (ttl 127, id 16992)     000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo request (ttl 128, id 52242)     0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply (ttl 127, id 16993)     000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo request (ttl 128, id 52243)     0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply (ttl 127, id 17008)     000c.29d6.7dca 0026.0b09.420c 0x0800 74: 192.168.1.2 > 4.2.2.2: icmp: echo request (ttl 128, id 52244)     0026.0b09.420c 000c.29d6.7dca 0x0800 74: 4.2.2.2 > 192.168.1.2: icmp: echo reply (ttl 127, id 17251)   show capture capout detail       0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request ( ttl 128, id 5224 1)     0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl 127, id 16992)     0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request (ttl 128, id 52242)     0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl 127, id 16993)     0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request (ttl 128, id 52243)     0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl 127, id 17008)     0026.0b09.420d 0022.556d.f140 0x0800 74: a.b.c.d > 4.2.2.2: icmp: echo request (ttl 128, id 52244)     0022.556d.f140 0026.0b09.420d 0x0800 74: 4.2.2.2 > a.b.c.d: icmp: echo reply (ttl 127, id 17251)   You can see we have some additonal information here like mac address, ip id, ttl etc.   TIP: For basic connectivity issues always check the following   -> interface access-list -> nat rules -> if pings are not working check for      inspect icmp -> Check if you have this in the policy-map, you can either add this or explicitly add acl's to permit icmp packets in access-lists on the lower security level interfaces        icmp deny any outside -> check if you have any deny statements like this, this statement means that we deny any icmp traffic on outisde  interface   Once you have checked the above use packet tracer and packet capture to isolate the issue further       Scenario - 2 - ICMP works but TCP traffic does not work     Users report that they can ping a particular server but cannot access any TCP services on it   Topology:       This is a typical case of asymmetric routing were users complain that they can ping the server but cannot access it on any of the tcp services   Lets see how captures help us in such scenarios       Commands to capture traffic   Inside:   access-list capi extended permit ip host 192.168.1.2 host x.x.x.x access-list capi extended permit ip host x.x.x.x host 192.168.1.2 capture capin interface inside access-list capi   Or   capture capin interface inside match ip host 192.168.1.2 host x.x.x.x   [this is possible in asa 8.0 and above and we do not need to be in config mode to put apply an capture]   Outside:   access-list capo extended permit ip host a.b.c.d host x.x.x.x access-list capo extended permit ip host x.x.x.x host a.b.c.d capture capout interface outside access-list capo   Or   capture capout interface outside match ip host a.b.c.d host x.x.x.x   [Note  that we are using the natted ip - so for capture  use the ip addresses  that you expect to see on the wire after all  processing is done for  egress interface and before any processing is  done for ingress  interface]     When the user tries to browse a webpage on this server   Capture Outputs   show cap capin   192.168.1.2.58458 > x.x.x.x.80: S 2331481362:2331481362(0) win 5840 <mss 1460,sackOK,timestamp 2359730964 0,nop,wscale 7>   192.168.1.2.58458 > x.x.x.x.80: . ack 712296789 win 5840   show cap capout   a.b.c.d .58458 > x.x.x.x.80: S 126872520:126872520(0) win 5840 <mss 1380,sackOK,timestamp 2359730964 0,nop,wscale 7>   Captures on client        192.168.1.2.58458 > x.x.x.x.80: S 2331481362:2331481362(0) win 5840 <mss 1460,sackOK,timestamp 2359730964 0,nop,wscale 7>        192.168.1.2.58458 > x.x.x.x.80: S 126872520:126872520(0) win 5840 <mss 1380,sackOK,timestamp 2359730964 0,nop,wscale 7>        192.168.1.2.58458 > x.x.x.x.80: . ack 712296789 win 5840       So as you can see the ASA just saw one side of traffic and dropped the ack from the client to the server becuase it did not see the syn-ack go to the client     So this suggests that there is asymmetric routing   Solution:   Correct this assymetric routing and make sure that ASA see's both sides of traffic. For any security appliance performing tcp checks it is important that it see's both sides of traffic. Sometimes it is unavoidable and we have to live with asymmetric routing in that case we can configure tcp state bypass for this traffic (you need to run asa version 8.2.1 and later )   http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a0080b2d922.shtml     Scenario - 3 - Capture Backplane traffic to troubleshoot module related issues   Some times you might need to capture backplane captures hwen troubleshooting module related issues   Here is the command to enable backplane captures on dataplane   capture <capname> interface asa_dataplane   Some scenario's where these could be useful -> Some websites not accesible when traffic passes through csc module -> If dataplane communication issue is reported in the logs   Here is the command to enable backplane captures on control plane   capture <capname> interface cplane   We will need control plane captures to troubleshoot issues related to communication between asa and module   If the issue is one of the above it will be helpful to attach the captures while opening a TAC case     Scenario - 4 - VPN troubleshooting using captures   A variety of VPN issues can be troubleshooted using packet captures. Packet captures are easy to read and understand if we know what exactly we need to capture. As far as VPN is concered we are mainly concerned about the traffic between peer IP's on the internet facing side and traffic between internal subnets on the internal side.   You can capture the traffic in the same way as explained in the previous sections, the intention on this section is to give an idea on what captures to apply for specific issues   Lan to Lan Tunnel (Site to Site)   Tunnel not coming UP, show crypto isakmp sa does not show up anything on either sides        There is something wrong in the config the tunnel was never initiated Tunnel not coming UP, show crypto isakmp sa shows that tunnel is initited on one of the side but on the responder nothing shows up in this output        The phase one for IPSEC VPN uses udp 500 so apply captures for this on both sides and verify that you are actually getting the packets on the responder side. If you dont receive the packets something in the way is blokcing these packets. Check if you have the required ports open on the upstream devices and check with both of the ISP's if they have the required ports open, if you receive the packets and still nothing shows up its probably a configuration issue check configuration Tunnel is UP, show crypto isakmp sa shows that the tunnel is up (state is MM_ACTIVE on ASA and QM_IDLE on Router) but you are unable to pass any traffic between the 2 sites        Apply captures between the peer ip's and check that there is 2 way traffic for ip protocol 50 or 51 (ESP or AH depending on what you are using). Again as before if you see packets leaving one side and not reaching the other check upstrean devices and check with both the ISP's If you see 2 way traffic for ESP or AH check if you see encap's/decap's in the output of show crypto ipsec sa, if you don't see incremtns then you might want to apply captures to check if we are sending and receiving packets for the traffic between the 2 subnets If any of the device is behind a nat device and nat traversal is enabled you will capture packets for udp 4500 instead of ESP or AH   Remote Access VPN   Most of the above is applicable to this as well, just one addition and that is sometimes we might use tcp 10000 for phase 2, so as i said before it is very important to know what traffic we are trying to capture in order to make troubleshooting easy using packet capture ... View more