12-21-2012 01:44 PM - edited 03-11-2019 05:39 PM
Hey Guys,
You've been very helpful in the past for me, and was wondering if you guys have seen this issue before. We run a 6500 with an FWSM with multiple security contexts as well as cacading contexts with a "shared VLAN"
There is a problem with regards to Linux machines and our shared network. For example, we have three Linux machines in production, each in three separate VLANs. For me to communicate to these boxes from one VLAN to another I must first ping the server. If I do not ping the server it will not bring up a connection like ssh or http, etc. Below is the error I get from the FWSM that hosts the Linux server, but like I said once I ping the server the error goes away. We only have this problem with Linux machines, and it is a problem for all three of them. Is the FWSM having issues understanding something with all three Linux boxes? Below is the error I get at first, when I try to SSH from one VLAN to another VLAN with the Linux machine.
6 | Dec 21 2012 | 16:33:54 | 106015 | 10.255.12.109 | 22 | 10.255.1.30 | 63000 | Deny TCP (no connection) from 10.255.12.109/22 to 10.255.1.30/63000 flags SYN ACK on interface inside |
Below is what happens when I initiate a ping to the Linux Server and then ssh again. Notice it builds the connection with no problem after the ping. During the ping it builds the dynamic translation, and then when I ssh it builds the TCP connection. Do you know why this could be?
6 | Dec 21 2012 | 16:35:08 | 305009 | 10.255.12.109 | 10.255.12.109 | Built dynamic translation from inside:10.255.12.109 to SHARED:10.255.12.109 | ||
6 | Dec 21 2012 | 16:35:17 | 302013 | 10.255.1.30 | 63073 | 10.255.12.109 | 22 | Built inbound TCP connection 144979159621177275 for SHARED:10.255.1.30/63073 (10.255.1.30/63073) to inside:10.255.12.109/22 (10.255.12.109/22) |
Thank you in advance for any help!
Solved! Go to Solution.
12-22-2012 06:35 AM
Hi,
There is a post on the first page of this forum section that has abit similiar logoutput when trying connections.
But it almost seems to me that when you attempt the connection at first the initial connection attempt to the server has gone through (by some other route than the firewall) but the reply (SYN ACK)) to that TCP connection forming (SYN) is coming through the FWSM Context which in turn gets blocked as it hasnt seen the SYN.
I find the NAT log line abit wierd too.
Could you share configurations related to one context?
Not sure if I can be of help but could always look through the configurations if there is something there.
- Jouni
12-22-2012 06:35 AM
Hi,
There is a post on the first page of this forum section that has abit similiar logoutput when trying connections.
But it almost seems to me that when you attempt the connection at first the initial connection attempt to the server has gone through (by some other route than the firewall) but the reply (SYN ACK)) to that TCP connection forming (SYN) is coming through the FWSM Context which in turn gets blocked as it hasnt seen the SYN.
I find the NAT log line abit wierd too.
Could you share configurations related to one context?
Not sure if I can be of help but could always look through the configurations if there is something there.
- Jouni
12-22-2012 08:03 AM
Hello Jouni, thanks for the response!
I've pasted the configs of both FWSM's below. I've omitted most of the uneeded config to try to simplify it for you. The packets should not be getting filtered by the FWSM, as you wil see in the configs the "shared" traffic routes to 10.255.255.2 which is the 6500 not the FWSM. Again, just to let you know we only have this issue with Linux boxes, but we have this same issue with three different Linux Machines, with different versions of OS.
NYSPAL03FW02-PRIMARY/FWSM1# show run
: Saved
:
FWSM Version 4.1(5)
!
names
dns-guard
!
interface Vlan105
nameif SHARED
security-level 90
ip address 10.255.255.12 255.255.255.0
!
interface Vlan12
nameif inside
security-level 100
ip address 10.255.12.1 255.255.255.0 standby 10.255.12.2
!
interface Vlan212
nameif outside
security-level 0
ip address 192.168.96.49 255.255.255.248 standby 192.168.96.50
!
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
access-list INSIDE_IN extended permit ip any any
access-list INSIDE_IN extended permit icmp any any
access-list SHARED_IN extended permit ip any any
pager lines 24
logging enable
logging asdm informational
mtu SHARED 1500
mtu inside 1500
mtu outside 1500
icmp permit any SHARED
icmp permit any inside
icmp permit any outside
no asdm history enable
arp timeout 14400
global (outside) 1 x.x.x.x
nat (inside) 1 10.255.12.0 255.255.255.0
access-group SHARED_IN in interface SHARED
access-group INSIDE_IN in interface inside
access-group OUTSIDE_IN in interface outside
route SHARED 10.255.1.0 255.255.255.0 10.255.255.2 1
route outside 0.0.0.0 0.0.0.0 192.168.96.51 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 1:00:00 h225 1:00:00 mgcp 0:05:00
timeout mgcp-pat 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout sip-invite 0:03:00 sip-disconnect 0:02:00
timeout pptp-gre 0:02:00
timeout uauth 0:05:00 absolute
telnet timeout 5
ssh timeout 5
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map global_policy
class inspection_default
inspect dns maximum-length 512
inspect ftp
inspect h323 h225
inspect h323 ras
inspect sunrpc
inspect rsh
inspect sqlnet
inspect skinny
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
!
service-policy global_policy global
: end
NYSPAL03FW02-PRIMARY/FWSM2# show run
: Saved
:
FWSM Version 4.1(5)
!
names
dns-guard
!
interface Vlan10
nameif inside
security-level 100
ip address 10.255.1.1 255.255.255.0 standby 10.255.1.2
!
interface Vlan105
nameif SHARED
security-level 90
ip address 10.255.255.1 255.255.255.0
!
interface Vlan150
nameif EDGE
security-level 100
ip address 10.255.150.1 255.255.255.0 standby 10.255.150.2
!
interface Vlan210
nameif outside
security-level 0
ip address 192.168.96.1 255.255.255.248 standby 192.168.96.2
!
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
access-list EDGE_IN extended permit ip any any
access-list EDGE_IN extended permit icmp any any
access-list SHARED_IN extended permit ip any any
pager lines 24
logging enable
logging monitor debugging
logging buffered debugging
logging asdm debugging
mtu inside 1500
mtu SHARED 1500
mtu EDGE 1500
mtu outside 1500
icmp permit any inside
icmp permit any SHARED
icmp permit any EDGE
icmp permit any outside
no asdm history enable
arp timeout 14400
global (outside) 1 x.x.x.x
nat (inside) 1 10.255.1.0 255.255.255.0
access-group INSIDE_IN in interface inside
access-group SHARED_IN in interface SHARED
access-group EDGE_IN in interface EDGE
access-group OUTSIDE_IN in interface outside
route inside 10.222.139.0 255.255.255.0 10.255.1.69 1
route SHARED 10.255.9.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.11.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.12.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.16.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.20.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.24.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.28.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.17.0 255.255.255.224 10.255.255.2 1
route SHARED 10.255.18.0 255.255.255.0 10.255.255.2 1
route SHARED 10.255.19.0 255.255.255.0 10.255.255.2 1
route outside 0.0.0.0 0.0.0.0 192.168.96.3 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 1:00:00 h225 1:00:00 mgcp 0:05:00
timeout mgcp-pat 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout sip-invite 0:03:00 sip-disconnect 0:02:00
timeout pptp-gre 0:02:00
timeout uauth 0:05:00 absolute
telnet timeout 5
ssh timeout 5
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map global_policy
class inspection_default
inspect dns maximum-length 512
inspect ftp
inspect h323 h225
inspect h323 ras
inspect sunrpc
inspect rsh
inspect sqlnet
inspect skinny
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect icmp
!
service-policy global_policy global
: end
NYSPAL03FW02-PRIMARY/MGMT#
12-22-2012 09:12 AM
Hi,
Seems theres not really much complicating the configuration that might explain the problem.
Don't really have a chance to lab this setup myself at the moment but I know we have had some problems with shared Vlan use on FWSM also but nothing that would really help me in this situation.
One thing I noticed that you are not doing any NAT other than for LAN -> WAN traffic, unless you have removed some configurations. I think the "nat-control" is at its default which would mean "no nat-control"
What also confuses me is that the log messages talk about dynamic translations when you have generated traffic. I wonder if making a Static NAT statement affec the situation at all. (Even though NAT configurations shouldnt be needed if you specifically want to NAT some address)
In the case of the first Context (that you posted) I guess the typical NAT configuration that we would do would be
static (inside,SHARED) 10.255.12.0 10.255.12.0 netmask 255.255.255.0
Which in case of "nat-control" would enable "inside" to communicate with "SHARED" interface hosts with their original IP address (Other option would be to configure NAT0/NAT Exempt)
Also one thing you havent configured, unless its taken from the configuration, is "sysopt noproxyarp SHARED". This command would be of any use in your setup. We usually configure this to customer context LAN interfaces which have direct connection to customer L2 network segment so the FWSM doesnt reply to any ARP request instead of a actual host.
Also one thing I noticed which probably isnt any issue but is different between the contexts is that the first one doesnt have the "inspect icmp" configuration. Though since you have opened traffic between the interfaces ICMP should work normally.
Sadly I can't give you an answer as to what is causing this.
Personally I would just go through the whole path of the traffic (even though you have stated that this only happens with these Linux devices) and possibly play around with the NAT configurations. Ofcourse I don't know how much chances you have to touch the configurations in these setups if they are part of a critical production enviroment.
Heres some quotes from Cisco FWSM documentation related to the "nat-control" and "sysopt noproxyarp" configuration commands
sysopt noproxyarp
To disable proxy ARP for NAT global addresses on an interface, use the sysopt noproxyarp command in global configuration mode. To reenable proxy ARP for global addresses, use the no form of this command.
sysopt noproxyarp interface_name
no sysopt noproxyarp interface_name
Syntax Description
Defaults
Proxy ARP for global addresses is enabled by default.
Command Modes
The following table shows the modes in which you can enter the command:
Command Mode Firewall Mode Security Context Routed Transparent Single Multiple Context SystemGlobal configuration
•
•
•
•
—
Command History
Usage Guidelines
In rare circumstances, you might want to disable proxy ARP for global addresses.
When a host sends IP traffic to another device on the same Ethernet network, the host needs to know the MAC address of the device. ARP is a Layer 2 protocol that resolves an IP address to a MAC address. A host sends an ARP request asking "Who is this IP address?" The device owning the IP address replies, "I own that IP address; here is my MAC address."
Proxy ARP is when a device responds to an ARP request with its own MAC address, even though the device does not own the IP address. The FWSM uses proxy ARP when you configure NAT and specify a global address that is on the same network as the FWSM interface. The only way traffic can reach the hosts is if the FWSM uses proxy ARP to claim that the FWSM MAC address is assigned to destination global addresses.
Examples
The following example disables proxy ARP on the inside interface:
hostname(config)# sysopt noproxyarp inside
nat-control
To enforce NAT control use the nat-control command in global configuration mode. NAT control requires NAT for inside hosts when they access the outside. To disable NAT control, use the no form of this command.
nat-control
no nat-control
Syntax Description
This command has no arguments or keywords.
Defaults
NAT control is disabled by default (no nat-control command). If you upgraded from an earlier version of software, however, NAT control might be enabled on your system because it was the default in some earlier versions.
Command Modes
The following table shows the modes in which you can enter the command:
Command Mode Firewall Mode Security Context Routed Transparent Single Multiple Context SystemGlobal configuration
•
•
•
•
—
Command History
Release Modification3.1(1)
This command was introduced.
3.2.(1)
NAT is now supported in transparent firewall mode.
Usage Guidelines
NAT control requires that packets traversing from an inside interface to an outside interface match a NAT rule; for any host on the inside network to access a host on the outside network, you must configure NAT to translate the inside host address.
Interfaces at the same security level are not required to use NAT to communicate.
By default, NAT control is disabled, so you do not need to perform NAT on any networks unless you choose to perform NAT.
If you want the added security of NAT control but do not want to translate inside addresses in some cases, you can apply a NAT exemption (nat 0 access-list) or identity NAT (nat 0 or static) rule on those addresses.
Note In multiple context mode, the packet classifier relies on the NAT configuration in some cases to assign packets to contexts. If you do not perform NAT because NAT control is disabled, then the classifier might require changes in your network configuration.
Examples
The following example enables NAT control:
hostname(config)# nat-control
Whole command reference can be found at
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/command/reference/fwsm_ref.html
- Jouni
12-22-2012 09:14 AM
By the way,
Do you have some Active/Standby FWSM setup between 2 FWSMs?
Also why doesnt the "SHARED" interface have a secondary IP address configured but the others do?
- Jouni
12-28-2012 07:32 AM
Hello Jouni, thanks for the reply! I tried all that you said, however those did not resolve the issue. The nat-control and noproxyarp both were unsuccesful. As for the shared FWSM, currently we don't run failover, but when we do I will add the standby interfaces to the SHARED interface as well for all FWSMs. The setup you see with standby is configured in advance.
Do you have any idea maybe what else I can try with regards to resolving this strange issue with Linux boxes and our FWSM.
01-10-2013 07:45 AM
Hey Jouni,
Just for your reference configuring tcp-state-bypass between the two private networks on both sides of the FWSM resolved this issue. So, it seems the firewall was doing a bit too much protecting when really these were two trusted networks.
01-28-2013 10:57 AM
Ok, so after further testing we confirmed that tcp-state-bypass fixed the issue with Linux servers communicating over shared contexts, however we still have an issue with Linux machines over a VPN. Currently, from time to time hosts cannot communicate over the VPN to our Linux machines in our DC. The Linux machine uses the FWSM as the gateway, and the FWSM forwards over to the ASA. We checked the logs and when we initiate the connection from the remote site to the Linux servers, our ASA sees the traffic come over the VPN, however the FWSM never sees it. This only happens with Linux machines, all other hosts work fine over the VPN. Could this be an issue with the FWSM looking for something in the packet or doing too much inspecting.
During this outage I've also confirmed that the inside interface of the ASA loses connection to the Linux server. Example I can ping a windows host on the same network as the Linux server without an issue, however I cannot ping the Linux Server, even they are on the same network. So this looks like an FWSM issue, I can provide logs, network diagrams, configs, anything that may help. Has anyone seen this type of issue before with only Linux machines and Cisco Firewalls? Thanks so much in advance for all the help.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide