04-21-2011 05:43 PM
I am seeing traffic leave from behind my CSS with its native address despite there being an ACL that directs it to use a servicegroup reverse NAT.
The distant end is seeing hits on their incoming ACL such as:
Apr 20 07:36:08 PDT: %SEC-6-IPACCESSLOGP: list INBOUND_RISE_VZ denied tcp 192.168.10.15(0) -> 66.150.171.154(0), 4 packet
Here are the relevant lines from my CSS script which has an ACL (clause 13) applied to VLAN 10 to prevent this from occurring. (By the way, HTTP traffic from the sources in question appears to be properly translated.) The problem hosts use the CSS as their default gateway.
group rise-outbound-nat
vip address 63.251.81.206
active
acl 10
clause 1 permit icmp any destination any
clause 2 permit any 192.168.10.0 255.255.255.0 destination 66.174.77.12 255.255.255.255 sourcegroup mod-aaa
clause 3 permit any 192.168.10.0 255.255.255.0 destination 66.174.91.12 255.255.255.255 sourcegroup mod-aaa
clause 4 permit any 192.168.10.0 255.255.255.0 destination 63.251.81.176 255.255.255.240 sourcegroup mod-aaa
clause 5 permit any 192.168.10.0 255.255.255.0 destination 162.115.180.0 255.255.255.0 sourcegroup mod-aaa
clause 6 permit any 192.168.10.0 255.255.255.0 destination 162.115.245.0 255.255.255.0 sourcegroup mod-aaa
clause 7 permit any 192.168.10.0 255.255.255.0 destination 162.115.116.223 255.255.255.255 sourcegroup mod-aaa
clause 8 permit any 192.168.10.0 255.255.255.0 destination 69.78.64.170 255.255.255.255 sourcegroup sp-outbound-nat
clause 9 permit any 192.168.10.0 255.255.255.0 destination 69.78.31.234 255.255.255.255 sourcegroup sp-outbound-nat
clause 10 permit any 192.168.10.0 255.255.255.0 destination 207.188.21.166 255.255.255.255 sourcegroup rise-outbound-nat
clause 11 permit any 192.168.10.0 255.255.255.0 destination 63.251.81.206 255.255.255.255 sourcegroup mod-aaa
clause 12 permit any 192.168.10.0 255.255.255.0 destination 63.251.81.145 255.255.255.255 sourcegroup mod-aaa
clause 13 permit any 192.168.10.0 255.255.255.0 destination 66.150.171.154 255.255.255.255 sourcegroup rise-outbound-nat
clause 100 bypass any 192.168.10.0 255.255.255.0 destination any
apply circuit-(VLAN10)
Solved! Go to Solution.
04-26-2011 12:02 PM
Hi,
I saw this problem many times with multiple clients, the flow works fine for some connections and suddenly it breaks and the CSS starts passing
the real server IP to the endpoint/client. The problem is related to the flow being garbage collected, I don't want to get to deep into garbage collection process but just to give you an idea the flow is removed from the conns table and recycled so when your server tries to send data the CSS doesn't recognize it as established flow and simply routes the packet.
Try adding a flow-timeout multiplier of say 30 under the group that should help.
group rise-outbound-nat
vip address 63.251.81.206
flow-timeout multiplier 30
active
HTH
__ __
Pablo
04-25-2011 06:57 PM
Can you get capture trace?
To identify the reason, we have to check the trace.
If syn retransmission occurs, your problem may hit the following bug.
CSCek36511
CSS does not NAT for retransmitting SYN packets.
04-25-2011 09:33 PM
Yuji,
I don't think I'm hittting the bug you mentioned since I am running:
Version: sg0820402 (08.20.4.02)
...and the bug should be fixed in that version.
I did a trace (flow trace-ip 66.150.171.154 combined with flow options 0x5) and got the output attached.
Note it does generally show traffic coming in from the real server IPs (192.168.10.14-17) and being sent out NATted with the (proper) NAT address (63.251.81.206). I am asking my peer to check if they saw access log hits during this period.
04-26-2011 12:02 PM
Hi,
I saw this problem many times with multiple clients, the flow works fine for some connections and suddenly it breaks and the CSS starts passing
the real server IP to the endpoint/client. The problem is related to the flow being garbage collected, I don't want to get to deep into garbage collection process but just to give you an idea the flow is removed from the conns table and recycled so when your server tries to send data the CSS doesn't recognize it as established flow and simply routes the packet.
Try adding a flow-timeout multiplier of say 30 under the group that should help.
group rise-outbound-nat
vip address 63.251.81.206
flow-timeout multiplier 30
active
HTH
__ __
Pablo
04-26-2011 07:12 PM
I checked attachment logs. Since there is no retransmission and the CSS runs 8.20(402), it doesn't hit CSCek36511.
So, I suspect the latter (client sends data packets during half-closed connection or idle timout flow).
When CSS receives fin or idle timer is expired, CSS removed the entry from active-list(show flows).
Actually, CSS doesn't remove this entry and sotres into internal cache (spoof-list). This means
CSS can handle final ack and expired flow even though the entry is not listed in show flows.
However, the internal cache is not guaranteed. So, the entry may be deleted/overwritten.
If the entry is deleted, CSS cannot do NAT/SNAT and then becomes routed.
To avoid this symptom, existing connection should not be timed out by idle timeout.
So, I think Pablo's suggestion is reasonable.
Regards,
Yuji
04-27-2011 11:38 AM
Thanks Pablo and Yuji,
I have to schedule a production change window to make this recommended change. I will advise as to the success after I do so.
- Marvin
05-26-2011 10:22 AM
Hello colleagues,
I finally got a change window and implemented Pablo's suggestion. All appears to be working well now.
Thanks again for the assist!
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide