topic Re: CSS Sourcegroup Issue in Application Networking

CSS Sourcegroup Issue

Marvin Rhoads — Fri, 22 Apr 2011 00:43:45 GMT

I am seeing traffic leave from behind my CSS with its native address despite there being an ACL that directs it to use a servicegroup reverse NAT.

The distant end is seeing hits on their incoming ACL such as:

Apr 20 07:36:08 PDT: %SEC-6-IPACCESSLOGP: list INBOUND_RISE_VZ denied tcp 192.168.10.15(0) -> 66.150.171.154(0), 4 packet

Here are the relevant lines from my CSS script which has an ACL (clause 13) applied to VLAN 10 to prevent this from occurring. (By the way, HTTP traffic from the sources in question appears to be properly translated.) The problem hosts use the CSS as their default gateway.

group rise-outbound-nat
vip address 63.251.81.206
active

acl 10
clause 1 permit icmp any destination any
clause 2 permit any 192.168.10.0 255.255.255.0 destination 66.174.77.12 255.255.255.255 sourcegroup mod-aaa
clause 3 permit any 192.168.10.0 255.255.255.0 destination 66.174.91.12 255.255.255.255 sourcegroup mod-aaa
clause 4 permit any 192.168.10.0 255.255.255.0 destination 63.251.81.176 255.255.255.240 sourcegroup mod-aaa
clause 5 permit any 192.168.10.0 255.255.255.0 destination 162.115.180.0 255.255.255.0 sourcegroup mod-aaa
clause 6 permit any 192.168.10.0 255.255.255.0 destination 162.115.245.0 255.255.255.0 sourcegroup mod-aaa
clause 7 permit any 192.168.10.0 255.255.255.0 destination 162.115.116.223 255.255.255.255 sourcegroup mod-aaa
clause 8 permit any 192.168.10.0 255.255.255.0 destination 69.78.64.170 255.255.255.255 sourcegroup sp-outbound-nat
clause 9 permit any 192.168.10.0 255.255.255.0 destination 69.78.31.234 255.255.255.255 sourcegroup sp-outbound-nat
clause 10 permit any 192.168.10.0 255.255.255.0 destination 207.188.21.166 255.255.255.255 sourcegroup rise-outbound-nat
clause 11 permit any 192.168.10.0 255.255.255.0 destination 63.251.81.206 255.255.255.255 sourcegroup mod-aaa
clause 12 permit any 192.168.10.0 255.255.255.0 destination 63.251.81.145 255.255.255.255 sourcegroup mod-aaa
clause 13 permit any 192.168.10.0 255.255.255.0 destination 66.150.171.154 255.255.255.255 sourcegroup rise-outbound-nat
clause 100 bypass any 192.168.10.0 255.255.255.0 destination any
apply circuit-(VLAN10)

Re: CSS Sourcegroup Issue

yushimaz — Tue, 26 Apr 2011 01:57:33 GMT

Can you get capture trace?

To identify the reason, we have to check the trace.

If syn retransmission occurs, your problem may hit the following bug.

CSCek36511

CSS does not NAT for retransmitting SYN packets.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCek36511

And, similar issue may occur when client sends data packets during

half-closed connection or idle timout flow.

Regards,

Yuji

Re: CSS Sourcegroup Issue

Marvin Rhoads — Tue, 26 Apr 2011 04:33:56 GMT

Yuji,

I don't think I'm hittting the bug you mentioned since I am running:

Version: sg0820402 (08.20.4.02)

...and the bug should be fixed in that version.

I did a trace (flow trace-ip 66.150.171.154 combined with flow options 0x5) and got the output attached.

Note it does generally show traffic coming in from the real server IPs (192.168.10.14-17) and being sent out NATted with the (proper) NAT address (63.251.81.206). I am asking my peer to check if they saw access log hits during this period.

Re: CSS Sourcegroup Issue

pablo.nxh — Tue, 26 Apr 2011 19:02:40 GMT

Hi,

I saw this problem many times with multiple clients, the flow works fine for some connections and suddenly it breaks and the CSS starts passing

the real server IP to the endpoint/client. The problem is related to the flow being garbage collected, I don't want to get to deep into garbage collection process but just to give you an idea the flow is removed from the conns table and recycled so when your server tries to send data the CSS doesn't recognize it as established flow and simply routes the packet.

Try adding a flow-timeout multiplier of say 30 under the group that should help.

group rise-outbound-nat
vip address 63.251.81.206

flow-timeout multiplier 30
active

HTH

__ __

Pablo

Re: CSS Sourcegroup Issue

yushimaz — Wed, 27 Apr 2011 02:12:05 GMT

I checked attachment logs. Since there is no retransmission and the CSS runs 8.20(402), it doesn't hit CSCek36511.

So, I suspect the latter (client sends data packets during half-closed connection or idle timout flow).

When CSS receives fin or idle timer is expired, CSS removed the entry from active-list(show flows).

Actually, CSS doesn't remove this entry and sotres into internal cache (spoof-list). This means

CSS can handle final ack and expired flow even though the entry is not listed in show flows.

However, the internal cache is not guaranteed. So, the entry may be deleted/overwritten.

If the entry is deleted, CSS cannot do NAT/SNAT and then becomes routed.

To avoid this symptom, existing connection should not be timed out by idle timeout.

So, I think Pablo's suggestion is reasonable.

Regards,

Yuji

Re: CSS Sourcegroup Issue

Marvin Rhoads — Wed, 27 Apr 2011 18:38:10 GMT

Thanks Pablo and Yuji,

I have to schedule a production change window to make this recommended change. I will advise as to the success after I do so.

- Marvin

Re: CSS Sourcegroup Issue

Marvin Rhoads — Thu, 26 May 2011 17:22:26 GMT

Hello colleagues,

I finally got a change window and implemented Pablo's suggestion. All appears to be working well now.

Thanks again for the assist!