ACE and NAT on 6500

AlexandrKry · ‎07-21-2011

Hello everyone.

First of all, I want you to know that I do understand that it's not a good design at all, but I still need to implement it.

My ACE20 in 6500 works in routed mode with VLAN 101 (10.0.101.0/24) as the client-side VLAN. I therefore have Vlan101 SVI (10.0.101.1/24) on 6500.

I also have VLAN 200 (10.0.200.0/24) on 6500 with SVI Vlan200 (10.0.200.1/24) and I want to create a static bidirectional NAT so that clients, connecting to IP address in VLAN 200 (10.0.200.64) would be NATed to VIP (10.0.101.10).

Let's say we have clients (10.0.500.0/24) connecting through gig1/0/1.Here is my current setup on 6500:

interface gig1/0/1

ip nat outside

interface Vlan101

ip nat inside

ip nat inside source static 10.0.101.10 10.0.200.64

This configuration works as expected: clients are able to access the VIP through 10.0.200.64.

But I need clients also to be able to access the VIP directly (10.0.500.0/24 <> 10.0.101.10).

It seems that I'm missing something simple and it may not even relate to ACE, but still.

Can someone point me into the right direction?

AlexandrKry · ‎08-02-2011

I still haven't found a solution.. Anyone?

Ahmad Basel Jaber · ‎08-02-2011

Hi Alexandr,

Let me understand your setup completety first

- You have mulitple clients coming through G1/0/1 for example 10.0.500.0/24 and 10.0.200.0/24, right?

- And you need the clients 10.0.200.0/24 to access the VIP by hitting 10.0.200.64 IP address and the 6K need to translated this IP to 10.0.101.10, then forward the request to the ACE, right?

- 10.0.101.10 is VIP address configured on the client VLAN on the ACE, right?

- You need the clients 10.0.500.0/24 to access the VIP directly (hitting 10.0.101.10 and not 10.0.200.64) through G1/0/1, right?

It would be great if you can attach the show tech from both devices (6K and ACE), if you can not just provide the full interfaces configuration on both devices, full NAT configuration, the output of show ip route from 6K and the output of show ip nat translations and show version from 6K.

Best regards,

Ahmad

AlexandrKry · ‎08-02-2011

Hi Ahmad,

Yes, I have VIP 10.0.101.10 configured on ACE in VLAN 101 (client-side VLAN).

Clients that connect through gig1/0/1 are coming from distribution layer (ex. 10.0.500.0/24), the VLAN 200 (10.0.200.0/24) is a server's vlan and it is available on the core layer (6k) only. There are a couple of switchport access interfaces on 6k that are used to connect servers on a farm and that's the whole point of this design.

We're moving an old service that is currently accessible via 10.0.200.64 IP address to ACE (with VIP 10.0.101.10) and we want to do this transparently for the clients. That is considered as a termporary solution and we want eventually to transit all the clients to the new ip address (10.0.101.10), but this will take some time.

That's why we want the service to be available via both old (10.0.200.64) and new (10.0.101.10) addresses.

How it behaves now:

If I remove ip nat inside source static 10.0.101.10 10.0.200.64 line from the 6k config, the client (10.0.500.0/24) is able to access the service via VIP 10.0.101.10 (but not via 10.0.200.64)

If I add this nat rule, it's vice-versa.

Here is the configuration snippets:

on ACE:

interfaces:

interface vlan 101

description ** Client side VLAN **

ip address 10.0.101.2 255.255.255.0

access-group input everyone

service-policy input Srv_PM

no shutdown

interface vlan 102

description ** Server side VLAN **

ip address 10.0.102.12 255.255.255.0

access-group input everyone

no shutdown

ip route 0.0.0.0 0.0.0.0 10.10.101.1

on 6k:

interfaces:

! gig 1/0/1 connects distib layer to the core: 10.0.0.1<>10.0.0.2

! 10.0.0.1 is a default route on the Dist L3 switch

interface gig1/0/1

no switchport

ip address 10.0.0.1 255.255.255.252

ip nat outside

interface Vlan101

ip address 10.0.101.1 255.255.255.0

ip nat inside

interface Vlan200

ip address 10.0.200.1 255.255.255.0

ip nat inside source static 10.0.101.10 10.0.200.64

routing:

10.0.101.0/24 is directly connected

10.0.200.0/24 is directly connected

10.0.500.0/24 is accessible via 10.0.0.2 (Dist L3 switch)

nat:

1) when client from 10.0.500.0/24 tries to reach service via NAT'ed ip 10.0.200.64:

#sh ip nat translations

Pro Inside global Inside local Outside local Outside global

tcp 10.0.200.64:22 10.0.101.10:22 10.0.500.240:49505 10.0.500.240:49505

--- 10.0.200.64 10.0.101.10 --- ---

2) when client from 10.0.500.0/24 tries to reach service dierctly via VIP 10.0.101.10:

#sh ip nat translations

Pro Inside global Inside local Outside local Outside global

tcp 10.0.200.64:22 10.0.101.10:22 10.0.500.240:39168 10.0.500.240:39168

--- 10.0.200.64 10.0.101.10 --- ---

I think the answer is hidden here. There shouldn't be NAT translation performed in the 2nd example, right?

What do I need to do to fix that?

Ahmad Basel Jaber · ‎08-02-2011

Hi Alexandr,

You are right, there should not be NAT translation for the second connection, but for some reason the 6K behave strangly, I am not a 6K expert but it looks strange for me also.

I would suggest you to add the following command, and see how it is go with the second connection:

ip nat inside source static 10.0.101.10 10.0.101.10

Then "clear ip nat tran *"

Then test both connections and let me know the result, if it did not work, please get fresh copy of show ip nat tran from the 6K.

Best regards,

Ahmad

AlexandrKry · ‎08-02-2011

It doesn't allow me to do that:

(config)# ip nat inside source static 10.0.101.10 10.0.101.10

% 10.0.101.10 already mapped (10.0.101.10 -> 10.0.200.64)

Ahmad Basel Jaber · ‎08-02-2011

Please run the following quick test:

- keep the original NAT command on the 6K "ip nat inside source static 10.0.101.10 10.0.200.64"

- configure capture rule on the ACE to capture all traffic coming from your client going to any (both ways).

- start the capture, try to connect to 10.0.101.10, stop the capture then provide me with its contains.

Note: let me know if you have no experiance with capture tool on ACE.

Best regards,

Ahmad

AlexandrKry · ‎08-02-2011

10.0.500.240 is the client who tries to access 10.0.101.10 (VIP) via ssh (tcp 22)

10.0.102.130 and 10.0.102.131 are the real servers

ACE_1/C# capture cap2 all access-list everyone

ACE_1/C# capture cap2 start

ACE_1/C# 14:25:43.652620 ec:30:91:e3:9:80 0:b:fc:fe:1b:2 0800 74: 10.0.500.240.59984 > 10.0.101.10.22: S [bad tcp cksum ec29!] 3966551577:3966551577(0) win 5840 (DF) (ttl 62, id 36377, len 60)

14:25:43.653004 0:b:fc:fe:1b:2 3c:4a:92:ec:7d:38 0800 74: 10.0.500.240.59984 > 10.0.102.130.22: S [bad tcp cksum 2f8a!] 3911427958:3911427958(0) win 5840 (DF) (ttl 62, id 36377, len 60, bad cksum 47ce!)

14:25:43.653361 3c:4a:92:ec:7d:38 f8:66:f2:9a:6:81 0800 60: 10.0.102.130.22 > 10.0.500.240.59984: S [tcp sum ok] 2431380500:2431380500(0) ack 3911427959 win 5840 (DF) (ttl 64, id 0, len 44)

14:25:43.653709 0:b:fc:fe:1b:2 ec:30:91:e3:9:80 0800 60: 10.0.101.10.22 > 10.0.500.240.59984: S [bad tcp cksum bc9f!] 2486504119:2486504119(0) ack 3966551578 win 5840 (DF) (ttl 64, id 0, len 44, bad cksum d37f!)

14:25:43.655326 ec:30:91:e3:9:80 0:b:fc:fe:1b:2 0800 86: 10.0.500.240 > 10.0.102.130: icmp: host 10.0.500.240 unreachable - admin prohibited (ttl 62, id 13409, len 72, bad cksum e17f!)

14:25:43.655499 0:b:fc:fe:1b:2 3c:4a:92:ec:7d:38 0800 86: 10.0.500.240 > 10.0.102.130: icmp: host 10.0.500.240 unreachable - admin prohibited (ttl 62, id 13409, len 72, bad cksum e17f!)

14:25:46.657366 ec:30:91:e3:9:80 0:b:fc:fe:1b:2 0800 74: 10.0.500.240.59984 > 10.0.101.10.22: S [tcp sum ok] 3966551577:3966551577(0) win 5840 (DF) (ttl 62, id 36378, len 60)

14:25:46.657567 0:b:fc:fe:1b:2 3c:4a:92:ec:7d:38 0800 74: 10.0.500.240.59984 > 10.0.102.130.22: S [bad tcp cksum 3b4a!] 3911427958:3911427958(0) win 5840 (DF) (ttl 62, id 36378, len 60, bad cksum 47cd!)

14:25:46.657726 3c:4a:92:ec:7d:38 f8:66:f2:9a:6:81 0800 60: 10.0.102.130.22 > 10.0.500.240.59984: S [tcp sum ok] 2434385670:2434385670(0) ack 3911427959 win 5840 (DF) (ttl 64, id 0, len 44)

14:25:46.657874 0:b:fc:fe:1b:2 ec:30:91:e3:9:80 0800 60: 10.0.101.10.22 > 10.0.500.240.59984: S [bad tcp cksum bc9f!] 2489509289:2489509289(0) ack 3966551578 win 5840 (DF) (ttl 64, id 0, len 44, bad cksum d37f!)

14:25:46.658853 ec:30:91:e3:9:80 0:b:fc:fe:1b:2 0800 86: 10.0.500.240 > 10.0.102.130: icmp: host 10.0.500.240 unreachable - admin prohibited (ttl 62, id 13410, len 72, bad cksum e17e!)

14:25:46.659031 0:b:fc:fe:1b:2 3c:4a:92:ec:7d:38 0800 86: 10.0.500.240 > 10.0.102.130: icmp: host 10.0.500.240 unreachable - admin prohibited (ttl 62, id 13410, len 72, bad cksum e17e!)

14:25:51.278800 0:b:fc:fe:1b:2 3c:4a:92:ec:7d:38 0800 54: 10.0.500.240.59984 > 10.0.102.130.22: R [bad tcp cksum b6c1!] 3911427959:3911427959(0) win 5840 (ttl 255, id 19148, len 40, bad cksum 0!)

14:25:51.278969 0:b:fc:fe:1b:2 ec:30:91:e3:9:80 0800 54: 10.0.101.10.22 > 10.0.500.240.59984: R [bad tcp cksum 1b30!] 1:1(0) ack 1 win 5840 (ttl 255, id 19149, len 40, bad cksum 0!)

14:25:52.673892 ec:30:91:e3:9:80 0:b:fc:fe:1b:2 0800 74: 10.0.500.240.59984 > 10.0.101.10.22: S [bad tcp cksum 471c!] 3966551577:3966551577(0) win 5840 (DF) (ttl 62, id 36379, len 60)

14:25:52.674064 0:b:fc:fe:1b:2 3c:4a:92:ec:5d:20 0800 74: 10.0.500.240.59984 > 10.0.102.131.22: S [bad tcp cksum 1b16!] 4103197627:4103197627(0) win 5840 (DF) (ttl 62, id 36379, len 60, bad cksum 47cc!)

14:25:52.674219 3c:4a:92:ec:5d:20 f8:66:f2:9a:6:81 0800 60: 10.0.102.131.22 > 10.0.500.240.59984: S [tcp sum ok] 2060901920:2060901920(0) ack 4103197628 win 5840 (DF) (ttl 64, id 0, len 44)

14:25:52.674362 0:b:fc:fe:1b:2 ec:30:91:e3:9:80 0800 60: 10.0.101.10.22 > 10.0.500.240.59984: S [bad tcp cksum 2c07!] 1924255870:1924255870(0) ack 3966551578 win 5840 (DF) (ttl 64, id 0, len 44, bad cksum d37e!)

14:25:52.675067 ec:30:91:e3:9:80 0:b:fc:fe:1b:2 0800 86: 10.0.500.240 > 10.0.102.131: icmp: host 10.0.500.240 unreachable - admin prohibited (ttl 62, id 13412, len 72, bad cksum e17c!)

14:25:52.675235 0:b:fc:fe:1b:2 3c:4a:92:ec:5d:20 0800 86: 10.0.500.240 > 10.0.102.131: icmp: host 10.0.500.240 unreachable - admin prohibited (ttl 62, id 13412, len 72, bad cksum e17c!)

ACE_1/C# capture cap2 stop

Ahmad Basel Jaber · ‎08-02-2011

Hi Alexandra,

I guess i figured out what is the issue, the 6K is behaving correctly in term of NAT

The issue is mainly caused by bidirectional NAT you have, the request will come from the client and will pass through the 6K without been translated, but when the reply comes back soured from 10.0.101.10 trying to reach the client it will hit the static NAT you have in the 6K and will be translated to 10.0.200.64, and that explains the first line you are getting in show ip nat tras:

#sh ip nat translations

Pro Inside global Inside local Outside local Outside global

tcp 10.0.200.64:22 10.0.101.10:22 10.0.500.240:39168 10.0.500.240:39168

--- 10.0.200.64 10.0.101.10 --- ---

Now the client will RST this connection immediately, and will keep trying to send SYN packets on the old connection, but it will timeout, does that make sense?

Note: I see unbelievable bug in your whole system, how the client is using 10.0.500.240??? This is not a valid IP address, how you are using 500 as the third octet??!! what is your client machine? what it is default gateway?

Please fix this and collect fresh of these captuers.

Best regards,

Ahmad

AlexandrKry · ‎08-02-2011

Dear Ahmad,

I'm sorry for the confusing address. 10.0.500.240 is just a substitution, the real address used is okay. Let's say it is 10.0.205.240.

Hmm, I may have got confused with NAT configuration in the first place. Here is the route path:

Client (10.0.205.240, VLAN 500) --> Distrib (L3 switch, iface vlan500, 10.0.205.1) ---> Core (6k, gig1/0/1, 10.0.0.1) --(NAT)--> ACE.

The client has the Distrib L3 switch as the default gateway. The Distrib L3 switch routes networks 10.0.101.0/24 and 10.0.200.0/24 through 10.0.0.1 (gig1/0/1 on 6k which it is attached to).

The interface gig1/0/1 (which has 10.0.0.1 ip address configured on it) is configured as nat outside, while the ACE Client-side VLAN interface vlan201 configured as nat inside. Does this configuration seems correct to you?

Why does it makes the translation of the packets on their way back from ACE but doesn't do the incoming ones?

Also, I just found out that the ICMP works fine. Here is the sh ip nat trans of ICMP connections:

3) when client from 10.0.205.0/24 sends ICMP echo to VIP 10.0.101.10:

#sh ip nat trans

Pro Inside global Inside local Outside local Outside global

icmp 10.0.200.64:28062 10.0.101.10:28062 10.0.205.240:28062 10.0.205.240:28062

icmp 10.0.200.64:28063 10.0.101.10:28063 10.0.205.240:28063 10.0.205.240:28063

--- 10.109.160.64 10.109.171.10 --- ---

Ahmad Basel Jaber · ‎08-02-2011

Hi Alex,

So you were removing the real IP address and adding this bad address?! Please explain this before doing that next time otherwise we will experience huge misunderstanding

Have you done this also on the capture you have provided on the ACE?

Back to your email

The interface gig1/0/1(which has 10.0.0.1 ip address configured on it) is configured as nat outside, while the ACE Client-side VLAN interface vlan201 configured as nat inside. Does this configuration seems correct to you?

-- Yes, it looks correct, but you need to keep in mind that you are using source/static/bidirectional NAT and the 6K is not stateful device which mean it works on per packet bases, that's why the initial request passed thorough the 6K without been NATted but the reply were NATted. Also note that all these points are assumption we dont have solid base to stand on and confirm the root cause, we could figure out the issue is completly different, but really that bad IP shifted my thinking completly

Its really interesting to have the ICMP working, let do another test now:

- install wireshark on your client machine and start capturing.

- enable the capture on the ACE.

- run the test again.

- collect the captures and attach them in .pcap format from both sides (ACE and the clients).

After checking these captuers I should be able to advice the next step and the root cause.

Best regards,

Ahmad

AlexandrKry · ‎08-02-2011

Dear Ahmad,

Please see my comment below.

It seems that using route map does the trick. It tells 6k to NAT only the traffic destined to that IP address I want to use for VIP NATing and do not involve any translation when users access VIP directly.

Side note:

I'm currently preparing for 640-802 composite CCNA (my first Cisco certification) and NAT is still a mystery for me in some cases. ICMP NATing is exactly one of those cases.

Once I needed to allow a specific host (1.1.1.1) to be able to send ICMP echo to the outside world, but nothing else.

We have a nat overload on our WAN interface with ACL as a source list.

I was quite confused when it turned out that putting the "permit icmp host 1.1.1.1 any" didn't work and that one is still on my long list to figure out

AlexandrKry · ‎08-02-2011

Well, it seems that I have figured out the solution myself, at least it seems to be working

1) create ACL to select traffic destined ONLY to the NATed address:

ip access-list extended ACE_Legacy_test

permit ip any host 10.0.200.64

2) create route map that permits traffic selected by this ACL:

route-map ACE_Legacy permit 10

match ip address ACE_Legacy_test

3) apply that route map to the NAT rule:

ip nat inside source static 10.0.101.10 10.0.200.64 route-map ACE_Legacy

This way 6k will not NAT traffic destined directly to 10.0.101.10, but only those destined to 10.0.200.64.

Thanks Ahmad for you kind help, I couldn't have figured it out without you.

Ahmad Basel Jaber · ‎08-02-2011

Great news

Have you tested both type of connections? Are they both working fine?

High 5 for your solution

Note: please confirm these two questions so we have useful discussion.

Best regards,

Ahmad

AlexandrKry · ‎08-02-2011

Yes, both types of connections work fine.

The thread can be closed