Problem with ACE4710 in L2 mode (dosn't pass traffic between vlans)

[Support Team]

Hello

I have ACE4710 appliance from Cisco Demo Depot. We configured RDP loadbalance in bridge mode, but ACE dosn't pass traffic between vlans. ICMP req and rep. passed. I can ping VIP and some of servers in server farm, but can't establish RDP session to VIP or directly to servers.

witch/terminalservers# sh conn detail

total current connections : 2

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
1555607    1  in  TCP   1501 172.17.9.147:4450     172.17.7.10:3389      ESTAB
          [ idle time   : 00:00:05,   byte count  : 147        ]
          [ elapsed time: 00:00:05,   packet count: 3          ]
1555608    1  out TCP   501  172.17.7.11:3389      172.17.9.147:4450     INIT
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:05,   byte count  : 0          ]
          [ elapsed time: 00:00:05,   packet count: 0          ]

switch/terminalservers# sh run
Generating configuration....

logging enable
logging console 7
logging timestamp
logging trap 5
logging buffered 7
logging monitor 7

access-list bpdu-fixup ethertype permit bpdu

access-list ALL line 8 extended permit ip any any
access-list ALL line 16 extended permit icmp any any

probe icmp ICMPPROBE
  interval 15
  passdetect interval 60

rserver host CORTERM3
  ip address 172.17.7.11
  inservice
rserver host CORTERM6
  ip address 172.17.7.12
  inservice

serverfarm host CORTERM
  probe ICMPPROBE
  rserver CORTERM3
    inservice
  rserver CORTERM6
    inservice

sticky ip-netmask 255.255.255.255 address both SG1
  timeout 120
  serverfarm CORTERM

class-map type management match-any remote-mgmt
  201 match protocol snmp any
  202 match protocol telnet any
  203 match protocol ssh any
  204 match protocol icmp any
  205 match protocol http any
  206 match protocol https any
  207 match protocol xml-https any
class-map match-all slb-corterm-vip
  2 match virtual-address 172.17.7.10 tcp eq rdp

policy-map type management first-match remote-mgmt
  class remote-mgmt
    permit

policy-map type loadbalance rdp first-match slb-corterm-vip
  class class-default
    sticky-serverfarm SG1

policy-map multi-match client-vips
  class slb-corterm-vip
    loadbalance vip inservice
    loadbalance policy slb-corterm-vip
    loadbalance vip icmp-reply

interface vlan 501
  bridge-group 1
  access-group input bpdu-fixup
  access-group input ALL
  access-group output ALL
  no shutdown
interface vlan 1501
  bridge-group 1
  access-group input bpdu-fixup
  access-group input ALL
  access-group output ALL
  service-policy input remote-mgmt
  service-policy input client-vips
  no shutdown

interface bvi 1
  ip address 172.17.7.4 255.255.255.0
  no shutdown

ip route 0.0.0.0 0.0.0.0 172.17.7.1

[rvavale]

Hello,

From show conn output it looks like Server response is bypassing ACE (asymmetrical traffic). Try configuring
Source NAT if it resolves this issue.

This link provides sample config on how to configure Source NAT on ACE,
http://www.cisco.com/en/US/products/hw/modules/ps2706/products_configuration_example09186a00809c3041.shtml

Hope this helps,

Best Regards,
Rahul

[Support Team]

ACE4710 work in bridge mode. Interface Vlan 501 and 1501 are members of bridge group 1.

[dario.didio]

Hi,

Can you confirm that:

Your servers are in VLAN 501 and your upstream router is in VLAN 1501?

Your default gw of your server is the upstream router in VLAN 1501?

You allocated a resource class for your context (it is needed to explicitly specify a resource class due to the stickiness)

Could you try to only match on the VIP address (remove the "tcp eq rdp").

Are you capable of doing an RDP session directly on the server, through the ACE but on the IP of the server?

Thanks for keeping us updated.

Br,

Dario

[Support Team]

Servers in VLAN 501, Upstream router in VLAN1501. Default GW (172.17.7.1) in VLAN1501. ARP table:

Context terminalservers
================================================================================
IP ADDRESS      MAC-ADDRESS        Interface  Type      Encap  NextArp(s) Status
================================================================================
172.17.7.11     00.1c.c4.be.20.a8  vlan501   RSERVER    21     26 sec       up
172.17.7.12     00.1c.c4.a8.0b.5e  vlan501   RSERVER    20     26 sec       up
172.17.7.1      00.1f.9d.03.e0.00  vlan1501  GATEWAY    19     26 sec       up
172.17.7.10     00.16.36.fc.ae.12  vlan1501  VSERVER    LOCAL     _         up
172.17.7.4      00.12.43.dc.a3.02  bvi1      INTERFACE  LOCAL     _         up
================================================================================
Total arp entries 5
switch/terminalservers#

Yes I'm try with math ip address rule, but also unsuccessfull

No I can't connect directly to server thru ACE (RDP and other protocols)

[dario.didio]

Can you post your Admin config?

If you look in your routers ARP table, do you see the servers IP addresses + VIP address (try doing a ping to generate traffic).

Can you ping your server from the ACE?

Can you telnet port 3389 on your server from the ACE?

[Support Team]

config

switch/Admin# sh run
Generating configuration....

logging enable
logging console 7
logging buffered 7
logging monitor 7

resource-class rc1
  limit-resource all minimum 0.00 maximum unlimited
  limit-resource sticky minimum 0.20 maximum unlimited

boot system image:c4710ace-mz.A3_2_4.bin

interface gigabitEthernet 1/1
  switchport access vlan 1000
  no shutdown
interface gigabitEthernet 1/2
  switchport trunk allowed vlan 501,1501
  no shutdown
interface gigabitEthernet 1/3
  shutdown
interface gigabitEthernet 1/4
  shutdown

ntp server 172.17.44.4 prefer

ntp server 172.17.45.4

access-list ALL line 8 extended permit ip any any

class-map type management match-any remote_access
  2 match protocol xml-https any
  3 match protocol icmp any
  4 match protocol telnet any
  5 match protocol ssh any
  6 match protocol http any
  7 match protocol https any
  8 match protocol snmp any

policy-map type management first-match remote_mgmt_allow_policy
  class remote_access
    permit

interface vlan 1000
  ip address 172.17.46.164 255.255.255.0
  access-group input ALL
  service-policy input remote_mgmt_allow_policy
  no shutdown

ip route 0.0.0.0 0.0.0.0 172.17.46.1

context terminalservers
  allocate-interface vlan 501
  allocate-interface vlan 1501
  member rc1

snmp-server contact "IT"
snmp-server location "Alfa Bank"
snmp-server community public group Network-Monitor

 
username admin password 5 $1$Lx0coeEJ$FurupifAcXl4k.rsb71lu1  role Admin domain default-domain
username www password 5 $1$OX.Wdxlk$k7NZOq0yWNnQmjJa4nN8H0  role Admin domain default-domain

ssh key rsa 1024 force

sh ver

switch/Admin# sh ver
Cisco Application Control Software (ACSW)
TAC support: http://www.cisco.com/tac
Copyright (c) 1985-2009 by Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.
  loader:    Version 0.95
  system:    Version A3(2.4) [build 3.0(0)A3(2.4) adbuild_11:46:02-2009/09/27_/auto/adbu-rel2/rel_a3_2_3_throttle/REL_3_0_0_A3_2_4]
  system image file: (hd0,1)/c4710ace-mz.A3_2_4.bin
  Device Manager version 1.2 (0) 20090925:1550

Software

  installed license: no feature license is installed

Hardware
  cpu info:
    Motherboard:
        number of cpu(s): 2
    Daughtercard:
        number of cpu(s): 16
  memory info:
    total: 6226388 kB, free: 4574524 kB
    shared: 0 kB, buffers: 19004 kB, cached 0 kB
  cf info:
    filesystem: /dev/hdb2
    total: 861668 kB, used: 728664 kB, available: 89232 kB

last boot reason:  reload command by admin
configuration register:  0x1
switch kernel uptime is 0 days 23 hours 44 minute(s) 7 second(s)

fsrv6509-1#sh arp | inc Vlan1501
Internet  172.17.7.10            28   0016.36fc.ae12  ARPA   Vlan1501
Internet  172.17.7.11             2   001c.c4be.20a8  ARPA   Vlan1501
Internet  172.17.7.12             7   001c.c4a8.0b5e  ARPA   Vlan1501
Internet  172.17.7.1              -   001f.9d03.e000  ARPA   Vlan1501
Internet  172.17.7.4              2   0012.43dc.a302  ARPA   Vlan1501
fsrv6509-1#

Yes I can ping servers from ACE

Yes I can telnet to 3389 port on servers from ACE

[dario.didio]

Could you try:

policy-map type loadbalance first-match slb-corterm-vip
  class class-default
    sticky-serverfarm SG1

instead of:

policy-map type loadbalance rdp first-match slb-corterm-vip
  class class-default
    sticky-serverfarm SG1

keep us posted.

[Support Team]

I'm try with policy-map type loadbalance first-match slb-corterm-vip but this also dosn't work. As I correctly understood ACE should pass traffic in bridge mode between interfaces w/o any problem?

[dario.didio]

Does your server has more than 1 interface?

Can you try sniffing on the server side to see if traffic arrives on the server?

Can you post a "show service-policy detail" output?

[Support Team]

Servers have one Teaming Interface (two NIC connected to two cisco blade swithes, blade switches connected to 65 Catalyst, ACE4710 also connected to 65 Catalyst)

Traffic arrives on server side (but only SYN). Some of TCP packets with incorrect cheksum.

switch/terminalservers# sh service-policy detail

Policy-map : client-vips
Status     : ACTIVE
Description: -----------------------------------------
Interface: vlan 1 1501
  service-policy: client-vips
    class: slb-corterm-vip
     VIP Address:    Protocol:  Port:
     172.17.7.10     any
      loadbalance:
        L7 loadbalance policy: slb-corterm-vip
        VIP ICMP Reply       : ENABLED
        VIP State: INSERVICE
        Persistence Rebalance: DISABLED
        curr conns       : 0         , hit count        : 7        
        dropped conns    : 7        
        client pkt count : 12        , client byte count: 579                
        server pkt count : 0         , server byte count: 0                  
        conn-rate-limit      : 0         , drop-count : 0        
        bandwidth-rate-limit : 0         , drop-count : 0        
        L7 Loadbalance policy : slb-corterm-vip
          class/match : class-default
            LB action: :
               sticky group: SG1
                  primary serverfarm: CORTERM
                  primary serverfarm: CORTERM
                    state:UP
                  backup serverfarm : -
            hit count        : 6        
            dropped conns    : 0        
            compression      : off
      compression:
        bytes_in  : 0                  
        bytes_out : 0                  
        Compression ratio : 0.00%

switch/terminalservers#

[dario.didio]

Hi,

It seems somewhere a problem exists because all your connections to the VIP are being dropped:

loadbalance:
        L7 loadbalance policy: slb-corterm-vip
        VIP ICMP Reply       : ENABLED
        VIP State: INSERVICE
        Persistence Rebalance: DISABLED
        curr conns       : 0         , hit count        : 7        
        dropped conns    : 7        

We are sure that nothing assymetric exists?

We have:

Router ------ VLAN 1501 ------ ACE -------- VLAN 501 -------- Servers

VLAN 1501 and VLAN 501 have the same IP subnet + mask?

[Support Team]

I replace ACE4710 with ACE20 module. Now traffic passed from client to servers, but servers work very slowly (i think some traffic blocked by ACE).

Correct scheme:

Router (6509 + VLAN1501 SVI) -> ACE -> Router (6509 + VLAN501) ->Servers

face20-1/TerminalServers# sh service-policy

Policy-map : client-vips
Status     : ACTIVE
-----------------------------------------
Interface: vlan 1501
  service-policy: client-vips
    class: slb-corterm-vip
      loadbalance:
        L7 loadbalance policy: slb-corterm-vip
        VIP Route Metric     : 77
        VIP Route Advertise  : DISABLED
        VIP ICMP Reply       : ENABLED
        VIP State: INSERVICE
        curr conns       : 1         , hit count        : 2        
        dropped conns    : 0        
        client pkt count : 5         , client byte count: 227                
        server pkt count : 4         , server byte count: 183                
        conn-rate-limit      : 0         , drop-count : 0        
        bandwidth-rate-limit : 0         , drop-count : 0

face20-1/TerminalServers# sh conn det

total current connections : 2

conn-id    np dir proto vlan source                destination           state
----------+--+---+-----+----+---------------------+---------------------+------+
32         2  in  TCP   1501 172.17.9.147:4728     172.17.7.10:3389      ESTAB
          [ idle time   : 00:00:04,   byte count  : 110191     ]
          [ elapsed time: 00:21:25,   packet count: 2107       ]
31         2  out TCP   501  172.17.7.11:3389      172.17.9.147:4728     ESTAB
          [ conn in reuse pool : FALSE]
          [ idle time   : 00:00:04,   byte count  : 527124     ]
          [ elapsed time: 00:21:25,   packet count: 2700       ]

[Support Team]

I solved this issue!!! Problem was in HP NIC Teaming software on blade servers. If Teaming mode Auto/Auto traffic no correctly returned to ACE. I change mode to Network Fault Tolerance Only (only one NIC active) and now all works.

[Peter Koltl]

Yes, this is a known feature acknowledged by TAC. ACE does not allow traffic from the same src IP address and different src MAC addresses. Only frames with the MAC address present in the ARP table can pass through. Load-balancing NIC teaming sends out frames with two different src MAC addresses. One part of them is dropped and you may experience 'ARP collision' error messages.