Strange ARP Problems with C170 and AsyncOS 9

[Secnet DRVBundGQ]

after upgrading to asyncOS 9.0 (Ironport C170) we have the following problem.

For better understanding a short explanation (without all network devices)

The traffic flow is

Lan --- Application Firewall ---Ironport 

During a connection between the Firewall and the Ironport, the Ironport is unable to make a  response.
It seems the Ironport is unable to make an arp resolution for the virtual cluster ip from the firewall.

E.g. ping from the firewall with the virtual cluster ip as source won't work.
Ping from the firewall with the physical interface as source works fine.

AsyncOS prior to version 9 has not such problems.

The arp table shows following entry for the virtual cluster ip (AsynOS) 

(xxx.xxx.103.254) at (incomplete) on em1 expired [ethernet]

Explantation:

xx.103.254 with mac  01:00:5e:19:67:fe = virtual cluster ip
xx.103.128 with mac  00:e0:ed:37:05:1a = physical interface ip

Ping from "xxx.103.254 Cluster IP" as source  to xxx.103.135 (cisco Ironport) as destination

The ICMP Packet went from the virtual Cluster Interface (xxx.25.103.254) with mac-adress 05:1a (physical interface) to the ironport.
The ironport makes an arp request...who is xxx.25.103.254?..and receives as answer the OTHER mac-address (virtual Clusterinterface) 67:fe.
I think, the ironport with the new asyncOS has some troubles with this 2 different mac-addresses.

No.     Time        Source                Destination           Protocol Length Info
     10 4.115231    xxx.25.103.254        xxx.25.103.135        ICMP     98     Echo (ping) request  id=0xaa26, seq=0/0, ttl=64 (no response found!)

Frame 10: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
Ethernet II, Src: Silicom_37:05:1a (00:e0:ed:37:05:1a), Dst: Cisco_9c:ba:3a (50:3d:e5:9c:ba:3a)
Internet Protocol Version 4, Src: xxx.25.103.254 (xxx.25.103.254), Dst: xxx.25.103.135 (xxx.25.103.135)
Internet Control Message Protocol

No.     Time        Source                Destination           Protocol Length Info
     11 4.115251    Cisco_9c:ba:3a        Broadcast             ARP      42     Who has xxx.25.103.254?  Tell xxx.25.103.135

Frame 11: 42 bytes on wire (336 bits), 42 bytes captured (336 bits)
Ethernet II, Src: Cisco_9c:ba:3a (50:3d:e5:9c:ba:3a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)

No.     Time        Source                Destination           Protocol Length Info
     12 4.115365    Silicom_37:05:1a      Cisco_9c:ba:3a        ARP      60     xxx.25.103.254 is at 01:00:5e:19:67:fe

Frame 12: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Ethernet II, Src: Silicom_37:05:1a (00:e0:ed:37:05:1a), Dst: Cisco_9c:ba:3a (50:3d:e5:9c:ba:3a)

any ideas?

[jawulf.chisago]

I just upgraded to asyncOS 9.1 (Ironport C170) and no running into the same issue.  I can ping our internal exchange servers but not the NLB IP. When I look at arp it shows (incomplete) on vlan1 expired [vlan].

Did you ever find a solution to this?

[Murad Al Halawa · Accepted solution]

Hello ,

this could be related to defect : https://tools.cisco.com/bugsearch/bug/CSCuu04069/?reffering_site=dumpcr

Workaround:
Configure the NLB in Unicast Mode rather than Multicast Mode.

To be sure please don't hesitate to open ticket with TAC.

Regards

Murad Al Halawa

[Nasir Abbas]

Hi Cisco Email Security Appliance Community,

Introduction

The Email Security Appliance(ESA) is having issues due to RFC compliance.

This document will describe this issue and provide available workarounds.

Problem Description

ESA AsynOS 9.X is not responding to multicast arps and become more strict with how it behaves with ARPs as it that freeBSD 9.2 that runs ASYNCOS 9.x is behaving like a router.  In a nutshell, Microsoft is using different/relaxed approach towards RFC 1812. 

Available Workarounds

From Routers and Switches

Administrators will have to configure a lot of static ARP entries on routers and switches that sit between the networks the ESA and the Network Load Balancer  (NBL) are housed. 

3.3.2 Address Resolution Protocol - ARP

Routers that implement ARP MUST be compliant and SHOULD be unconditionally compliant with the requirements as below:

The link layer MUST NOT report a Destination Unreachable error to IP solely because there is no ARP cache entry for a destination; it SHOULD queue up to a small number of datagrams breifly while performing the ARP request/reply sequence, and reply that the destination is unreachable to one of the queued datagrams only when this proves fruitless.

 A router MUST not believe any ARP reply that claims that the Link Layer address of another host or router is a broadcast or multicast

Further reading: https://tools.ietf.org/html/rfc1812#section-3.3.2

Microsoft System Workaround:

There are three available options:

1. Apply the hot patch provided by MS support
   1. Further reading on the available hotpatch is available https://support.microsoft.com/en-us/kb/960916
2. Change to mode on MS NLB to use unicast
3. Create Static MAC entry on MS server as provided below:

Add a static ARP table entry for the Default Gateway IP address on the NLB Node.

Command to add static ARP entry

Arp –s <IP address> <Mac Address>

Note: Microsoft is aware of this issue and we will keep you posted regarding its status. For more information regarding use of the ARP command in Windows, see the following:

http://technet.microsoft.com/en-us/library/bb490864.aspx

Cisco ESA workarounds:

1.  Configure the physical IP addresses of MS Exchange server in SMTPROUTE and Sender group.
2.  Revert back to previous version of AsyncOS of 8.5.6  (Not recommended)

NOTE: All configuration and data will be removed when revert command is being used. Please make sure to have a copy of configuration with passwords unmasked of version AsyncOS 8.5.6 is available to be loaded after a revert is used.

Related Information

Following technote from Microsoft provides detail information about what is changed in behavior between 2003 and 2008.

http://blogs.technet.com/b/networking/archive/2009/01/15/unable-to-connect-to-windows-server-2008-nlb-virtual-ip-address-from-hosts-in-different-subnets-when-nlb-is-in-multicast-mode.aspx

[rajebalab]

Hi Team ,

We are facing the similar problem where we are using the older version, already there is TAC request has been raised but there was no response for the same.

Current Version
===============
Product: Cisco IronPort C370 Messaging Gateway(tm) Appliance
Model: C370
Version: 8.5.6-092
Build Date: 2014-09-02
Install Date: 2014-12-13 05:08:42
BIOS: 2.2.17C
RAID: 1.21.02-0528, 2.01.00, 1.02-014B
RAID Status: Optimal
RAID Type: 1

Is there anything else which we can work on and get this resolved, apart from adding the MAC address in Load Balance

[Secnet DRVBundGQ]

Many thanks for your answer.

We just found a workaround to circumvent the problem.