I am running 2 x Active/Active 6509-E chassis with ACE & FWSM Mods in multi context in a high avaliability Data Centre.
Ruuning IOS 122-33.SXI2a
Running FWSM 4.1(2)
Running ACE A2(3.2
We are seeing some wierd issues and wonder if you can help. Using the network for predominately load balancing about 70k Proxy Users.
Most our contexts do this CORE HSRP MSFC ------> Routed ACE ---------> FWSM ---------> Servers. The Core MSFC is the central layer 2/3 gateway for all devices.
1) I have MAC flap noticed on the ISP provider switches for the FWSM at Site A) over a QinQ Trunk but no loops in the network. We don't notice internally any MAC flaps. There are no other MAC flaps for any other products.
2) Intermitent Proxy Access, some times fast, others painfully slow. Extensive sniffer traces on the Rservers and domain controllers, show high duplicate ACKs and OoO packets. For a while this was stable of about 1 week, after I removed TCP normalisation and issued sysopt np on the FWSM .... but again it has reared its ugly head.
3) We hit an ACL limit of 14,000 on a particular context after adding some rules which gave us a config error. After this, and dring a major change for most of the day we started to get issues related to that context. Traffic not getting through etc. Packets getting lost in transit from one context to another. After scratching our heads for most the day, I back tracked and decided to totally removing all ACLs on the effected conext. I added them back and traffic resumed as normal.
4) After a recent failover test we lost a 0.0.0.0 route on a particular context. We duplicated the test and it happened again. However the customer is now concerned about product stability.
5) Again during a failover everything seemed normal and we were runiing ACTIVE/ACTIVE at site B) for all groups. However we started to loose packets during our ping tests and it didn't become stable until we failed back.
6) There is sporadic consitency from Site A) & B) to various FWSM standby interfaces as if its geting confused who owns what.