Virtualized Voice Browser and F5 load balancers

[clearbluelou]

We are in the process of migrating from CVP 9.0 using VXML Gateways on 3925E routers to CVP 11.6 using VVB 11.6 virtual machines.  We are using the same F5 load balancers for distributing HTTP requests for both VXML and media (WAV files) in both environments but with different VIP for each.  The CVP9 VXML VIP and the CVP11 VXML VIP were configured to use the same persistence profile with cookie insert.  During testing, we found that once we put about four or more calls into CVP11, we started randomly encountering a "no_session_error" message reported by the VXML server to the VVB.  After much digging through logs and packet captures with TAC and BU engineers to no avail, we decided to set the CVP11 VXML VIP to persistence based on the source IP instead of the cookie.  This completely fixed the problem.  We pumped 40+ concurrent calls through with no errors.

Our conclusion, though TAC won't agree, is that the VVB handles the HTTP conversation with the VXML server differently from the VXML Gateways.  The packet captures confirmed that the VVB does not change the cookie (i.e. JSESSSIONID) when it shouldn't.  Yet, the F5 randomly load balanced the HTTP request to the wrong VXML server, resulting in the "no_session_error".  We even tested the F5 persistence by pointing the VVB servers to the same VXML VIP that we're using in production with CVP 9.0.  We still received the random no_session_error message, thus proving the VIP config was not the problem.

I asked TAC to issue a Technote regarding the use of third-party load balancers with the VVB servers and persistent HTTP sessions.  They did not see the need.  I just hope our experience helps someone else avoid this headache.

[bill.king1]

Thank you for sharing. Since you already have a case open, maybe you can ask them to add to/clarify the documentation on the compatibility matrix re: load balancers to add in the additional detail/requirements you're seeing? Right now it only mentions

• Persistence - cookie-insert
• Distribution algorithm - Round-robin
  
  

And it technically doesn't list VVB as part of the supported applications. At least this way it gets documented somewhere?

[clearbluelou]

I tried.  The TAC engineer told me that he didn't think it was necessary to update the documentation.

[bill.king1]

So the documentation that is online is different than what TAC is telling you it should be for it to work correctly? That is crazy. Did the TAC employee's manager agree?

[clearbluelou]

So that's where Cisco has gotten a bit shifty.  I presume you're talking about this documentation:  https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/icm_enterprise/ucce_compatibility/matrix/ucce_11_6_x_compat.html#Load_Balancers

If so, notice that it's pretty ambiguous until it gets to the link calling out the F5 BIG-IP specifically.  If you follow that link, it opens a document dated 2013.  I already knew about this document because it's what I used to build the CVP 9.0 system with VXML Gateways that we have in production today.  When I asked why they had not updated the F5 documentation since 2013, especially in relation to the VVB, they said they have found they can't keep up with all of the changes third-parties make and so have left it up to the customer to consult with the third-party documentation directly.  When we did so, we found that our F5 was configured exactly the way it should be for cookie-insert persistence.  Yet, something about the VVB caused the F5 to randomly drop persistence, thus our decision to switch to source-IP persistence.

As for the CVP 11.6 documentation, the Configuration Guide only references using Cisco ACE for load-balancing (Chapter 19, page 399).  No mention of third-party load balancers.  The VVB 11.6 documentation is even more sparse, making no mention of HA design whatsoever.