This is really more of a pfSense question but there are some very bright people on here so I thought I'd post for a few suggestions.
We have a failover pair of Dell 610s, each server has inbuilt 4 x 1Gbps NIC (Broadcom - bce) and we have -
bce0 - WAN
bce1 - LAN
bce2 - SYNC
they are connected to a pair of 6500 switches on modules that are wire speed (WS-X6748-GE-TX which can be oversubscribed if you use more than 20 ports per 24 but we haven't)
Although the bandwidth on the WAN side never really gets above 300 - 400 Mbps combined we see that when the number of pps exceeds around 950, 000 (they generally run between 500 - 700 pps fine) we are getting some packet loss and as this is a VoIP setup it starts affecting call quality.
I have made some modifications to the buffer size on the broadcom NICs as per Netgate docs and this seems to have made some improvement ie. still some drops but lot less than before.
You can setup etherchannel (LAGG) on the firewalls but as I am nowhere near the 1Gbps limit on a single NIC I am not sure how this could help but open to suggestions.
I can also do some configuration on the firewalls to prioritize VoIP but as it is all VoIP not sure that is going to help either.
It may just be that the firewalls cannot handle that load although the resources on the firewall are not being taxed at all.
I am not really seeing any significant drops on the physical interface connected to the firewall on the WAN side either in or out although I do see quite a few on the SVI inbound but I believe that is purely down to way the firewalls have been configured ie. a lot of CARP VIPs on the WAN side and each VIP would be sending CARPs which will be dropped by the SVI (correct me if I am wrong).
Bit of a loss as to what to look at next so like I say any suggestions are welcome.
Jon
