Trouble locating bandwidth bottleneck on network.

[Mogwai]

Hey all, I'm having some trouble locating a bandwidth bottleneck in our environment & not sure where it may lie. We've got a 2Gb link from out ISP, but traffic will always cap out at 800Mbps.  We've got a network with a rough topology as outlined below:

ISP (2Gb) > Firepower 2130 > Cisco 4500X L3 Switches x2 in HA > Cisco 3850 Switches x18. All the SFP's trunk / uplink SFP's are 10G capable with single-mode fiber between them.

There are no QOS policies enabled on either of the 4500X's nor are there any enabled QOS policies on the Firepower. 

The only thing I can see is that the link from the Firepower 2130 to the 4500X is currently on a PortChannel (bonded) and when looking at the settings for the Firepower interfaces I see that the speed for the PortChannel has a max of 1Gbps per interface, while a standard it has a max of 10Gbps. 

Is there anywhere else that a potential bottleneck could be hiding?

[Kasun Bandara]

@Mogwai hi, in C3850 what is the port speed which connecting to your speed testing PC? i assume you are using wired connection for testing. also can you extract the port speed when you are dong the speed test.

[Mogwai]

Hello Kasun Thanks for the answer & I guess I should've clarified my initial statement. The access ports on the 3850's are setup as full-duplex 10/100/1000, but the speed test I was referring to is actually a bandwidth monitor on the egress interface of the firewall.

[Joseph W. Doherty]

"Is there anywhere else that a potential bottleneck could be hiding?"

Etherchannel restricts a single flow to only a single link's bandwidth.

Worst case Etherchannel hashing will only use a single link for all traffic.

Dual Etherchannel, at best, usually provides about an additional 50% bandwidth.

When you start to use multigig WAN connections, you may bump into insufficient TCP's RWIN allocations to support BDP (bandwidth delay product) and/or LFN (long fat network) issues.

The forgoing are possible issues, but insufficient information to say any are actual issues.  Also there are other possibilities, but the above are more likely based on the information provided in OP.

[Mogwai]

Thanks & that was my suspicion as well, but I guess I'll have to circle back to this once we approve some downtime in order to swap the interface over. 

[Leo Laohoo]

Bypass the FW.