The inspect FTP is likely NOT causing the issue. This command will open up any secondary ports required by the FTP protocol. You may want to confirm the speed/duplex settings on the interface. To determine the cause of the delay, you may consider doing a packet capture at the time of the delay - on the ingress and egress interface of the ASA. By analyzing this packet capture, you may be able to determine why there is a delay - dropped packets, duplicate acknowledgements, etc. You may also want to consider the MTU of the packets.
The link may be large but you may also have some upstream QoS configured limiting the throughput and/or other proxy device that may also impact the flow.