Hi,
As per your provided description, there could be several reasons of audio bad quality, as per my understanding please make sure the below point and then check.
1. Make the placement of your access-point with the respect to the client, so the client get proper coverage with good signal strength.
2. As 100ms latency is between the client and subnet gateway not accurately between client to AP, so you need to check the wired connection between the AP and WLC and port configuration as well.
3. As you have mentioned specially audio quality is very bad, then enable configure QoS, WMM feature on WLC and access-point.