There are a lot of reasons this could happen. Most important thing to keep in mind, wireless is a "hub" based network, only one client can send or receive at the same time. All the others must wait (higher latency). To avoid collisions on the same frequency, the APs also honor other sending/receiving APs on the same channel to a certain degree -> more latency.
There are many more reasons for this to happen, but those are the most common ones.
To avoid them, have the correct signal strength configured on the AP and make sure not to many APs are on the same channels. Then you might also want to consider QoS or AVC, to prefer applications which are latency sensitive.
