Does it become poor voice quality for everybody or just the third guy?
Are you certain you are associated to the AP you think you are or you associated to one a couple of rooms away?
For the wireless side I would lower the power on both the AP's and the phones. The object is to keep that AP's cells from bleeding over too much into the next cell. You may have to remove AP's.
Three users per AP should not be taxing the network. A voice call is like 8k or something really low like that after the initial 'hello'.
I would keep the data rates locked at 11Mbps unless I absolutely had to drop them to 5.5Mbps (some phones require a lower data rate to connect...)This keeps the cell sizes controllable.
Check both the wireless and the phone side for any configuration mistakes...
Something to think about anyway...