Unless it is broken device then it is network problem. Either NAT or statefull firewall decide not to allow packet to flow from outside to phone.
Catch packets on the wire - if you will see uninterupted bidirectional flow of RTP packets, then phone is broken, if packets dissapear in one direction then network is wrong.