This is true and worth doing, but I'll go ahead and leap to a conclusion.
The ISDN debug is more than likely going to show a recovery on timer expiry. We're likely waiting on CONNECT_ACK from the PBX. This is ISDN timer T313 which defaults to four seconds, just like you're seeing. You would hear audio properly during this period. And if we're expecting/waiting on CONNECT_ACK, we're acting as the User side of the PRI. And if we're not getting it, the PBX is also acting as User side. One side needs to be Network and one side needs to be User.
If you run these debugs and verify this is the problem, you should configure the gateway as the Network side. If you're using H.323, set "isdn protocol-emulate network" on the Serial interface. If you're using CallManager/MGCP, you set it on the Gateway configuration page in CCMAdmin.
I'm trying to figure out how you could get away with calling back and forth properly between IP phones and the PBX; I'm not sure how that's working. Anyway, check out the top half of this thread below where another person had the same problem. There's debugs posted there which will hopefully match what you're seeing.
http://forums.cisco.com/eforum/servlet/NetProf?page=netprof&CommCmd=MB%3Fcmd%3Ddisplay_location%26location%3D.1dd614f9
