Do you have a voice gateway that has two physical connections to your IP network?
Have you bound a specific interface to H.323? For example:
interface Loopback0 ip address 10.3.254.2 255.255.255.255 h323-gateway voip bind srcaddr 10.3.254.2 interface GigabitEthernet0/0 ip address 10.3.100.1 255.255.255.0 interface GigabitEthernet0/1
ip address 10.3.200.1 255.255.255.0
If you don't properly bind h323 to an interface, you can experience symptoms like you describe.
If that is not the problem: Can you pin this down to one particular called or calling party? Can you pin this down to an area in your IP network (i.e. calls to Floor A have issues while Floor B, C, etc. are ok)? Can you identify any pattern to good versus bad calls? Start with these questions and see if it helps narrow the problem down.