Firstly, you should be able to login to the web interface of your TP endpoint during a call to check for packet loss - optimally, it should be at 0% or very close to (less than 1 - 2%)
What are you using to monitor your bandwidth? Most tools will look at a 5-minute average, so a 5 minute average of 3Mbps will likely include many shorter duration spikes.
How much bandwidth are you using per video call? If you only have a 4Mbps link, most QoS policies won't allocate more than 30% of bandwidth to real time traffic, so if you're running any more than around 1Mbps per call you would likely exceed this, and that's assuming your switches and routers are configured properly for QoS.
QoS is unfortunately a complicated beast, and you're likely to have difficulties running video on a 4Mbps link if you're sharing the link with other applications.